DPDispatcher’s documentation

DPDispatcher generates job scripts for HPC schedulers such as Slurm, PBS, LSF, JH UniScheduler, and cloud backends. It submits those jobs, monitors them until completion, and transfers declared input and result files.

The same Python API works with local shell execution, SSH-connected clusters, distributed filesystems, and supported cloud services.

Please cite the following paper if you use this project in your work:

• Fengbo Yuan, Zhaohan Ding, Yun-Pei Liu, Kai Cao, Jiahao Fan, Cao Thang Nguyen, Yuzhi Zhang, Haidi Wang, Yixiao Chen, Jiameng Huang, Tongqi Wen, Mingkang Liu, Yifan Li, Yong-Bin Zhuang, Hao Yu, Ping Tuo, Yaotang Zhang, Yibo Wang, Linfeng Zhang, Han Wang, Jinzhe Zeng, DPDispatcher: Scalable HPC Task Scheduling for AI-Driven Science, J. Chem. Inf. Model., 2025.

Contents:

• Install DPDispatcher
• Getting Started
• Python API guide
  • Minimal local submission
  • Paths and file staging
  • Task grouping and parallelism
  • Loading configuration
  • Submission lifecycle
  • Extending DPDispatcher
• Supported contexts
  • LazyLocal
  • Local
  • SSH
  • Bohrium
  • HDFS
  • OpenAPI
• Supported batch job systems
  • Shell
  • Slurm
  • OpenPBS or PBSPro
  • TORQUE
  • LSF
  • JH UniScheduler
  • Bohrium
  • DistributedShell
  • Fugaku
  • OpenAPI
  • SGE
• Support DPDispatcher on Yarn
  • Background
  • Design
  • Implement
• Machine parameters
• Resources parameters
• Task parameters
• Environment variables
• Run Python scripts
  • $ref support
• Submit from JSON file
  • Options
• Agent Skills
  • List of skills
  • Related reference
  • Install skills
  • Minimal verification
• Command line interface
  • Valid subcommands
  • Sub-commands
• DPDispatcher API
  • dpdispatcher package

Examples

• Running the DeePMD-kit on the Expanse cluster
• Running Gaussian 16 with failure allowed
• Running multiple MD tasks on a GPU workstation
• Customizing the submission script header

Project details

• Authors

Indices and tables

• Index

• Module Index

• Search Page