Python API guide

Python API guide#

Most Python integrations use four objects imported directly from dpdispatcher:

Object

Responsibility

Machine

Selects a batch backend and an execution context.

Resources

Describes the resources and grouping strategy for each generated job.

Task

Describes one command, its working directory, and the files to transfer.

Submission

Groups tasks, generates jobs, transfers files, monitors execution, and downloads results.

Job is also public because it is useful for inspection and recovery, but applications normally let Submission.generate_jobs create jobs from tasks.

Minimal local submission#

The following example runs a shell task in the current directory without an HPC scheduler or file transfer:

from pathlib import Path

from dpdispatcher import Machine, Resources, Submission, Task

work_root = Path.cwd()
machine = Machine.load_from_dict(
    {
        "batch_type": "Shell",
        "context_type": "LazyLocalContext",
        "local_root": str(work_root),
    }
)
resources = Resources(
    number_node=1,
    cpu_per_node=1,
    gpu_per_node=0,
    queue_name="",
    group_size=1,
)
task = Task(
    command="printf 'done\\n' > result.txt",
    task_work_path=".",
    backward_files=["result.txt"],
    outlog="task.out",
    errlog="task.err",
)

submission = Submission(
    work_base=".",
    machine=machine,
    resources=resources,
    task_list=[task],
)
submission.run_submission(check_interval=1, clean=False)

Retry exhaustion is fail-fast by default. To finish monitoring sibling jobs and report all exhausted jobs together, opt in either when constructing the submission or for a single run:

submission = Submission(
    work_base=".",
    machine=machine,
    resources=resources,
    task_list=[task],
    continue_on_failure=True,
)
submission.run_submission(check_interval=1, clean=False)
# Alternatively: submission.run_submission(continue_on_failure=True)

Use clean=False while debugging so the execution directory and generated scripts remain available for inspection. For remote contexts, the default clean=True removes the submission-specific remote directory after declared outputs have been downloaded.

Paths and file staging#

DPDispatcher resolves paths in layers:

  1. machine.local_root identifies the local project root.

  2. submission.work_base identifies the submission directory below that root.

  3. task.task_work_path identifies a task directory below work_base.

  4. forward_files and backward_files are resolved below the task directory.

Files shared by every task belong in forward_common_files or backward_common_files on the submission. Keep task working paths relative; absolute task paths bypass the staging model and are not supported.

LazyLocalContext runs in the existing local directory. LocalContext stages work into another directory on the same host. Remote contexts such as SSHContext stage files to a submission-specific directory below machine.remote_root.

Task grouping and parallelism#

Resources.group_size and Resources.para_deg control different layers:

  • group_size is the maximum number of tasks packed into one scheduler job. A value of 0 places all tasks in one generated job.

  • para_deg controls how many tasks inside that generated job run concurrently.

Start with both values set to 1. Increase group_size to reduce scheduler overhead, and increase para_deg only when the allocated CPUs or GPUs can run multiple task commands safely.

Loading configuration#

The core objects accept dictionaries directly and provide JSON/YAML loaders:

machine = Machine.load_from_json("machine.json")
resources = Resources.load_from_yaml("resources.yaml")
task = Task.load_from_dict(task_config)

All loaders normalize and validate configuration with dargs. External $ref resolution is disabled by default because it reads additional local files. Pass allow_ref=True only for trusted configuration:

task = Task.load_from_json("task.json", allow_ref=True)
machine = Machine.load_from_dict(machine_config, allow_ref=True)
resources = Resources.load_from_dict(resources_config, allow_ref=True)

Note

The Machine and Resources JSON/YAML convenience loaders currently load one file directly and do not expose allow_ref. Read trusted files into a mapping and use Machine.load_from_dict or Resources.load_from_dict when $ref resolution is required.

Submission lifecycle#

Submission.run_submission performs the complete lifecycle:

  1. Generate deterministic groups of jobs from the registered tasks.

  2. Recover compatible scheduler state from a previous run when available.

  3. Upload task files and generated scripts.

  4. Submit or resubmit jobs and poll their normalized status.

  5. Download declared results.

  6. Save recovery state and, by default, clean the execution directory.

Important keyword arguments are:

  • dry_run=True: stage inputs and scripts without submitting jobs.

  • exit_on_submit=True: submit jobs and return without polling for completion.

  • clean=False: retain the execution directory after downloading results.

  • check_interval=N: poll scheduler status every N seconds.

  • continue_on_failure=True: continue monitoring sibling jobs after retry exhaustion; omitted or False preserves fail-fast behavior.

For concurrent submissions, use Submission.async_run_submission. The asynchronous wrapper runs the blocking lifecycle in an executor and defaults to clean=False, which avoids one submission removing files another operation may still need.

Extending DPDispatcher#

A new scheduler subclasses Machine and implements the scheduler-specific submission, status, script-header, and finish-tag methods. A new execution environment subclasses BaseContext and implements transfer, cleanup, file access, and command execution. Subclasses register automatically under their class name and aliases when imported.

The generated DPDispatcher API pages contain the complete module and method reference.

Contents