Run Python scripts

DPDispatcher can be used to directly run a single Python script:

dpdisp run script.py

The script must include inline script metadata compliant with PEP 723. An example of the script is shown below.

# /// script
# # dpdispatcher doesn't use `requires-python` and `dependencies`
# requires-python = ">=3"
# dependencies = [
# ]
# [tool.dpdispatcher]
# work_base = "./"
# forward_common_files=[]
# backward_common_files=[]
# [tool.dpdispatcher.machine]
# batch_type = "Shell"
# local_root = "./"
# context_type = "LazyLocalContext"
# [tool.dpdispatcher.resources]
# number_node = 1
# cpu_per_node = 1
# gpu_per_node = 0
# group_size = 0
# [[tool.dpdispatcher.task_list]]
# # no need to contain the script filename
# command = "python"
# # can be a glob pattern
# task_work_path = "./"
# forward_files = []
# backward_files = ["log"]
# ///

print("hello world!")

The PEP 723 metadata entries for tool.dpdispatcher are defined as follows:

pep723:

type: dict

argument path: pep723

PEP 723 metadata

work_base:

type: str, optional, default: ./

argument path: pep723/work_base

Base directory for the work

forward_common_files:

type: typing.List[str], optional, default: []

argument path: pep723/forward_common_files

Common files to forward to the remote machine

backward_common_files:

type: typing.List[str], optional, default: []

argument path: pep723/backward_common_files

Common files to backward from the remote machine

machine:

type: dict

argument path: pep723/machine

Machine configuration. See related documentation for details.

resources:

type: dict

argument path: pep723/resources

Resources configuration. See related documentation for details.

task_list:

type: list

argument path: pep723/task_list

List of tasks to execute.

This argument takes a list with each element containing the following:

command:

type: str, optional, default: python

argument path: pep723/task_list/command

Python interpreter or launcher. No need to contain the Python script filename.

task_work_path:

type: str

argument path: pep723/task_list/task_work_path

The dir where the command to be executed. Can be a glob pattern.

forward_files:

type: typing.List[str], optional, default: []

argument path: pep723/task_list/forward_files

The files to be uploaded in task_work_path before the task exectued.

backward_files:

type: typing.List[str], optional, default: []

argument path: pep723/task_list/backward_files

The files to be download to local_root in task_work_path after the task finished

outlog:

type: str | NoneType, optional, default: log

argument path: pep723/task_list/outlog

The out log file name. redirect from stdout

errlog:

type: str | NoneType, optional, default: err

argument path: pep723/task_list/errlog

The err log file name. redirect from stderr