Resources parameters

Note

One can load, modify, and export the input file by using our effective web-based tool DP-GUI online or hosted using the command line interface dpdisp gui. All parameters below can be set in DP-GUI. By clicking “SAVE JSON”, one can download the input file for.

resources:

type: dict

argument path: resources

number_node:

type: int, optional, default: 1

argument path: resources/number_node

Number of nodes requested for each scheduler job generated by DPDispatcher.

cpu_per_node:

type: int, optional, default: 1

argument path: resources/cpu_per_node

Number of CPUs requested on each node for each scheduler job.

gpu_per_node:

type: int, optional, default: 0

argument path: resources/gpu_per_node

Number of GPUs requested on each node for each scheduler job.

queue_name:

type: str, optional, default: (empty string)

argument path: resources/queue_name

Queue or partition name used by the selected batch system. For local Shell runs this is usually an empty string; for Slurm it typically maps to a partition.

group_size:

type: int

argument path: resources/group_size

How many tasks are packed into one scheduler job. For example, 20 tasks with group_size=5 are typically split into 4 jobs. Use 1 for the simplest one-task workflow. 0 means no explicit upper limit in the grouping logic.

custom_flags:

type: typing.List[str], optional

argument path: resources/custom_flags

Extra scheduler-header lines inserted into the generated submission script, typically for backend-specific options that are not covered by the standard fields.

strategy:

type: dict, optional

argument path: resources/strategy

Strategy options that affect how DPDispatcher generates and evaluates submission scripts.

if_cuda_multi_devices:

type: bool, optional, default: False

argument path: resources/strategy/if_cuda_multi_devices

If a node has multiple NVIDIA GPUs, assign different tasks inside the same job to different GPUs by setting CUDA_VISIBLE_DEVICES automatically. Usually used together with para_deg > 1 and task-level resource awareness.

ratio_unfinished:

type: float, optional, default: 0.0

argument path: resources/strategy/ratio_unfinished

Maximum fraction of tasks allowed to remain unfinished when evaluating job completion. Use 0.0 for the strict default that requires every task to finish.

customized_script_header_template_file:

type: str, optional

argument path: resources/strategy/customized_script_header_template_file

Custom template file for the scheduler-header portion of generated submission scripts. Overrides the default template.

para_deg:

type: int, optional, default: 1

argument path: resources/para_deg

How many tasks inside one generated job are run in parallel. This is different from group_size: group_size controls how many tasks are bundled into a job, while para_deg controls concurrency within that job. Keep para_deg=1 for the safest default.

source_list:

type: typing.List[str], optional, default: []

argument path: resources/source_list

Shell scripts or environment files sourced before task commands run. Useful on HPC systems for activating software stacks explicitly instead of relying on login-shell defaults.

module_purge:

type: bool, optional, default: False

argument path: resources/module_purge

Whether to run ‘module purge’ before applying module_unload_list and module_list. Mainly useful on HPC systems.

module_unload_list:

type: typing.List[str], optional, default: []

argument path: resources/module_unload_list

Modules to unload before loading the requested modules. Mainly relevant on HPC systems with environment modules.

module_list:

type: typing.List[str], optional, default: []

argument path: resources/module_list

Modules to load before executing tasks. Mainly relevant on HPC systems with environment modules.

envs:

type: dict, optional, default: {}

argument path: resources/envs

Environment variables exported before executing tasks.

prepend_script:

type: typing.List[str], optional, default: []

argument path: resources/prepend_script

Optional shell lines inserted before task commands in the generated job script.

append_script:

type: typing.List[str], optional, default: []

argument path: resources/append_script

Optional shell lines inserted after task commands in the generated job script.

wait_time:

type: float | int, optional, default: 0

argument path: resources/wait_time

Delay in seconds inserted after a job is submitted or resubmitted. Usually keep 0 unless the scheduler/site asks you to throttle submission pace.

Depending on the value of batch_type, different sub args are accepted.

batch_type:

type: str (flag key)

argument path: resources/batch_type

possible choices: OpenAPI, DistributedShell, SGE, Shell, PBS, Fugaku, Torque, SlurmJobArray, JH_UniScheduler, LSF, Bohrium, Slurm

The batch job system type loaded from machine/batch_type.

When batch_type is set to OpenAPI (or its alias openapi):

kwargs:

type: dict, optional

argument path: resources[OpenAPI]/kwargs

This field is empty for this batch.

When batch_type is set to DistributedShell (or its alias distributedshell):

kwargs:

type: dict, optional

argument path: resources[DistributedShell]/kwargs

This field is empty for this batch.

When batch_type is set to SGE (or its alias sge):

kwargs:

type: dict

argument path: resources[SGE]/kwargs

SGE-specific extra arguments.

pe_name:

type: str, optional, default: mpi, alias: sge_pe_name

argument path: resources[SGE]/kwargs/pe_name

Parallel environment name used by SGE, for example mpi. This controls the #$ -pe … header line in SGE mode.

job_name:

type: str, optional, default: wDPjob

argument path: resources[SGE]/kwargs/job_name

Job name shown by SGE for this submission.

When batch_type is set to Shell (or its alias shell):

kwargs:

type: dict, optional

argument path: resources[Shell]/kwargs

This field is empty for this batch.

When batch_type is set to PBS (or its alias pbs):

kwargs:

type: dict, optional

argument path: resources[PBS]/kwargs

This field is empty for this batch.

When batch_type is set to Fugaku (or its alias fugaku):

kwargs:

type: dict, optional

argument path: resources[Fugaku]/kwargs

This field is empty for this batch.

When batch_type is set to Torque (or its alias torque):

kwargs:

type: dict, optional

argument path: resources[Torque]/kwargs

This field is empty for this batch.

When batch_type is set to SlurmJobArray (or its alias slurmjobarray):

kwargs:

type: dict, optional

argument path: resources[SlurmJobArray]/kwargs

Slurm-specific extra arguments.

custom_gpu_line:

type: str | NoneType, optional, default: None

argument path: resources[SlurmJobArray]/kwargs/custom_gpu_line

Custom GPU header line starting with #SBATCH. When set, it overrides DPDispatcher’s default Slurm GPU line generated from gpu_per_node.

slurm_job_size:

type: int, optional, default: 1

argument path: resources[SlurmJobArray]/kwargs/slurm_job_size

For SlurmJobArray, how many DPDispatcher tasks are grouped into one array element / Slurm job script branch.

When batch_type is set to JH_UniScheduler (or its alias jh_unischeduler):

kwargs:

type: dict

argument path: resources[JH_UniScheduler]/kwargs

JH_UniScheduler-specific extra arguments.

custom_gpu_line:

type: str | NoneType, optional, default: None

argument path: resources[JH_UniScheduler]/kwargs/custom_gpu_line

Custom GPU header line starting with #JSUB. When set, it overrides the default UniScheduler GPU line generated from gpu_per_node.

When batch_type is set to LSF (or its alias lsf):

kwargs:

type: dict

argument path: resources[LSF]/kwargs

LSF-specific extra arguments.

gpu_usage:

type: bool, optional, default: False

argument path: resources[LSF]/kwargs/gpu_usage

Whether DPDispatcher should emit an LSF GPU request line at all. If False, no GPU request header is added.

gpu_new_syntax:

type: bool, optional, default: False

argument path: resources[LSF]/kwargs/gpu_new_syntax

Whether to use the newer #BSUB -gpu syntax instead of the older resource string syntax. This is typically used on newer LSF versions.

gpu_exclusive:

type: bool, optional, default: True

argument path: resources[LSF]/kwargs/gpu_exclusive

Only meaningful when gpu_new_syntax is enabled. Controls whether the submitted job requests GPUs in exclusive mode.

custom_gpu_line:

type: str | NoneType, optional, default: None

argument path: resources[LSF]/kwargs/custom_gpu_line

Custom GPU header line starting with #BSUB. When set, it overrides the GPU-related LSF header generated from the other GPU kwargs.

When batch_type is set to Bohrium (or its aliases bohrium, Lebesgue, lebesgue, DpCloudServer, dpcloudserver):

kwargs:

type: dict, optional

argument path: resources[Bohrium]/kwargs

This field is empty for this batch.

When batch_type is set to Slurm (or its alias slurm):

kwargs:

type: dict, optional

argument path: resources[Slurm]/kwargs

Slurm-specific extra arguments.

custom_gpu_line:

type: str | NoneType, optional, default: None

argument path: resources[Slurm]/kwargs/custom_gpu_line

Custom GPU header line starting with #SBATCH. When set, it overrides DPDispatcher’s default Slurm GPU line generated from gpu_per_node.