Resources parameters

Note

One can load, modify, and export the input file by using our effective web-based tool DP-GUI. All parameters below can be set in DP-GUI. By clicking “SAVE JSON”, one can download the input file for.

resources:
type: dict
argument path: resources
number_node:
type: int
argument path: resources/number_node

The number of node need for each job

cpu_per_node:
type: int
argument path: resources/cpu_per_node

cpu numbers of each node assigned to each job.

gpu_per_node:
type: int
argument path: resources/gpu_per_node

gpu numbers of each node assigned to each job.

queue_name:
type: str
argument path: resources/queue_name

The queue name of batch job scheduler system.

group_size:
type: int
argument path: resources/group_size

The number of tasks in a job.

custom_flags:
type: list, optional
argument path: resources/custom_flags

The extra lines pass to job submitting script header

strategy:
type: dict, optional
argument path: resources/strategy

strategies we use to generation job submitting scripts.

if_cuda_multi_devices:
type: bool, optional, default: False
argument path: resources/strategy/if_cuda_multi_devices
ratio_unfinished:
type: float, optional, default: 0.0
argument path: resources/strategy/ratio_unfinished
para_deg:
type: int, optional, default: 1
argument path: resources/para_deg

Decide how many tasks will be run in parallel.

source_list:
type: list, optional, default: []
argument path: resources/source_list

The env file to be sourced before the command execution.

module_purge:
type: bool, optional, default: False
argument path: resources/module_purge

Remove all modules on HPC system before module load (module_list)

module_unload_list:
type: list, optional, default: []
argument path: resources/module_unload_list

The modules to be unloaded on HPC system before submitting jobs

module_list:
type: list, optional, default: []
argument path: resources/module_list

The modules to be loaded on HPC system before submitting jobs

envs:
type: dict, optional, default: {}
argument path: resources/envs

The environment variables to be exported on before submitting jobs

wait_time:
type: int | float, optional, default: 0
argument path: resources/wait_time

The waitting time in second after a single task submitted

Depending on the value of batch_type, different sub args are accepted.

batch_type:
type: str (flag key)
argument path: resources/batch_type
possible choices: Lebesgue, DistributedShell, Torque, Shell, SlurmJobArray, PBS, Slurm, DpCloudServer, LSF

The batch job system type loaded from machine/batch_type.

When batch_type is set to Lebesgue (or its alias lebesgue):

kwargs:
type: dict, optional
argument path: resources[Lebesgue]/kwargs

This field is empty for this batch.

When batch_type is set to DistributedShell (or its alias distributedshell):

kwargs:
type: dict, optional
argument path: resources[DistributedShell]/kwargs

This field is empty for this batch.

When batch_type is set to Torque (or its alias torque):

kwargs:
type: dict, optional
argument path: resources[Torque]/kwargs

This field is empty for this batch.

When batch_type is set to Shell (or its alias shell):

kwargs:
type: dict, optional
argument path: resources[Shell]/kwargs

This field is empty for this batch.

When batch_type is set to SlurmJobArray (or its alias slurmjobarray):

kwargs:
type: dict, optional
argument path: resources[SlurmJobArray]/kwargs

Extra arguments.

custom_gpu_line:
type: NoneType | str, optional, default: None
argument path: resources[SlurmJobArray]/kwargs/custom_gpu_line

Custom GPU configuration, starting with #SBATCH

When batch_type is set to PBS (or its alias pbs):

kwargs:
type: dict, optional
argument path: resources[PBS]/kwargs

This field is empty for this batch.

When batch_type is set to Slurm (or its alias slurm):

kwargs:
type: dict, optional
argument path: resources[Slurm]/kwargs

Extra arguments.

custom_gpu_line:
type: NoneType | str, optional, default: None
argument path: resources[Slurm]/kwargs/custom_gpu_line

Custom GPU configuration, starting with #SBATCH

When batch_type is set to DpCloudServer (or its alias dpcloudserver):

kwargs:
type: dict, optional
argument path: resources[DpCloudServer]/kwargs

This field is empty for this batch.

When batch_type is set to LSF (or its alias lsf):

kwargs:
type: dict
argument path: resources[LSF]/kwargs

Extra arguments.

gpu_usage:
type: bool, optional, default: False
argument path: resources[LSF]/kwargs/gpu_usage

Choosing if GPU is used in the calculation step.

gpu_new_syntax:
type: bool, optional, default: False
argument path: resources[LSF]/kwargs/gpu_new_syntax

For LFS >= 10.1.0.3, new option -gpu for #BSUB could be used. If False, and old syntax would be used.

gpu_exclusive:
type: bool, optional, default: True
argument path: resources[LSF]/kwargs/gpu_exclusive

Only take effect when new syntax enabled. Control whether submit tasks in exclusive way for GPU.

custom_gpu_line:
type: NoneType | str, optional, default: None
argument path: resources[LSF]/kwargs/custom_gpu_line

Custom GPU configuration, starting with #BSUB