Contents

pep723:#
type: dict
argument path: pep723

PEP 723 metadata

work_base:#
type: str, optional, default: ./
argument path: pep723/work_base

Base directory for the work

forward_common_files:#
type: typing.List[str], optional, default: []
argument path: pep723/forward_common_files

Common files to forward to the remote machine

backward_common_files:#
type: typing.List[str], optional, default: []
argument path: pep723/backward_common_files

Common files to backward from the remote machine

machine:#
type: dict
argument path: pep723/machine

Machine configuration. See related documentation for details.

batch_type:#
type: str
argument path: pep723/machine/batch_type

The batch job system type. Option: Torque, PBS, OpenAPI, DistributedShell, LSF, Slurm, SGE, Fugaku, SlurmJobArray, Bohrium, Shell, JH_UniScheduler

local_root:#
type: str | NoneType
argument path: pep723/machine/local_root

The dir where the tasks and relating files locate. Typically the project dir.

remote_root:#
type: str | NoneType, optional
argument path: pep723/machine/remote_root

The dir where the tasks are executed on the remote machine. Only needed when context is not lazy-local.

clean_asynchronously:#
type: bool, optional, default: False
argument path: pep723/machine/clean_asynchronously

Clean the remote directory asynchronously after the job finishes.

retry_count:#
type: int, optional, default: 3
argument path: pep723/machine/retry_count

Number of retries to resubmit failed jobs.

Depending on the value of context_type, different sub args are accepted.

context_type:#
type: str (flag key)
argument path: pep723/machine/context_type
possible choices: OpenAPIContext, LocalContext, HDFSContext, SSHContext, LazyLocalContext, BohriumContext

The connection used to remote machine. Option: LocalContext, LazyLocalContext, SSHContext, BohriumContext, HDFSContext, OpenAPIContext

When context_type is set to OpenAPIContext (or its aliases openapicontext, OpenAPI, openapi):

remote_profile:#
type: dict, optional
argument path: pep723/machine[OpenAPIContext]/remote_profile

The information used to maintain the connection with remote machine. This field is empty for this context.

When context_type is set to LocalContext (or its aliases localcontext, Local, local):

remote_profile:#
type: dict, optional
argument path: pep723/machine[LocalContext]/remote_profile

The information used to maintain the local machine.

When context_type is set to HDFSContext (or its aliases hdfscontext, HDFS, hdfs):

remote_profile:#
type: dict, optional
argument path: pep723/machine[HDFSContext]/remote_profile

The information used to maintain the connection with remote machine. This field is empty for this context.

When context_type is set to SSHContext (or its aliases sshcontext, SSH, ssh):

remote_profile:#
type: dict
argument path: pep723/machine[SSHContext]/remote_profile

The information used to maintain the connection with remote machine.

hostname:#
type: str
argument path: pep723/machine[SSHContext]/remote_profile/hostname

hostname or ip of ssh connection.

username:#
type: str
argument path: pep723/machine[SSHContext]/remote_profile/username

username of target linux system

password:#
type: str, optional
argument path: pep723/machine[SSHContext]/remote_profile/password

(deprecated) password of linux system. Please use SSH keys instead to improve security.

port:#
type: int, optional, default: 22
argument path: pep723/machine[SSHContext]/remote_profile/port

ssh connection port.

key_filename:#
type: str | NoneType, optional, default: None
argument path: pep723/machine[SSHContext]/remote_profile/key_filename

key filename used by ssh connection. If left None, find key in ~/.ssh or use password for login

passphrase:#
type: str | NoneType, optional, default: None
argument path: pep723/machine[SSHContext]/remote_profile/passphrase

passphrase of key used by ssh connection

timeout:#
type: int, optional, default: 10
argument path: pep723/machine[SSHContext]/remote_profile/timeout

timeout of ssh connection

totp_secret:#
type: str | NoneType, optional, default: None
argument path: pep723/machine[SSHContext]/remote_profile/totp_secret

Time-based one time password secret. It should be a base32-encoded string extracted from the 2D code.

tar_compress:#
type: bool, optional, default: True
argument path: pep723/machine[SSHContext]/remote_profile/tar_compress

The archive will be compressed in upload and download if it is True. If not, compression will be skipped.

look_for_keys:#
type: bool, optional, default: True
argument path: pep723/machine[SSHContext]/remote_profile/look_for_keys

enable searching for discoverable private key files in ~/.ssh/

execute_command:#
type: str | NoneType, optional, default: None
argument path: pep723/machine[SSHContext]/remote_profile/execute_command

execute command after ssh connection is established.

proxy_command:#
type: str | NoneType, optional, default: None
argument path: pep723/machine[SSHContext]/remote_profile/proxy_command

ProxyCommand to use for SSH connection through intermediate servers.

When context_type is set to LazyLocalContext (or its aliases lazylocalcontext, LazyLocal, lazylocal):

remote_profile:#
type: dict, optional
argument path: pep723/machine[LazyLocalContext]/remote_profile

The information used to maintain the connection with remote machine. This field is empty for this context.

When context_type is set to BohriumContext (or its aliases bohriumcontext, Bohrium, bohrium, DpCloudServerContext, dpcloudservercontext, DpCloudServer, dpcloudserver, LebesgueContext, lebesguecontext, Lebesgue, lebesgue):

remote_profile:#
type: dict
argument path: pep723/machine[BohriumContext]/remote_profile

The information used to maintain the connection with remote machine.

email:#
type: str, optional
argument path: pep723/machine[BohriumContext]/remote_profile/email

Email

password:#
type: str, optional
argument path: pep723/machine[BohriumContext]/remote_profile/password

Password

program_id:#
type: int, alias: project_id
argument path: pep723/machine[BohriumContext]/remote_profile/program_id

Program ID

retry_count:#
type: NoneType | int, optional, default: 2
argument path: pep723/machine[BohriumContext]/remote_profile/retry_count

The retry count when a job is terminated

ignore_exit_code:#
type: bool, optional, default: True
argument path: pep723/machine[BohriumContext]/remote_profile/ignore_exit_code
The job state will be marked as finished if the exit code is non-zero when set to True. Otherwise,

the job state will be designated as terminated.

keep_backup:#
type: bool, optional
argument path: pep723/machine[BohriumContext]/remote_profile/keep_backup

keep download and upload zip

input_data:#
type: dict
argument path: pep723/machine[BohriumContext]/remote_profile/input_data

Configuration of job

resources:#
type: dict
argument path: pep723/resources

Resources configuration. See related documentation for details.

number_node:#
type: int, optional, default: 1
argument path: pep723/resources/number_node

The number of nodes required for each job.

cpu_per_node:#
type: int, optional, default: 1
argument path: pep723/resources/cpu_per_node

CPU numbers of each node assigned to each job.

gpu_per_node:#
type: int, optional, default: 0
argument path: pep723/resources/gpu_per_node

GPU numbers of each node assigned to each job.

queue_name:#
type: str, optional, default: (empty string)
argument path: pep723/resources/queue_name

The queue name of batch job scheduler system.

group_size:#
type: int
argument path: pep723/resources/group_size

The number of tasks in a job. 0 means infinity.

custom_flags:#
type: typing.List[str], optional
argument path: pep723/resources/custom_flags

The extra lines pass to job submitting script header

strategy:#
type: dict, optional
argument path: pep723/resources/strategy

strategies we use to generation job submitting scripts.

if_cuda_multi_devices:#
type: bool, optional, default: False
argument path: pep723/resources/strategy/if_cuda_multi_devices

If there are multiple nvidia GPUS on the node, and we want to assign the tasks to different GPUS.If true, dpdispatcher will manually export environment variable CUDA_VISIBLE_DEVICES to different task.Usually, this option will be used with Task.task_need_resources variable simultaneously.

ratio_unfinished:#
type: float, optional, default: 0.0
argument path: pep723/resources/strategy/ratio_unfinished

The ratio of tasks that can be unfinished.

customized_script_header_template_file:#
type: str, optional
argument path: pep723/resources/strategy/customized_script_header_template_file

The customized template file to generate job submitting script header, which overrides the default file.

para_deg:#
type: int, optional, default: 1
argument path: pep723/resources/para_deg

Decide how many tasks will be run in parallel.

source_list:#
type: typing.List[str], optional, default: []
argument path: pep723/resources/source_list

The env file to be sourced before the command execution.

module_purge:#
type: bool, optional, default: False
argument path: pep723/resources/module_purge

Remove all modules on HPC system before module load (module_list)

module_unload_list:#
type: typing.List[str], optional, default: []
argument path: pep723/resources/module_unload_list

The modules to be unloaded on HPC system before submitting jobs

module_list:#
type: typing.List[str], optional, default: []
argument path: pep723/resources/module_list

The modules to be loaded on HPC system before submitting jobs

envs:#
type: dict, optional, default: {}
argument path: pep723/resources/envs

The environment variables to be exported on before submitting jobs

prepend_script:#
type: typing.List[str], optional, default: []
argument path: pep723/resources/prepend_script

Optional script run before jobs submitted.

append_script:#
type: typing.List[str], optional, default: []
argument path: pep723/resources/append_script

Optional script run after jobs submitted.

wait_time:#
type: float | int, optional, default: 0
argument path: pep723/resources/wait_time

The waitting time in second after a single task submitted

kwargs:#
type: dict, optional
argument path: pep723/resources/kwargs

Vary by different machines.

batch_type:#
type: str, optional
argument path: pep723/resources/batch_type

Allow this key when strict checking.

task_list:#
type: list
argument path: pep723/task_list

List of tasks to execute.

This argument takes a list with each element containing the following:

command:#
type: str, optional, default: python
argument path: pep723/task_list/command

Python interpreter or launcher. No need to contain the Python script filename.

task_work_path:#
type: str
argument path: pep723/task_list/task_work_path

The dir where the command to be executed. Can be a glob pattern.

forward_files:#
type: typing.List[str], optional, default: []
argument path: pep723/task_list/forward_files

The files to be uploaded in task_work_path before the task exectued.

backward_files:#
type: typing.List[str], optional, default: []
argument path: pep723/task_list/backward_files

The files to be download to local_root in task_work_path after the task finished

outlog:#
type: str | NoneType, optional, default: log
argument path: pep723/task_list/outlog

The out log file name. redirect from stdout

errlog:#
type: str | NoneType, optional, default: err
argument path: pep723/task_list/errlog

The err log file name. redirect from stderr

Contents