Machine parameters

machine:

type: dict

argument path: machine

batch_type:

type: str

argument path: machine/batch_type

The batch job system type. Option: PBS, Shell, DistributedShell, SlurmJobArray, Lebesgue, Slurm, DpCloudServer, Torque, LSF

local_root:

type: str

argument path: machine/local_root

The dir where the tasks and relating files locate. Typically the project dir.

remote_root:

type: str, optional

argument path: machine/remote_root

The dir where the tasks are executed on the remote machine. Only needed when context is not lazy-local.

clean_asynchronously:

type: bool, optional, default: False

argument path: machine/clean_asynchronously

Clean the remote directory asynchronously after the job finishes.

Depending on the value of context_type, different sub args are accepted.

context_type:

type: str (flag key)

argument path: machine/context_type

possible choices: SSHContext, DpCloudServerContext, HDFSContext, LebesgueContext, LazyLocalContext, LocalContext

The connection used to remote machine. Option: SSHContext, HDFSContext, DpCloudServerContext, LazyLocalContext, LocalContext, LebesgueContext

When context_type is set to SSHContext (or its aliases sshcontext, SSH, ssh):

remote_profile:

type: dict

argument path: machine[SSHContext]/remote_profile

The information used to maintain the connection with remote machine.

hostname:

type: str

argument path: machine[SSHContext]/remote_profile/hostname

hostname or ip of ssh connection.

username:

type: str

argument path: machine[SSHContext]/remote_profile/username

username of target linux system

password:

type: str, optional

argument path: machine[SSHContext]/remote_profile/password

password of linux system

port:

type: int, optional, default: 22

argument path: machine[SSHContext]/remote_profile/port

ssh connection port.

key_filename:

type: NoneType | str, optional, default: None

argument path: machine[SSHContext]/remote_profile/key_filename

key filename used by ssh connection. If left None, find key in ~/.ssh or use password for login

passphrase:

type: NoneType | str, optional, default: None

argument path: machine[SSHContext]/remote_profile/passphrase

passphrase of key used by ssh connection

timeout:

type: int, optional, default: 10

argument path: machine[SSHContext]/remote_profile/timeout

timeout of ssh connection

totp_secret:

type: NoneType | str, optional, default: None

argument path: machine[SSHContext]/remote_profile/totp_secret

Time-based one time password secret. It should be a base32-encoded string extracted from the 2D code.

When context_type is set to DpCloudServerContext (or its aliases dpcloudservercontext, DpCloudServer, dpcloudserver):

remote_profile:

type: dict

argument path: machine[DpCloudServerContext]/remote_profile

The information used to maintain the connection with remote machine.

email:

type: str

argument path: machine[DpCloudServerContext]/remote_profile/email

Email

password:

type: str

argument path: machine[DpCloudServerContext]/remote_profile/password

Password

program_id:

type: int

argument path: machine[DpCloudServerContext]/remote_profile/program_id

Program ID

input_data:

type: dict

argument path: machine[DpCloudServerContext]/remote_profile/input_data

Configuration of job

job_name:

type: str, optional

argument path: machine[DpCloudServerContext]/remote_profile/input_data/job_name

Job name

image_name:

type: str, optional

argument path: machine[DpCloudServerContext]/remote_profile/input_data/image_name

Name of the image which run the job, optional when platform is not ali/oss.

disk_size:

type: str, optional

argument path: machine[DpCloudServerContext]/remote_profile/input_data/disk_size

disk size (GB), optional when platform is not ali/oss.

scass_type:

type: str

argument path: machine[DpCloudServerContext]/remote_profile/input_data/scass_type

machine configuration.

platform:

type: str

argument path: machine[DpCloudServerContext]/remote_profile/input_data/platform

Job run in which platform.

log_file:

type: str, optional

argument path: machine[DpCloudServerContext]/remote_profile/input_data/log_file

location of log file.

checkpoint_files:

type: list | str, optional

argument path: machine[DpCloudServerContext]/remote_profile/input_data/checkpoint_files

location of checkpoint files when it is list type. record file changes when it is string value ‘sync_files’

checkpoint_time:

type: int, optional, default: 15

argument path: machine[DpCloudServerContext]/remote_profile/input_data/checkpoint_time

interval of checkpoint data been stored minimum 15.

backward_files:

type: list, optional

argument path: machine[DpCloudServerContext]/remote_profile/input_data/backward_files

which files to be uploaded to remote resources. Upload all the files when it is None or empty.

When context_type is set to HDFSContext (or its aliases hdfscontext, HDFS, hdfs):

remote_profile:

type: dict, optional

argument path: machine[HDFSContext]/remote_profile

The information used to maintain the connection with remote machine. This field is empty for this context.

When context_type is set to LebesgueContext (or its aliases lebesguecontext, Lebesgue, lebesgue):

remote_profile:

type: dict

argument path: machine[LebesgueContext]/remote_profile

The information used to maintain the connection with remote machine.

email:

type: str

argument path: machine[LebesgueContext]/remote_profile/email

Email

password:

type: str

argument path: machine[LebesgueContext]/remote_profile/password

Password

program_id:

type: int

argument path: machine[LebesgueContext]/remote_profile/program_id

Program ID

input_data:

type: dict

argument path: machine[LebesgueContext]/remote_profile/input_data

Configuration of job

job_name:

type: str, optional

argument path: machine[LebesgueContext]/remote_profile/input_data/job_name

Job name

image_name:

type: str, optional

argument path: machine[LebesgueContext]/remote_profile/input_data/image_name

Name of the image which run the job, optional when platform is not ali/oss.

disk_size:

type: str, optional

argument path: machine[LebesgueContext]/remote_profile/input_data/disk_size

disk size (GB), optional when platform is not ali/oss.

scass_type:

type: str

argument path: machine[LebesgueContext]/remote_profile/input_data/scass_type

machine configuration.

platform:

type: str

argument path: machine[LebesgueContext]/remote_profile/input_data/platform

Job run in which platform.

log_file:

type: str, optional

argument path: machine[LebesgueContext]/remote_profile/input_data/log_file

location of log file.

checkpoint_files:

type: list | str, optional

argument path: machine[LebesgueContext]/remote_profile/input_data/checkpoint_files

location of checkpoint files when it is list type. record file changes when it is string value ‘sync_files’

checkpoint_time:

type: int, optional, default: 15

argument path: machine[LebesgueContext]/remote_profile/input_data/checkpoint_time

interval of checkpoint data been stored minimum 15.

backward_files:

type: list, optional

argument path: machine[LebesgueContext]/remote_profile/input_data/backward_files

which files to be uploaded to remote resources. Upload all the files when it is None or empty.

When context_type is set to LazyLocalContext (or its aliases lazylocalcontext, LazyLocal, lazylocal):

remote_profile:

type: dict, optional

argument path: machine[LazyLocalContext]/remote_profile

The information used to maintain the connection with remote machine. This field is empty for this context.

When context_type is set to LocalContext (or its aliases localcontext, Local, local):

remote_profile:

type: dict, optional

argument path: machine[LocalContext]/remote_profile

The information used to maintain the connection with remote machine. This field is empty for this context.