dpdispatcher.machines package

Machines.

Submodules

dpdispatcher.machines.JH_UniScheduler module

Implement job submission through JH UniScheduler commands.

class dpdispatcher.machines.JH_UniScheduler.JH_UniScheduler(*args: Any, **kwargs: Any)

Bases: Machine

JH_UniScheduler batch.

Methods

arginfo()	Build the dargs schema for machine and context configuration.
bind_context(context)	Bind the execution context used for files and remote commands.
check_finish_tag(job)	Return whether the success marker for a job is present.
check_if_recover(submission)	Return whether serialized state exists for a remote submission.
default_resources(res)	Return backend defaults for an incomplete resource specification.
deserialize(machine_dict)	Reconstruct a machine from its serialized dictionary.
gen_command_env_cuda_devices(resources)	Assign a GPU to the next task when automatic GPU mapping is enabled.
gen_local_script(job)	Generate a local staging script for cloud backends.
gen_script(job)	Generate the complete scheduler submission script for a job.
gen_script_command(job)	Generate task commands, logging redirection, and completion tags.
gen_script_custom_flags_lines(job)	Render user-provided scheduler directives as script lines.
gen_script_end(job)	Generate job finalization, failure checks, and append commands.
gen_script_env(job)	Generate environment setup shared by every task in a job.
gen_script_header(job)	Generate scheduler directives and the script shebang for a job.
gen_script_run_command(job)	Return the command that sources the generated per-task script.
gen_script_wait(resources)	Generate synchronization commands for the configured parallelism.
get_exit_code(job)	Get exit code of the job.
get_job_error(job)	Return a text error diagnostic for a job, if available.
kill(job)	Kill the job.
load_from_dict(machine_dict[, allow_ref])	Load a Machine from a dict.
load_from_json(json_path)	Load a machine configuration from a JSON file.
load_from_yaml(yaml_path)	Load a machine configuration from a YAML file.
resources_arginfo()	Generate the resources arginfo.
resources_subfields()	Generate the resources subfields.
serialize([if_empty_remote_profile])	Return a normalized dictionary representation of the machine.
sub_script_cmd(res)	Return the legacy scheduler launch command for resources.
sub_script_head(res)	Return the legacy scheduler header for a resource specification.

check_status
do_submit

check_finish_tag(job: Job) → bool

Return whether the success marker for a job is present.

check_status(**kwargs: Any) → Any

Query the scheduler and return the current status of a job.

do_submit(**kwargs: Any) → Any

Submit a single job, assuming that no job is running there.

gen_script(job: Job) → str

Generate the complete scheduler submission script for a job.

gen_script_header(job: Job) → str

Generate scheduler directives and the script shebang for a job.

kill(job: Job) → None

Kill the job.

Parameters:

jobJob

job

classmethod resources_subfields() → list[Argument]

Generate the resources subfields.

Returns:

list[Argument]

resources subfields

dpdispatcher.machines.distributed_shell module

Implement Hadoop YARN DistributedShell submission.

class dpdispatcher.machines.distributed_shell.DistributedShell(*args: Any, **kwargs: Any)

Bases: Machine

Run archived DPDispatcher jobs in a YARN DistributedShell container.

Methods

arginfo()	Build the dargs schema for machine and context configuration.
bind_context(context)	Bind the execution context used for files and remote commands.
check_finish_tag(job)	Return whether the success marker for a job is present.
check_if_recover(submission)	Return whether serialized state exists for a remote submission.
check_status(job)	Query the scheduler and return the current status of a job.
default_resources(res)	Return backend defaults for an incomplete resource specification.
deserialize(machine_dict)	Reconstruct a machine from its serialized dictionary.
do_submit(job)	Submit th job to yarn using distributed shell.
gen_command_env_cuda_devices(resources)	Assign a GPU to the next task when automatic GPU mapping is enabled.
gen_local_script(job)	Generate a local staging script for cloud backends.
gen_script(job)	Generate the complete scheduler submission script for a job.
gen_script_command(job)	Generate task commands, logging redirection, and completion tags.
gen_script_custom_flags_lines(job)	Render user-provided scheduler directives as script lines.
gen_script_end(job)	Generate job finalization, failure checks, and append commands.
gen_script_env(job)	Generate environment setup shared by every task in a job.
gen_script_header(job)	Generate scheduler directives and the script shebang for a job.
gen_script_run_command(job)	Return the command that sources the generated per-task script.
gen_script_wait(resources)	Generate synchronization commands for the configured parallelism.
get_exit_code(job)	Get exit code of the job.
get_job_error(job)	Return a text error diagnostic for a job, if available.
kill(job)	Kill the job.
load_from_dict(machine_dict[, allow_ref])	Load a Machine from a dict.
load_from_json(json_path)	Load a machine configuration from a JSON file.
load_from_yaml(yaml_path)	Load a machine configuration from a YAML file.
resources_arginfo()	Generate the resources arginfo.
resources_subfields()	Generate the resources subfields.
serialize([if_empty_remote_profile])	Return a normalized dictionary representation of the machine.
sub_script_cmd(res)	Return the legacy scheduler launch command for resources.
sub_script_head(res)	Return the legacy scheduler header for a resource specification.

check_finish_tag(job: Job) → bool

Return whether the success marker for a job is present.

check_status(job: Job) → JobStatus

Query the scheduler and return the current status of a job.

do_submit(job: Job) → int

Submit th job to yarn using distributed shell.

Parameters:

jobJob class instance

job to be submitted

Returns:

job_id: string

submit process id

gen_script_end(job: Job) → str

Generate job finalization, failure checks, and append commands.

gen_script_env(job: Job) → str

Generate environment setup shared by every task in a job.

gen_script_header(job: Job) → str

Generate scheduler directives and the script shebang for a job.

dpdispatcher.machines.dp_cloud_server module

Implement the legacy Bohrium cloud-service batch backend.

class dpdispatcher.machines.dp_cloud_server.Bohrium(*args: Any, **kwargs: Any)

Bases: Machine

Submit jobs through the legacy Bohrium API and object storage.

Methods

arginfo()	Build the dargs schema for machine and context configuration.
bind_context(context)	Bind the execution context used for files and remote commands.
check_finish_tag(job)	Return whether the success marker for a job is present.
check_if_recover(submission)	Return whether serialized state exists for a remote submission.
check_status(job)	Query the scheduler and return the current status of a job.
default_resources(res)	Return backend defaults for an incomplete resource specification.
deserialize(machine_dict)	Reconstruct a machine from its serialized dictionary.
do_submit(job)	Submit a single job, assuming that no job is running there.
gen_command_env_cuda_devices(resources)	Assign a GPU to the next task when automatic GPU mapping is enabled.
gen_local_script(job)	Generate a local staging script for cloud backends.
gen_script(job)	Generate a job script with the common machine implementation.
gen_script_command(job)	Generate task commands, logging redirection, and completion tags.
gen_script_custom_flags_lines(job)	Render user-provided scheduler directives as script lines.
gen_script_end(job)	Generate job finalization, failure checks, and append commands.
gen_script_env(job)	Generate environment setup shared by every task in a job.
gen_script_header(job)	Generate scheduler directives and the script shebang for a job.
gen_script_run_command(job)	Return the command that sources the generated per-task script.
gen_script_wait(resources)	Generate synchronization commands for the configured parallelism.
get_exit_code(job)	Get exit code of the job.
get_job_error(job)	Retrieve diagnostics from the cloud result or job log.
kill(job)	Kill the job.
load_from_dict(machine_dict[, allow_ref])	Load a Machine from a dict.
load_from_json(json_path)	Load a machine configuration from a JSON file.
load_from_yaml(yaml_path)	Load a machine configuration from a YAML file.
resources_arginfo()	Generate the resources arginfo.
resources_subfields()	Generate the resources subfields.
serialize([if_empty_remote_profile])	Return a normalized dictionary representation of the machine.
sub_script_cmd(res)	Return the legacy scheduler launch command for resources.
sub_script_head(res)	Return the legacy scheduler header for a resource specification.

map_dp_job_state

alias: tuple[str, ...] = ('Lebesgue', 'DpCloudServer')

check_finish_tag(job: Job) → bool

Return whether the success marker for a job is present.

check_if_recover(submission: Submission) → bool

Return whether serialized state exists for a remote submission.

check_status(job: Job) → JobStatus

Query the scheduler and return the current status of a job.

do_submit(job: Job) → str

Submit a single job, assuming that no job is running there.

gen_local_script(job: Job) → str

Generate a local staging script for cloud backends.

Only cloud machine implementations support this operation; the base declaration keeps their context interface type-safe.

gen_script(job: Job) → str

Generate a job script with the common machine implementation.

gen_script_header(job: Job) → str

Generate scheduler directives and the script shebang for a job.

get_exit_code(job: Job) → int

Get exit code of the job.

Parameters:

jobJob

job

get_job_error(job: Job) → str | None

Retrieve diagnostics from the cloud result or job log.

kill(job: Job) → None

Kill the job.

Parameters:

jobJob

job

static map_dp_job_state(status: int | JobStatus, exit_code: int, ignore_exit_code: bool = True) → JobStatus

dpdispatcher.machines.dp_cloud_server.DpCloudServer

alias of Bohrium

dpdispatcher.machines.dp_cloud_server.Lebesgue

alias of Bohrium

dpdispatcher.machines.fugaku module

Implement Fujitsu Fugaku-compatible scheduler submission.

class dpdispatcher.machines.fugaku.Fugaku(*args: Any, **kwargs: Any)

Bases: Machine

Generate and submit jobs using Fujitsu’s pjsub interface.

Methods

arginfo()	Build the dargs schema for machine and context configuration.
bind_context(context)	Bind the execution context used for files and remote commands.
check_finish_tag(job)	Return whether the success marker for a job is present.
check_if_recover(submission)	Return whether serialized state exists for a remote submission.
check_status(job)	Query the scheduler and return the current status of a job.
default_resources(res)	Return backend defaults for an incomplete resource specification.
deserialize(machine_dict)	Reconstruct a machine from its serialized dictionary.
do_submit(job)	Submit a single job, assuming that no job is running there.
gen_command_env_cuda_devices(resources)	Assign a GPU to the next task when automatic GPU mapping is enabled.
gen_local_script(job)	Generate a local staging script for cloud backends.
gen_script(job)	Generate the complete scheduler submission script for a job.
gen_script_command(job)	Generate task commands, logging redirection, and completion tags.
gen_script_custom_flags_lines(job)	Render user-provided scheduler directives as script lines.
gen_script_end(job)	Generate job finalization, failure checks, and append commands.
gen_script_env(job)	Generate environment setup shared by every task in a job.
gen_script_header(job)	Generate scheduler directives and the script shebang for a job.
gen_script_run_command(job)	Return the command that sources the generated per-task script.
gen_script_wait(resources)	Generate synchronization commands for the configured parallelism.
get_exit_code(job)	Get exit code of the job.
get_job_error(job)	Return a text error diagnostic for a job, if available.
kill(job)	Kill the job.
load_from_dict(machine_dict[, allow_ref])	Load a Machine from a dict.
load_from_json(json_path)	Load a machine configuration from a JSON file.
load_from_yaml(yaml_path)	Load a machine configuration from a YAML file.
resources_arginfo()	Generate the resources arginfo.
resources_subfields()	Generate the resources subfields.
serialize([if_empty_remote_profile])	Return a normalized dictionary representation of the machine.
sub_script_cmd(res)	Return the legacy scheduler launch command for resources.
sub_script_head(res)	Return the legacy scheduler header for a resource specification.

check_finish_tag(job: Job) → bool

Return whether the success marker for a job is present.

check_status(job: Job) → JobStatus

Query the scheduler and return the current status of a job.

do_submit(job: Job) → str

Submit a single job, assuming that no job is running there.

gen_script(job: Job) → str

Generate the complete scheduler submission script for a job.

gen_script_header(job: Job) → str

Generate scheduler directives and the script shebang for a job.

dpdispatcher.machines.lsf module

Implement IBM Spectrum LSF scheduler submission.

class dpdispatcher.machines.lsf.LSF(*args: Any, **kwargs: Any)

Bases: Machine

LSF batch.

Methods

arginfo()	Build the dargs schema for machine and context configuration.
bind_context(context)	Bind the execution context used for files and remote commands.
check_finish_tag(job)	Return whether the success marker for a job is present.
check_if_recover(submission)	Return whether serialized state exists for a remote submission.
default_resources(res)	Return backend defaults for an incomplete resource specification.
deserialize(machine_dict)	Reconstruct a machine from its serialized dictionary.
gen_command_env_cuda_devices(resources)	Assign a GPU to the next task when automatic GPU mapping is enabled.
gen_local_script(job)	Generate a local staging script for cloud backends.
gen_script(job)	Generate the complete scheduler submission script for a job.
gen_script_command(job)	Generate task commands, logging redirection, and completion tags.
gen_script_custom_flags_lines(job)	Render user-provided scheduler directives as script lines.
gen_script_end(job)	Generate job finalization, failure checks, and append commands.
gen_script_env(job)	Generate environment setup shared by every task in a job.
gen_script_header(job)	Generate scheduler directives and the script shebang for a job.
gen_script_run_command(job)	Return the command that sources the generated per-task script.
gen_script_wait(resources)	Generate synchronization commands for the configured parallelism.
get_exit_code(job)	Get exit code of the job.
get_job_error(job)	Return a text error diagnostic for a job, if available.
kill(job)	Kill the job.
load_from_dict(machine_dict[, allow_ref])	Load a Machine from a dict.
load_from_json(json_path)	Load a machine configuration from a JSON file.
load_from_yaml(yaml_path)	Load a machine configuration from a YAML file.
resources_arginfo()	Generate the resources arginfo.
resources_subfields()	Generate the resources subfields.
serialize([if_empty_remote_profile])	Return a normalized dictionary representation of the machine.
sub_script_cmd(res)	Return the legacy scheduler launch command for resources.
sub_script_head(res)	Return the legacy scheduler header for a resource specification.

check_status
do_submit

check_finish_tag(job: Job) → bool

Return whether the success marker for a job is present.

check_status(**kwargs: Any) → Any

Query the scheduler and return the current status of a job.

do_submit(**kwargs: Any) → Any

Submit a single job, assuming that no job is running there.

gen_script(job: Job) → str

Generate the complete scheduler submission script for a job.

gen_script_header(job: Job) → str

Generate scheduler directives and the script shebang for a job.

kill(job: Job) → None

Kill the job.

Parameters:

jobJob

job

classmethod resources_subfields() → list[Argument]

Generate the resources subfields.

Returns:

list[Argument]

resources subfields

sub_script_cmd(res: Resources) → str

Return the legacy scheduler launch command for resources.

sub_script_head(res: Resources) → str

Return the legacy scheduler header for a resource specification.

dpdispatcher.machines.openapi module

Implement the SDK-based Bohrium OpenAPI batch backend.

class dpdispatcher.machines.openapi.OpenAPI(*args: Any, **kwargs: Any)

Bases: Machine

Submit and monitor jobs through the Bohrium SDK OpenAPI.

Methods

arginfo()	Build the dargs schema for machine and context configuration.
bind_context(context)	Bind the execution context used for files and remote commands.
check_finish_tag(job)	Return whether the success marker for a job is present.
check_if_recover(submission)	Return whether serialized state exists for a remote submission.
check_status(job)	Query the scheduler and return the current status of a job.
default_resources(res)	Return backend defaults for an incomplete resource specification.
deserialize(machine_dict)	Reconstruct a machine from its serialized dictionary.
do_submit(job)	Submit a single job, assuming that no job is running there.
gen_command_env_cuda_devices(resources)	Assign a GPU to the next task when automatic GPU mapping is enabled.
gen_local_script(job)	Generate a local staging script for cloud backends.
gen_script(job)	Generate the complete scheduler submission script for a job.
gen_script_command(job)	Generate task commands, logging redirection, and completion tags.
gen_script_custom_flags_lines(job)	Render user-provided scheduler directives as script lines.
gen_script_end(job)	Generate job finalization, failure checks, and append commands.
gen_script_env(job)	Generate environment setup shared by every task in a job.
gen_script_header(job)	Generate scheduler directives and the script shebang for a job.
gen_script_run_command(job)	Return the command that sources the generated per-task script.
gen_script_wait(resources)	Generate synchronization commands for the configured parallelism.
get_exit_code(job)	Get exit code of the job.
get_job_error(job)	Retrieve diagnostics from the cloud result or job log.
kill(job)	Kill the job.
load_from_dict(machine_dict[, allow_ref])	Load a Machine from a dict.
load_from_json(json_path)	Load a machine configuration from a JSON file.
load_from_yaml(yaml_path)	Load a machine configuration from a YAML file.
resources_arginfo()	Generate the resources arginfo.
resources_subfields()	Generate the resources subfields.
serialize([if_empty_remote_profile])	Return a normalized dictionary representation of the machine.
sub_script_cmd(res)	Return the legacy scheduler launch command for resources.
sub_script_head(res)	Return the legacy scheduler header for a resource specification.

map_dp_job_state

check_finish_tag(job: SubmissionJob) → bool

Return whether the success marker for a job is present.

check_if_recover(submission: Submission) → bool

Return whether serialized state exists for a remote submission.

check_status(job: SubmissionJob) → JobStatus

Query the scheduler and return the current status of a job.

do_submit(job: SubmissionJob) → int

Submit a single job, assuming that no job is running there.

gen_local_script(job: SubmissionJob) → str

Generate a local staging script for cloud backends.

Only cloud machine implementations support this operation; the base declaration keeps their context interface type-safe.

gen_script(job: SubmissionJob) → str

Generate the complete scheduler submission script for a job.

gen_script_header(job: SubmissionJob) → str

Generate scheduler directives and the script shebang for a job.

get_exit_code(job: SubmissionJob) → int

Get exit code of the job.

Parameters:

jobJob

job

Returns:

int

exit code

get_job_error(job: SubmissionJob) → str | None

Retrieve diagnostics from the cloud result or job log.

kill(job: SubmissionJob) → None

Kill the job.

Parameters:

jobJob

job

static map_dp_job_state(status: int | JobStatus, exit_code: int, ignore_exit_code: bool = True) → JobStatus

dpdispatcher.machines.openapi.unzip_file(zip_file: str, out_dir: str = './') → None

Extract a ZIP archive into a local directory.

dpdispatcher.machines.pbs module

Implement PBS-family schedulers, including Torque and SGE variants.

class dpdispatcher.machines.pbs.PBS(*args: Any, **kwargs: Any)

Bases: Machine

Submit and monitor jobs through OpenPBS or PBS Professional.

Methods

arginfo()	Build the dargs schema for machine and context configuration.
bind_context(context)	Bind the execution context used for files and remote commands.
check_finish_tag(job)	Return whether the success marker for a job is present.
check_if_recover(submission)	Return whether serialized state exists for a remote submission.
check_status(job)	Query the scheduler and return the current status of a job.
default_resources(res)	Return backend defaults for an incomplete resource specification.
deserialize(machine_dict)	Reconstruct a machine from its serialized dictionary.
do_submit(job)	Submit a single job, assuming that no job is running there.
gen_command_env_cuda_devices(resources)	Assign a GPU to the next task when automatic GPU mapping is enabled.
gen_local_script(job)	Generate a local staging script for cloud backends.
gen_script(job)	Generate the complete scheduler submission script for a job.
gen_script_command(job)	Generate task commands, logging redirection, and completion tags.
gen_script_custom_flags_lines(job)	Render user-provided scheduler directives as script lines.
gen_script_end(job)	Generate job finalization, failure checks, and append commands.
gen_script_env(job)	Generate environment setup shared by every task in a job.
gen_script_header(job)	Generate scheduler directives and the script shebang for a job.
gen_script_run_command(job)	Return the command that sources the generated per-task script.
gen_script_wait(resources)	Generate synchronization commands for the configured parallelism.
get_exit_code(job)	Get exit code of the job.
get_job_error(job)	Return a text error diagnostic for a job, if available.
kill(job)	Kill the job.
load_from_dict(machine_dict[, allow_ref])	Load a Machine from a dict.
load_from_json(json_path)	Load a machine configuration from a JSON file.
load_from_yaml(yaml_path)	Load a machine configuration from a YAML file.
resources_arginfo()	Generate the resources arginfo.
resources_subfields()	Generate the resources subfields.
serialize([if_empty_remote_profile])	Return a normalized dictionary representation of the machine.
sub_script_cmd(res)	Return the legacy scheduler launch command for resources.
sub_script_head(res)	Return the legacy scheduler header for a resource specification.

check_finish_tag(job: Job) → bool

Return whether the success marker for a job is present.

check_status(job: Job) → JobStatus

Query the scheduler and return the current status of a job.

do_submit(job: Job) → str

Submit a single job, assuming that no job is running there.

gen_script(job: Job) → str

Generate the complete scheduler submission script for a job.

gen_script_header(job: Job) → str

Generate scheduler directives and the script shebang for a job.

kill(job: Job) → None

Kill the job.

Parameters:

jobJob

job

class dpdispatcher.machines.pbs.SGE(*args: Any, **kwargs: Any)

Bases: PBS

Adapt PBS-style scripts and commands for Sun Grid Engine.

Methods

arginfo()	Build the dargs schema for machine and context configuration.
bind_context(context)	Bind the execution context used for files and remote commands.
check_finish_tag(job)	Return whether the success marker for a job is present.
check_if_recover(submission)	Return whether serialized state exists for a remote submission.
check_status(job)	Query the scheduler and return the current status of a job.
default_resources(res)	Return backend defaults for an incomplete resource specification.
deserialize(machine_dict)	Reconstruct a machine from its serialized dictionary.
do_submit(job)	Submit a single job, assuming that no job is running there.
gen_command_env_cuda_devices(resources)	Assign a GPU to the next task when automatic GPU mapping is enabled.
gen_local_script(job)	Generate a local staging script for cloud backends.
gen_script(job)	Generate the complete scheduler submission script for a job.
gen_script_command(job)	Generate task commands, logging redirection, and completion tags.
gen_script_custom_flags_lines(job)	Render user-provided scheduler directives as script lines.
gen_script_end(job)	Generate job finalization, failure checks, and append commands.
gen_script_env(job)	Generate environment setup shared by every task in a job.
gen_script_header(job)	Generate scheduler directives and the script shebang for a job.
gen_script_run_command(job)	Return the command that sources the generated per-task script.
gen_script_wait(resources)	Generate synchronization commands for the configured parallelism.
get_exit_code(job)	Get exit code of the job.
get_job_error(job)	Return a text error diagnostic for a job, if available.
kill(job)	Kill the job.
load_from_dict(machine_dict[, allow_ref])	Load a Machine from a dict.
load_from_json(json_path)	Load a machine configuration from a JSON file.
load_from_yaml(yaml_path)	Load a machine configuration from a YAML file.
resources_arginfo()	Generate the resources arginfo.
resources_subfields()	Generate the resources subfields.
serialize([if_empty_remote_profile])	Return a normalized dictionary representation of the machine.
sub_script_cmd(res)	Return the legacy scheduler launch command for resources.
sub_script_head(res)	Return the legacy scheduler header for a resource specification.

check_finish_tag(job: Job) → bool

Return whether the success marker for a job is present.

check_status(job: Job) → JobStatus

Query the scheduler and return the current status of a job.

do_submit(job: Job) → str

Submit a single job, assuming that no job is running there.

gen_script_header(job: Job) → str

Generate scheduler directives and the script shebang for a job.

classmethod resources_subfields() → list[Argument]

Generate the resources subfields.

pe_name : str

The parallel environment name of SGE.

Returns:

list[Argument]

resources subfields

class dpdispatcher.machines.pbs.Torque(*args: Any, **kwargs: Any)

Bases: PBS

Adapt PBS submission and status parsing for the Torque scheduler.

Methods

arginfo()	Build the dargs schema for machine and context configuration.
bind_context(context)	Bind the execution context used for files and remote commands.
check_finish_tag(job)	Return whether the success marker for a job is present.
check_if_recover(submission)	Return whether serialized state exists for a remote submission.
check_status(job)	Query the scheduler and return the current status of a job.
default_resources(res)	Return backend defaults for an incomplete resource specification.
deserialize(machine_dict)	Reconstruct a machine from its serialized dictionary.
do_submit(job)	Submit a single job, assuming that no job is running there.
gen_command_env_cuda_devices(resources)	Assign a GPU to the next task when automatic GPU mapping is enabled.
gen_local_script(job)	Generate a local staging script for cloud backends.
gen_script(job)	Generate the complete scheduler submission script for a job.
gen_script_command(job)	Generate task commands, logging redirection, and completion tags.
gen_script_custom_flags_lines(job)	Render user-provided scheduler directives as script lines.
gen_script_end(job)	Generate job finalization, failure checks, and append commands.
gen_script_env(job)	Generate environment setup shared by every task in a job.
gen_script_header(job)	Generate scheduler directives and the script shebang for a job.
gen_script_run_command(job)	Return the command that sources the generated per-task script.
gen_script_wait(resources)	Generate synchronization commands for the configured parallelism.
get_exit_code(job)	Get exit code of the job.
get_job_error(job)	Return a text error diagnostic for a job, if available.
kill(job)	Kill the job.
load_from_dict(machine_dict[, allow_ref])	Load a Machine from a dict.
load_from_json(json_path)	Load a machine configuration from a JSON file.
load_from_yaml(yaml_path)	Load a machine configuration from a YAML file.
resources_arginfo()	Generate the resources arginfo.
resources_subfields()	Generate the resources subfields.
serialize([if_empty_remote_profile])	Return a normalized dictionary representation of the machine.
sub_script_cmd(res)	Return the legacy scheduler launch command for resources.
sub_script_head(res)	Return the legacy scheduler header for a resource specification.

check_status(job: Job) → JobStatus

Query the scheduler and return the current status of a job.

gen_script_header(job: Job) → str

Generate scheduler directives and the script shebang for a job.

dpdispatcher.machines.shell module

Implement local or remote background execution through a shell.

class dpdispatcher.machines.shell.Shell(*args: Any, **kwargs: Any)

Bases: Machine

Run generated job scripts as background shell processes.

Methods

arginfo()	Build the dargs schema for machine and context configuration.
bind_context(context)	Bind the execution context used for files and remote commands.
check_finish_tag(job)	Return whether the success marker for a job is present.
check_if_recover(submission)	Return whether serialized state exists for a remote submission.
check_status(job)	Query the scheduler and return the current status of a job.
default_resources(res)	Return backend defaults for an incomplete resource specification.
deserialize(machine_dict)	Reconstruct a machine from its serialized dictionary.
do_submit(job)	Submit a single job, assuming that no job is running there.
gen_command_env_cuda_devices(resources)	Assign a GPU to the next task when automatic GPU mapping is enabled.
gen_local_script(job)	Generate a local staging script for cloud backends.
gen_script(job)	Generate the complete scheduler submission script for a job.
gen_script_command(job)	Generate task commands, logging redirection, and completion tags.
gen_script_custom_flags_lines(job)	Render user-provided scheduler directives as script lines.
gen_script_end(job)	Generate job finalization, failure checks, and append commands.
gen_script_env(job)	Generate environment setup shared by every task in a job.
gen_script_header(job)	Generate scheduler directives and the script shebang for a job.
gen_script_run_command(job)	Return the command that sources the generated per-task script.
gen_script_wait(resources)	Generate synchronization commands for the configured parallelism.
get_exit_code(job)	Get exit code of the job.
get_job_error(job)	Return a text error diagnostic for a job, if available.
kill(job)	Kill the job.
load_from_dict(machine_dict[, allow_ref])	Load a Machine from a dict.
load_from_json(json_path)	Load a machine configuration from a JSON file.
load_from_yaml(yaml_path)	Load a machine configuration from a YAML file.
resources_arginfo()	Generate the resources arginfo.
resources_subfields()	Generate the resources subfields.
serialize([if_empty_remote_profile])	Return a normalized dictionary representation of the machine.
sub_script_cmd(res)	Return the legacy scheduler launch command for resources.
sub_script_head(res)	Return the legacy scheduler header for a resource specification.

check_finish_tag(job: Job) → bool

Return whether the success marker for a job is present.

check_status(job: Job) → JobStatus

Query the scheduler and return the current status of a job.

do_submit(job: Job) → int

Submit a single job, assuming that no job is running there.

gen_script(job: Job) → str

Generate the complete scheduler submission script for a job.

gen_script_header(job: Job) → str

Generate scheduler directives and the script shebang for a job.

kill(job: Job) → None

Kill the job.

Parameters:

jobJob

job

dpdispatcher.machines.slurm module

Implement Slurm scheduler submission and Slurm job arrays.

class dpdispatcher.machines.slurm.Slurm(*args: Any, **kwargs: Any)

Bases: Machine

Submit and monitor jobs through Slurm command-line tools.

Methods

arginfo()	Build the dargs schema for machine and context configuration.
bind_context(context)	Bind the execution context used for files and remote commands.
check_finish_tag(job)	Return whether the success marker for a job is present.
check_if_recover(submission)	Return whether serialized state exists for a remote submission.
default_resources(res)	Return backend defaults for an incomplete resource specification.
deserialize(machine_dict)	Reconstruct a machine from its serialized dictionary.
gen_command_env_cuda_devices(resources)	Assign a GPU to the next task when automatic GPU mapping is enabled.
gen_local_script(job)	Generate a local staging script for cloud backends.
gen_script(job)	Generate the complete scheduler submission script for a job.
gen_script_command(job)	Generate task commands, logging redirection, and completion tags.
gen_script_custom_flags_lines(job)	Render user-provided scheduler directives as script lines.
gen_script_end(job)	Generate job finalization, failure checks, and append commands.
gen_script_env(job)	Generate environment setup shared by every task in a job.
gen_script_header(job)	Generate scheduler directives and the script shebang for a job.
gen_script_run_command(job)	Return the command that sources the generated per-task script.
gen_script_wait(resources)	Generate synchronization commands for the configured parallelism.
get_exit_code(job)	Get exit code of the job.
get_job_error(job)	Return a text error diagnostic for a job, if available.
kill(job)	Kill the job.
load_from_dict(machine_dict[, allow_ref])	Load a Machine from a dict.
load_from_json(json_path)	Load a machine configuration from a JSON file.
load_from_yaml(yaml_path)	Load a machine configuration from a YAML file.
resources_arginfo()	Generate the resources arginfo.
resources_subfields()	Generate the resources subfields.
serialize([if_empty_remote_profile])	Return a normalized dictionary representation of the machine.
sub_script_cmd(res)	Return the legacy scheduler launch command for resources.
sub_script_head(res)	Return the legacy scheduler header for a resource specification.

check_status
do_submit

check_finish_tag(job: Job) → bool

Return whether the success marker for a job is present.

check_status(**kwargs: Any) → Any

Query the scheduler and return the current status of a job.

do_submit(**kwargs: Any) → Any

Submit a single job, assuming that no job is running there.

gen_script(job: Job) → str

Generate the complete scheduler submission script for a job.

gen_script_header(job: Job) → str

Generate scheduler directives and the script shebang for a job.

kill(job: Job) → None

Kill the job.

Parameters:

jobJob

job

classmethod resources_subfields() → list[Argument]

Generate the resources subfields.

Returns:

list[Argument]

resources subfields

class dpdispatcher.machines.slurm.SlurmJobArray(*args: Any, **kwargs: Any)

Bases: Slurm

Slurm with job array enabled for multiple tasks in a job.

Methods

arginfo()	Build the dargs schema for machine and context configuration.
bind_context(context)	Bind the execution context used for files and remote commands.
check_finish_tag(job)	Return whether the success marker for a job is present.
check_if_recover(submission)	Return whether serialized state exists for a remote submission.
default_resources(res)	Return backend defaults for an incomplete resource specification.
deserialize(machine_dict)	Reconstruct a machine from its serialized dictionary.
gen_command_env_cuda_devices(resources)	Assign a GPU to the next task when automatic GPU mapping is enabled.
gen_local_script(job)	Generate a local staging script for cloud backends.
gen_script(job)	Generate the complete scheduler submission script for a job.
gen_script_command(job)	Generate a Slurm array command without mutating job resources.
gen_script_custom_flags_lines(job)	Render user-provided scheduler directives as script lines.
gen_script_end(job)	Generate job finalization, failure checks, and append commands.
gen_script_env(job)	Generate environment setup shared by every task in a job.
gen_script_header(job)	Generate scheduler directives and the script shebang for a job.
gen_script_run_command(job)	Return the command that sources the generated per-task script.
gen_script_wait(resources)	Generate synchronization commands for the configured parallelism.
get_exit_code(job)	Get exit code of the job.
get_job_error(job)	Return a text error diagnostic for a job, if available.
kill(job)	Kill the job.
load_from_dict(machine_dict[, allow_ref])	Load a Machine from a dict.
load_from_json(json_path)	Load a machine configuration from a JSON file.
load_from_yaml(yaml_path)	Load a machine configuration from a YAML file.
resources_arginfo()	Generate the resources arginfo.
resources_subfields()	Generate the resources subfields.
serialize([if_empty_remote_profile])	Return a normalized dictionary representation of the machine.
sub_script_cmd(res)	Return the legacy scheduler launch command for resources.
sub_script_head(res)	Return the legacy scheduler header for a resource specification.

check_status
do_submit

check_finish_tag(job: Job) → bool

Return whether the success marker for a job is present.

check_status(**kwargs: Any) → Any

Query the scheduler and return the current status of a job.

gen_script_command(job: Job) → str

Generate a Slurm array command without mutating job resources.

gen_script_end(job: Job) → str

Generate job finalization, failure checks, and append commands.

gen_script_header(job: Job) → str

Generate scheduler directives and the script shebang for a job.

classmethod resources_subfields() → list[Argument]

Generate the resources subfields.

Returns:

list[Argument]

resources subfields