Supported contexts

Supported contexts#

Context is the way to connect to the remote server. One needs to set context_type to one of the following values:

LazyLocal#

context_type: LazyLocal

LazyLocal directly runs jobs in the local server and local directory.

Since bash -l is used in the shebang line of the submission scripts, the login shell startup files will be executed, potentially overriding the current environment variables. Therefore, it’s advisable to explicitly set the environment variables using envs or source_list.

Local#

context_type: Local

Local runs jobs in the local server, but in a different directory. Files will be symlinked to the remote directory before jobs start and copied back after jobs finish. If the local directory is not accessible with the batch system, turn off symlink, and then files on the local directory will be copied to the remote directory.

Since bash -l is used in the shebang line of the submission scripts, the login shell startup files will be executed, potentially overriding the current environment variables. Therefore, it’s advisable to explicitly set the environment variables using envs or source_list.

SSH#

context_type: SSH

SSH runs jobs in a remote server. Files will be copied to the remote directory via SSH channels before jobs start and copied back after jobs finish. To use SSH, provide the connection parameters in remote_profile, including username and hostname.

By default, DPDispatcher creates only the final remote_root component and requires its parent directory to already exist. To recursively create missing parent directories, set create_remote_root to true.

Large backward-file archives can be transferred in smaller pieces by setting archive_chunk_size to a positive byte count. For example, 104857600 limits each temporary transfer to 100 MiB. The remote host must provide the split command. The default value 0 keeps the single-archive transfer behavior.

It’s suggested to generate SSH keys and transfer the public key to the remote server in advance, which is more secure than password authentication.

Note that SSH context is non-login, so bash_profile files will not be executed outside the submission script.

SSH Jump Host (Bastion Server)#

For connecting to internal servers through a jump host (bastion server), SSH context supports jump host configuration. This allows connecting to internal servers that are not directly accessible from the internet.

Specify the ProxyCommand directly using proxy_command:

{
  "context_type": "SSHContext",
  "remote_profile": {
    "hostname": "internal-server.company.com",
    "username": "user",
    "key_filename": "/path/to/internal_key",
    "proxy_command": "ssh -W internal-server.company.com:22 -i /path/to/jump_key jumpuser@bastion.company.com"
  }
}

Note: Unlike OpenSSH, Paramiko’s ProxyCommand requires explicit hostname and port instead of %h:%p placeholders. The proxy command must specify the actual target hostname and port that matches the hostname and port in the remote profile.

This configuration establishes the connection path: Local → Jump Host → Target Server.

Bohrium#

context_type: Bohrium

Bohrium is the cloud platform for scientific computing. Read Bohrium documentation for details. To use Bohrium, one needs to provide necessary parameters in remote_profile.

HDFS#

context_type: HDFS

The Hadoop Distributed File System (HDFS) is a distributed file system. Read Support DPDispatcher on Yarn for details.

OpenAPI#

context_type: OpenAPI

OpenAPI submits jobs to Bohrium using an AccessKey instead of a username and password. Read the Bohrium documentation for details. To use OpenAPI, one needs to provide necessary parameters in remote_profile.

Contents