dpgen2.utils package

Submodules

dpgen2.utils.binary_file_input module

Binary file inputs

class dpgen2.utils.binary_file_input.BinaryFileInput(path: Union[str, Path], ext: Optional[str] = None)

Bases: object

Methods

save_as_file

save_as_file(path: Union[str, Path]) → None

dpgen2.utils.bohrium_config module

dpgen2.utils.bohrium_config.bohrium_config_from_dict(bohrium_config)

dpgen2.utils.chdir module

dpgen2.utils.chdir.chdir(path_key: str)

Returns a decorator that can change the current working path.

Parameters:

path_keystr

key to OPIO

Examples

>>> class SomeOP(OP):
...     @chdir("path")
...     def execute(self, ip: OPIO):
...         do_something()

dpgen2.utils.chdir.set_directory(path: Path)

Sets the current working path within the context.

Parameters:

pathPath

The path to the cwd

Yields:

None

Examples

>>> with set_directory("some_path"):
...    do_something()

dpgen2.utils.dflow_config module

dpgen2.utils.dflow_config.dflow_config(config_data)

set the dflow config by config_data

the keys starting with “s3_” will be treated as s3_config keys, other keys are treated as config keys.

dpgen2.utils.dflow_config.dflow_config_lower(dflow_config)

dpgen2.utils.dflow_config.dflow_s3_config(config_data)

set the s3 config by config_data

dpgen2.utils.dflow_config.dflow_s3_config_lower(dflow_s3_config_data)

dpgen2.utils.dflow_config.workflow_config_from_dict(wf_config)

dpgen2.utils.dflow_query module

dpgen2.utils.dflow_query.find_slice_ranges(keys: List[str], sliced_subkey: str)

find range of sliced OPs that matches the pattern ‘iter-[0-9]*–{sliced_subkey}-[0-9]*’

dpgen2.utils.dflow_query.get_all_schedulers(wf: Any, keys: List[str])

get the output Scheduler of the all the iterations

dpgen2.utils.dflow_query.get_iteration(key: str)

dpgen2.utils.dflow_query.get_last_iteration(keys: List[str])

get the index of the last iteraction from a list of step keys.

dpgen2.utils.dflow_query.get_last_scheduler(wf: Any, keys: List[str])

get the output Scheduler of the last successful iteration

dpgen2.utils.dflow_query.get_subkey(key: str, idx: int = -1)

dpgen2.utils.dflow_query.matched_step_key(all_keys: List[str], step_keys: Optional[List[str]] = None)

returns the keys in all_keys that matches any of the step_keys

dpgen2.utils.dflow_query.print_keys_in_nice_format(keys: List[str], sliced_subkey: List[str], idx_fmt_len: int = 8)

dpgen2.utils.dflow_query.sort_slice_ops(keys: List[str], sliced_subkey: List[str])

sort the keys of the sliced ops. the keys of the sliced ops contains sliced_subkey

dpgen2.utils.download_dpgen2_artifacts module

class dpgen2.utils.download_dpgen2_artifacts.DownloadDefinition

Bases: object

Methods

add_def
add_input
add_output

add_def(tdict, key, suffix=None)

add_input(input_key, suffix=None)

add_output(output_key, suffix=None)

dpgen2.utils.download_dpgen2_artifacts.download_dpgen2_artifacts(wf: Workflow, key: str, prefix: Optional[str] = None, chk_pnt: bool = False)

download the artifacts of a step. the key should be of format ‘iter-xxxxxx–subkey-of-step-xxxxxx’ the input and output artifacts will be downloaded to prefix/iter-xxxxxx/key-of-step/inputs/ and prefix/iter-xxxxxx/key-of-step/outputs/

the downloaded input and output artifacts of steps are defined by op_download_setting

dpgen2.utils.download_dpgen2_artifacts.download_dpgen2_artifacts_by_def(wf: Workflow, iterations: Optional[List[int]] = None, step_defs: Optional[List[str]] = None, prefix: Optional[str] = None, chk_pnt: bool = False)

dpgen2.utils.download_dpgen2_artifacts.print_op_download_setting(op_download_setting={'collect-data': <dpgen2.utils.download_dpgen2_artifacts.DownloadDefinition object>, 'prep-run-fp': <dpgen2.utils.download_dpgen2_artifacts.DownloadDefinition object>, 'prep-run-lmp': <dpgen2.utils.download_dpgen2_artifacts.DownloadDefinition object>, 'prep-run-train': <dpgen2.utils.download_dpgen2_artifacts.DownloadDefinition object>})

dpgen2.utils.obj_artifact module

dpgen2.utils.obj_artifact.dump_object_to_file(obj, fname)

pickle dump object to a file

dpgen2.utils.obj_artifact.load_object_from_file(fname)

pickle load object from a file

dpgen2.utils.run_command module

dpgen2.utils.run_command.run_command(cmd: Union[str, List[str]], shell: bool = False) → Tuple[int, str, str]

dpgen2.utils.step_config module

dpgen2.utils.step_config.dispatcher_args()

free style dispatcher args

dpgen2.utils.step_config.gen_doc(*, make_anchor=True, make_link=True, **kwargs)

dpgen2.utils.step_config.init_executor(executor_dict)

dpgen2.utils.step_config.normalize(data)

dpgen2.utils.step_config.step_conf_args()

dpgen2.utils.step_config.template_conf_args()

dpgen2.utils.step_config.template_slice_conf_args()

dpgen2.utils.step_config.variant_executor()