dpgen2.utils package

Submodules

dpgen2.utils.bohrium_config module

dpgen2.utils.bohrium_config.bohrium_config_from_dict(bohrium_config)[source]

dpgen2.utils.chdir module

dpgen2.utils.chdir.chdir(path_key: str)[source]

Returns a decorator that can change the current working path.

Parameters
path_keystr

key to OPIO

Examples

>>> class SomeOP(OP):
...     @chdir("path")
...     def execute(self, ip: OPIO):
...         do_something() 
dpgen2.utils.chdir.set_directory(path: Path)[source]

Sets the current working path within the context.

Parameters
pathPath

The path to the cwd

Yields
None

Examples

>>> with set_directory("some_path"):
...    do_something()

dpgen2.utils.dflow_config module

dpgen2.utils.dflow_config.dflow_config(config_data)[source]

set the dflow config by config_data

the keys starting with “s3_” will be treated as s3_config keys, other keys are treated as config keys.

dpgen2.utils.dflow_config.dflow_config_lower(dflow_config)[source]
dpgen2.utils.dflow_config.dflow_s3_config(config_data)[source]

set the s3 config by config_data

dpgen2.utils.dflow_config.dflow_s3_config_lower(dflow_s3_config_data)[source]
dpgen2.utils.dflow_config.workflow_config_from_dict(wf_config)[source]

dpgen2.utils.dflow_query module

dpgen2.utils.dflow_query.find_slice_ranges(keys: List[str], sliced_subkey: str)[source]

find range of sliced OPs that matches the pattern ‘iter-[0-9]*–{sliced_subkey}-[0-9]*’

dpgen2.utils.dflow_query.get_all_schedulers(wf: Any, keys: List[str])[source]

get the output Scheduler of the all the iterations

dpgen2.utils.dflow_query.get_iteration(key: str)[source]
dpgen2.utils.dflow_query.get_last_iteration(keys: List[str])[source]

get the index of the last iteraction from a list of step keys.

dpgen2.utils.dflow_query.get_last_scheduler(wf: Any, keys: List[str])[source]

get the output Scheduler of the last successful iteration

dpgen2.utils.dflow_query.get_subkey(key: str, idx: int = -1)[source]
dpgen2.utils.dflow_query.matched_step_key(all_keys: List[str], step_keys: Optional[List[str]] = None)[source]

returns the keys in all_keys that matches any of the step_keys

dpgen2.utils.dflow_query.print_keys_in_nice_format(keys: List[str], sliced_subkey: List[str], idx_fmt_len: int = 8)[source]
dpgen2.utils.dflow_query.sort_slice_ops(keys: List[str], sliced_subkey: List[str])[source]

sort the keys of the sliced ops. the keys of the sliced ops contains sliced_subkey

dpgen2.utils.download_dpgen2_artifacts module

class dpgen2.utils.download_dpgen2_artifacts.DownloadDefinition[source]

Bases: object

Methods

add_def

add_input

add_output

add_def(tdict, key, suffix=None)[source]
add_input(input_key, suffix=None)[source]
add_output(output_key, suffix=None)[source]
dpgen2.utils.download_dpgen2_artifacts.download_dpgen2_artifacts(wf: Workflow, key: str, prefix: Optional[str] = None, chk_pnt: bool = False)[source]

download the artifacts of a step. the key should be of format ‘iter-xxxxxx–subkey-of-step-xxxxxx’ the input and output artifacts will be downloaded to prefix/iter-xxxxxx/key-of-step/inputs/ and prefix/iter-xxxxxx/key-of-step/outputs/

the downloaded input and output artifacts of steps are defined by op_download_setting

dpgen2.utils.obj_artifact module

dpgen2.utils.obj_artifact.dump_object_to_file(obj, fname)[source]

pickle dump object to a file

dpgen2.utils.obj_artifact.load_object_from_file(fname)[source]

pickle load object from a file

dpgen2.utils.run_command module

dpgen2.utils.run_command.run_command(cmd: Union[str, List[str]], shell: bool = False) Tuple[int, str, str][source]

dpgen2.utils.step_config module

dpgen2.utils.step_config.dispatcher_args()[source]

free style dispatcher args

dpgen2.utils.step_config.gen_doc(*, make_anchor=True, make_link=True, **kwargs)[source]
dpgen2.utils.step_config.init_executor(executor_dict)[source]
dpgen2.utils.step_config.lebesgue_executor_args()[source]
dpgen2.utils.step_config.lebesgue_extra_args()[source]
dpgen2.utils.step_config.normalize(data)[source]
dpgen2.utils.step_config.step_conf_args()[source]
dpgen2.utils.step_config.template_conf_args()[source]
dpgen2.utils.step_config.variant_executor()[source]