dpgen2.utils package
Submodules
dpgen2.utils.binary_file_input module
Binary file inputs
dpgen2.utils.bohrium_config module
dpgen2.utils.chdir module
dpgen2.utils.dflow_config module
dpgen2.utils.dflow_query module
- dpgen2.utils.dflow_query.find_slice_ranges(keys: List[str], sliced_subkey: str)[source]
find range of sliced OPs that matches the pattern ‘iter-[0-9]*–{sliced_subkey}-[0-9]*’
- dpgen2.utils.dflow_query.get_all_schedulers(wf: Any, keys: List[str])[source]
get the output Scheduler of the all the iterations
- dpgen2.utils.dflow_query.get_last_iteration(keys: List[str])[source]
get the index of the last iteraction from a list of step keys.
- dpgen2.utils.dflow_query.get_last_scheduler(wf: Any, keys: List[str])[source]
get the output Scheduler of the last successful iteration
- dpgen2.utils.dflow_query.matched_step_key(all_keys: List[str], step_keys: Optional[List[str]] = None)[source]
returns the keys in all_keys that matches any of the step_keys
dpgen2.utils.download_dpgen2_artifacts module
- class dpgen2.utils.download_dpgen2_artifacts.DownloadDefinition[source]
Bases:
object
Methods
add_def
add_input
add_output
- dpgen2.utils.download_dpgen2_artifacts.download_dpgen2_artifacts(wf: Workflow, key: str, prefix: Optional[str] = None, chk_pnt: bool = False)[source]
download the artifacts of a step. the key should be of format ‘iter-xxxxxx–subkey-of-step-xxxxxx’ the input and output artifacts will be downloaded to prefix/iter-xxxxxx/key-of-step/inputs/ and prefix/iter-xxxxxx/key-of-step/outputs/
the downloaded input and output artifacts of steps are defined by op_download_setting
- dpgen2.utils.download_dpgen2_artifacts.download_dpgen2_artifacts_by_def(wf: Workflow, iterations: Optional[List[int]] = None, step_defs: Optional[List[str]] = None, prefix: Optional[str] = None, chk_pnt: bool = False)[source]
- dpgen2.utils.download_dpgen2_artifacts.print_op_download_setting(op_download_setting={'collect-data': <dpgen2.utils.download_dpgen2_artifacts.DownloadDefinition object>, 'prep-run-fp': <dpgen2.utils.download_dpgen2_artifacts.DownloadDefinition object>, 'prep-run-lmp': <dpgen2.utils.download_dpgen2_artifacts.DownloadDefinition object>, 'prep-run-train': <dpgen2.utils.download_dpgen2_artifacts.DownloadDefinition object>})[source]