dpgen2.utils package#
Submodules#
dpgen2.utils.artifact_uri module#
dpgen2.utils.binary_file_input module#
Binary file inputs
dpgen2.utils.bohrium_config module#
dpgen2.utils.chdir module#
dpgen2.utils.dflow_config module#
dpgen2.utils.dflow_query module#
- dpgen2.utils.dflow_query.find_slice_ranges(keys: List[str], sliced_subkey: str)[source]#
find range of sliced OPs that matches the pattern ‘iter-[0-9]*–{sliced_subkey}-[0-9]*’
- dpgen2.utils.dflow_query.get_all_schedulers(wf: Any, keys: List[str])[source]#
get the output Scheduler of the all the iterations
- dpgen2.utils.dflow_query.get_last_iteration(keys: List[str])[source]#
get the index of the last iteraction from a list of step keys.
- dpgen2.utils.dflow_query.get_last_scheduler(wf: Any, keys: List[str])[source]#
get the output Scheduler of the last successful iteration
- dpgen2.utils.dflow_query.matched_step_key(all_keys: List[str], step_keys: List[str] | None = None)[source]#
returns the keys in all_keys that matches any of the step_keys
dpgen2.utils.download_dpgen2_artifacts module#
- class dpgen2.utils.download_dpgen2_artifacts.DownloadDefinition[source]#
Bases:
object
Methods
add_def
add_input
add_output
- dpgen2.utils.download_dpgen2_artifacts.download_dpgen2_artifacts(wf: Workflow, key: str, prefix: str | None = None, chk_pnt: bool = False)[source]#
download the artifacts of a step. the key should be of format ‘iter-xxxxxx–subkey-of-step-xxxxxx’ the input and output artifacts will be downloaded to prefix/iter-xxxxxx/key-of-step/inputs/ and prefix/iter-xxxxxx/key-of-step/outputs/
the downloaded input and output artifacts of steps are defined by op_download_setting
- dpgen2.utils.download_dpgen2_artifacts.download_dpgen2_artifacts_by_def(wf: Workflow, iterations: List[int] | None = None, step_defs: List[str] | None = None, prefix: str | None = None, chk_pnt: bool = False)[source]#
- dpgen2.utils.download_dpgen2_artifacts.print_op_download_setting(op_download_setting={'collect-data': <dpgen2.utils.download_dpgen2_artifacts.DownloadDefinition object>, 'prep-run-explore': <dpgen2.utils.download_dpgen2_artifacts.DownloadDefinition object>, 'prep-run-fp': <dpgen2.utils.download_dpgen2_artifacts.DownloadDefinition object>, 'prep-run-train': <dpgen2.utils.download_dpgen2_artifacts.DownloadDefinition object>})[source]#