dpdata.formats.deepmd.lmdb package

dpdata.formats.deepmd.lmdb package#

class dpdata.formats.deepmd.lmdb.LMDBFormat[source]#

Bases: Format

DeePMD-kit compatible LMDB format.

A single flat LMDB stores all frames from one or many systems. The same on-disk format is produced regardless of whether the source is a standard or a mixed-type system, so the output is always readable by DeePMD-kit’s LmdbDataReader.

The mixed_type keyword controls only how frames are mapped back to dpdata objects on read (see from_multi_systems()).

Methods

MultiModes()

File mode for MultiSystems.

dump_systems(systems, directory[, map_size, ...])

Write an ordered sequence of systems, one frame_system_id each.

from_bond_order_system(file_name, **kwargs)

Implement BondOrderSystem.from that converts from this format to BondOrderSystem.

from_labeled_system(file_name, **kwargs)

Load the first composition group as a LabeledSystem.

from_multi_systems(directory[, mixed_type, ...])

Load systems from a flat LMDB.

from_system(file_name, **kwargs)

Load the first composition group as a System.

get_formats()

Get all registered formats.

get_from_methods()

Get all registered from methods.

get_to_methods()

Get all registered to methods.

mix_system(*system, type_map, **kwargs)

Mix the systems into mixed_type ones according to the unified given type_map.

post(func_name)

Register a post function for from method.

register(key)

Register a format plugin.

register_from(key)

Register a from method if the target method name is not default.

register_to(key)

Register a to method if the target method name is not default.

to_bond_order_system(data, rdkit_mol, *args, ...)

Implement BondOrderSystem.to that converts from BondOrderSystem to this format.

to_labeled_system(data, file_name, **kwargs)

Save a single LabeledSystem to an LMDB database.

to_multi_systems(formulas, directory[, ...])

Write multiple dpdata systems to one LMDB.

to_system(data, file_name, **kwargs)

Save a single unlabeled System to an LMDB database.

Examples

Write a single labeled system:

>>> import dpdata
>>> ls = dpdata.LabeledSystem("OUTCAR", fmt="vasp/outcar")
>>> ls.to("deepmd/lmdb", "data.lmdb")

Write many systems into one LMDB, forcing a global type map:

>>> ms = dpdata.MultiSystems(s1, s2, s3)
>>> ms.to("deepmd/lmdb", "data.lmdb", type_map=["H", "C", "N", "O"])

Read back as standard (per-composition) systems:

>>> ms = dpdata.MultiSystems.from_file("data.lmdb", fmt="deepmd/lmdb")

Read back keeping the full global type map on every system:

>>> ms = dpdata.MultiSystems.from_file(
...     "data.lmdb", fmt="deepmd/lmdb", mixed_type=True
... )

Note that loading through dpdata.MultiSystems normalises the atom_names order (the element set is kept, but reordered); a direct single-system load preserves the stored order.

dump_systems(systems, directory, map_size: int = 1099511627776, frame_idx_fmt: str = '012d', type_map: list[str] | None = None, write_batch_size: int = 1000, overwrite: bool = False)[source]#

Write an ordered sequence of systems, one frame_system_id each.

Unlike to_multi_systems() (the path used by MultiSystems.to('deepmd/lmdb', ...)), the systems are not merged by formula: every element of systems becomes exactly one source system in the database, numbered 0, 1, 2, ... in iteration order. This preserves the system partition recorded in frame_system_ids, which DeePMD-kit uses for prob_sys_size based sampling.

Parameters:
systemsIterable[System]

An ordered iterable of dpdata.System / dpdata.LabeledSystem objects (or raw data dicts). A dpdata.MultiSystems must not be used here, because its frames are already merged by formula.

directorystr

Output LMDB directory.

map_sizeint, optional

Maximum LMDB size in bytes. Default is 1 TiB (sparse).

frame_idx_fmtstr, optional

Format used for the per-frame integer key. Default "012d".

type_maplist[str], optional

Global element table. If None, the union of the elements of all systems is used, in first-appearance order. When the systems are produced lazily (a generator) a type_map should be given so that the whole sequence need not be held in memory.

write_batch_sizeint, optional

Number of frames committed per LMDB write transaction.

overwritebool, optional

Whether to replace an existing destination after the new database has been written and validated. This option is supported on POSIX systems only. The default is False.

from_labeled_system(file_name, **kwargs)[source]#

Load the first composition group as a LabeledSystem.

Parameters:
file_namestr, os.PathLike, or dict

LMDB directory, or an already decoded system-data dictionary used internally by MultiSystems loading.

**kwargsdict

Reader options accepted by from_multi_systems(), including mixed_type, type_map, and max_frames.

Returns:
dict

Labeled data for the first composition group.

Raises:
LMDBFrameError

If the selected data has no energy labels.

from_multi_systems(directory, mixed_type: bool = False, type_map: list[str] | None = None, max_frames: int | None = 100000, **kwargs)[source]#

Load systems from a flat LMDB.

Frames are grouped by atom-count composition. Atom order is canonicalized by a stable sort on the global atom type, and every registered atomic field follows the same permutation. Each composition becomes one dpdata system.

Parameters:
directorystr

Path to the LMDB directory.

mixed_typebool, optional

If False (default) each system’s atom_names is the compact set of elements it actually contains. If True every system keeps the full global type_map as atom_names (with zero counts for absent elements).

type_maplist[str], optional

Requested element table for the returned systems. When the file stores a type_map, the stored global indices are remapped to this table by element name (consistent with DeePMD-kit); every element in the file must be present in type_map. When the file has no type_map, the indices are named positionally from this argument. Defaults to the type_map stored in the file.

max_framesint or None, optional

Maximum number of frames loaded into memory. The default is 100,000. Set to None only when sufficient memory is available.

**kwargsdict

other parameters

Yields:
dict

system data dictionary for each composition group.

from_system(file_name, **kwargs)[source]#

Load the first composition group as a System.

Parameters:
file_namestr, os.PathLike, or dict

LMDB directory, or an already decoded system-data dictionary used internally by MultiSystems loading.

**kwargsdict

Reader options accepted by from_multi_systems(), including mixed_type, type_map, and max_frames.

Returns:
dict

Unlabeled data for the first composition group.

Warns:
UserWarning

If the database contains more than one composition. Use dpdata.MultiSystems.from_file() to load all groups.

to_labeled_system(data, file_name, **kwargs)[source]#

Save a single LabeledSystem to an LMDB database.

Parameters:
datadict

LabeledSystem data to write.

file_namestr or os.PathLike

Destination LMDB directory.

**kwargsdict

Writer options accepted by to_multi_systems(), including map_size, frame_idx_fmt, type_map, write_batch_size, and overwrite.

to_multi_systems(formulas, directory, map_size: int = 1099511627776, frame_idx_fmt: str = '012d', type_map: list[str] | None = None, write_batch_size: int = 1000, overwrite: bool = False, **kwargs)[source]#

Write multiple dpdata systems to one LMDB.

Parameters:
formulaslist[str]

One handle per system (the value is not used on disk).

directorystr

Output LMDB directory.

map_sizeint, optional

Maximum LMDB size in bytes. Default is 1 TiB (sparse).

frame_idx_fmtstr, optional

Format used for the per-frame integer key. Default "012d".

type_maplist[str], optional

Global element table. If None, the element list of the first system written is used (for a MultiSystems this is the union of all systems’ elements).

write_batch_sizeint, optional

Number of frames committed per LMDB write transaction.

overwritebool, optional

Whether to replace an existing destination after the new database has been written and validated. This option is supported on POSIX systems only. The default is False.

**kwargsdict

other parameters

Yields:
tuple

(self, formula) handle consumed by to_system().

to_system(data, file_name, **kwargs)[source]#

Save a single unlabeled System to an LMDB database.

Parameters:
datadict

System data to write.

file_namestr or os.PathLike

Destination LMDB directory.

**kwargsdict

Writer options accepted by to_multi_systems(), including map_size, frame_idx_fmt, type_map, write_batch_size, and overwrite.

dpdata.formats.deepmd.lmdb.dump_systems(systems, file_name, type_map: list[str] | None = None, map_size: int = 1099511627776, frame_idx_fmt: str = '012d', write_batch_size: int = 1000, overwrite: bool = False)[source]#

Write an ordered sequence of systems to one LMDB, preserving identity.

Each element of systems is stored as a distinct source system, numbered 0, 1, 2, ... in iteration order, and recorded in the frame_system_ids metadata. In contrast to MultiSystems.to('deepmd/lmdb', ...), systems are not merged by formula, so the system partition used by DeePMD-kit’s prob_sys_size is kept.

Parameters:
systemsIterable[System]

An ordered iterable of dpdata.System / dpdata.LabeledSystem objects or raw system data dictionaries. Do not pass a dpdata.MultiSystems, whose frames are already merged by formula.

file_namestr

Output LMDB directory.

type_maplist[str], optional

Global element table. If None, the union of the elements of all systems is used. Provide it explicitly to stream a generator without materialising the whole sequence.

map_sizeint, optional

Maximum LMDB size in bytes. Default is 1 TiB (sparse).

frame_idx_fmtstr, optional

Format used for the per-frame integer key. Default "012d".

write_batch_sizeint, optional

Number of frames committed per LMDB write transaction.

overwritebool, optional

Whether to replace an existing destination after the new database has been written and validated. This option is supported on POSIX systems only. The default is False.

Examples

>>> import dpdata
>>> from dpdata.formats.deepmd.lmdb import dump_systems
>>> systems = [
...     dpdata.LabeledSystem(d, fmt="deepmd/npy") for d in directories
... ]
>>> dump_systems(systems, "data.lmdb", type_map=["H", "C", "N", "O"])

Submodules#

dpdata.formats.deepmd.lmdb.format module#

LMDB format that is fully interoperable with DeePMD-kit.

The on-disk layout is a flat sequence of frames, identical to what DeePMD-kit’s LmdbDataReader consumes and what the community npy_to_lmdb / json_to_lmdb converters produce:

  • __metadata__ (msgpack dict, string keys):

    {
    "nframes": int,
    "frame_idx_fmt": "012d",
    "type_map": [str, ...],          # global element table
    "frame_nlocs": [int, ...],       # atoms per frame
    "frame_system_ids": [int, ...],  # source-system index per frame
}

  • one entry per frame, keyed by the zero-padded global frame index ("000000000000" by default), whose value is a msgpack dict with string keys. Array values use the manual encoding {"type": str(dtype), "shape": [...], "data": <bytes>} and atom_numbs is a plain list of per-type counts over type_map.

    Core per-frame keys use the plural names consumed by DeePMD-kit (coords, cells, energies, forces, virials, atom_types). Registered additional fields use dpdata.data_type.DataType.deepmd_name.

atom_types is stored as int32 global indices into type_map.

Notes#

This reader loads frames into memory, mirroring dpdata’s in-memory System model. A configurable frame-count guard prevents accidental decoding of very large databases; training datasets should normally be consumed directly by DeePMD-kit’s streaming dataloader.

exception dpdata.formats.deepmd.lmdb.format.LMDBError[source]#

Bases: Exception

Base class for LMDB errors.

class dpdata.formats.deepmd.lmdb.format.LMDBFormat[source]#

Bases: Format

DeePMD-kit compatible LMDB format.

A single flat LMDB stores all frames from one or many systems. The same on-disk format is produced regardless of whether the source is a standard or a mixed-type system, so the output is always readable by DeePMD-kit’s LmdbDataReader.

The mixed_type keyword controls only how frames are mapped back to dpdata objects on read (see from_multi_systems()).

Methods

MultiModes()

File mode for MultiSystems.

dump_systems(systems, directory[, map_size, ...])

Write an ordered sequence of systems, one frame_system_id each.

from_bond_order_system(file_name, **kwargs)

Implement BondOrderSystem.from that converts from this format to BondOrderSystem.

from_labeled_system(file_name, **kwargs)

Load the first composition group as a LabeledSystem.

from_multi_systems(directory[, mixed_type, ...])

Load systems from a flat LMDB.

from_system(file_name, **kwargs)

Load the first composition group as a System.

get_formats()

Get all registered formats.

get_from_methods()

Get all registered from methods.

get_to_methods()

Get all registered to methods.

mix_system(*system, type_map, **kwargs)

Mix the systems into mixed_type ones according to the unified given type_map.

post(func_name)

Register a post function for from method.

register(key)

Register a format plugin.

register_from(key)

Register a from method if the target method name is not default.

register_to(key)

Register a to method if the target method name is not default.

to_bond_order_system(data, rdkit_mol, *args, ...)

Implement BondOrderSystem.to that converts from BondOrderSystem to this format.

to_labeled_system(data, file_name, **kwargs)

Save a single LabeledSystem to an LMDB database.

to_multi_systems(formulas, directory[, ...])

Write multiple dpdata systems to one LMDB.

to_system(data, file_name, **kwargs)

Save a single unlabeled System to an LMDB database.

Examples

Write a single labeled system:

>>> import dpdata
>>> ls = dpdata.LabeledSystem("OUTCAR", fmt="vasp/outcar")
>>> ls.to("deepmd/lmdb", "data.lmdb")

Write many systems into one LMDB, forcing a global type map:

>>> ms = dpdata.MultiSystems(s1, s2, s3)
>>> ms.to("deepmd/lmdb", "data.lmdb", type_map=["H", "C", "N", "O"])

Read back as standard (per-composition) systems:

>>> ms = dpdata.MultiSystems.from_file("data.lmdb", fmt="deepmd/lmdb")

Read back keeping the full global type map on every system:

>>> ms = dpdata.MultiSystems.from_file(
...     "data.lmdb", fmt="deepmd/lmdb", mixed_type=True
... )

Note that loading through dpdata.MultiSystems normalises the atom_names order (the element set is kept, but reordered); a direct single-system load preserves the stored order.

dump_systems(systems, directory, map_size: int = 1099511627776, frame_idx_fmt: str = '012d', type_map: list[str] | None = None, write_batch_size: int = 1000, overwrite: bool = False)[source]#

Write an ordered sequence of systems, one frame_system_id each.

Unlike to_multi_systems() (the path used by MultiSystems.to('deepmd/lmdb', ...)), the systems are not merged by formula: every element of systems becomes exactly one source system in the database, numbered 0, 1, 2, ... in iteration order. This preserves the system partition recorded in frame_system_ids, which DeePMD-kit uses for prob_sys_size based sampling.

Parameters:
systemsIterable[System]

An ordered iterable of dpdata.System / dpdata.LabeledSystem objects (or raw data dicts). A dpdata.MultiSystems must not be used here, because its frames are already merged by formula.

directorystr

Output LMDB directory.

map_sizeint, optional

Maximum LMDB size in bytes. Default is 1 TiB (sparse).

frame_idx_fmtstr, optional

Format used for the per-frame integer key. Default "012d".

type_maplist[str], optional

Global element table. If None, the union of the elements of all systems is used, in first-appearance order. When the systems are produced lazily (a generator) a type_map should be given so that the whole sequence need not be held in memory.

write_batch_sizeint, optional

Number of frames committed per LMDB write transaction.

overwritebool, optional

Whether to replace an existing destination after the new database has been written and validated. This option is supported on POSIX systems only. The default is False.

from_labeled_system(file_name, **kwargs)[source]#

Load the first composition group as a LabeledSystem.

Parameters:
file_namestr, os.PathLike, or dict

LMDB directory, or an already decoded system-data dictionary used internally by MultiSystems loading.

**kwargsdict

Reader options accepted by from_multi_systems(), including mixed_type, type_map, and max_frames.

Returns:
dict

Labeled data for the first composition group.

Raises:
LMDBFrameError

If the selected data has no energy labels.

from_multi_systems(directory, mixed_type: bool = False, type_map: list[str] | None = None, max_frames: int | None = 100000, **kwargs)[source]#

Load systems from a flat LMDB.

Frames are grouped by atom-count composition. Atom order is canonicalized by a stable sort on the global atom type, and every registered atomic field follows the same permutation. Each composition becomes one dpdata system.

Parameters:
directorystr

Path to the LMDB directory.

mixed_typebool, optional

If False (default) each system’s atom_names is the compact set of elements it actually contains. If True every system keeps the full global type_map as atom_names (with zero counts for absent elements).

type_maplist[str], optional

Requested element table for the returned systems. When the file stores a type_map, the stored global indices are remapped to this table by element name (consistent with DeePMD-kit); every element in the file must be present in type_map. When the file has no type_map, the indices are named positionally from this argument. Defaults to the type_map stored in the file.

max_framesint or None, optional

Maximum number of frames loaded into memory. The default is 100,000. Set to None only when sufficient memory is available.

**kwargsdict

other parameters

Yields:
dict

system data dictionary for each composition group.

from_system(file_name, **kwargs)[source]#

Load the first composition group as a System.

Parameters:
file_namestr, os.PathLike, or dict

LMDB directory, or an already decoded system-data dictionary used internally by MultiSystems loading.

**kwargsdict

Reader options accepted by from_multi_systems(), including mixed_type, type_map, and max_frames.

Returns:
dict

Unlabeled data for the first composition group.

Warns:
UserWarning

If the database contains more than one composition. Use dpdata.MultiSystems.from_file() to load all groups.

to_labeled_system(data, file_name, **kwargs)[source]#

Save a single LabeledSystem to an LMDB database.

Parameters:
datadict

LabeledSystem data to write.

file_namestr or os.PathLike

Destination LMDB directory.

**kwargsdict

Writer options accepted by to_multi_systems(), including map_size, frame_idx_fmt, type_map, write_batch_size, and overwrite.

to_multi_systems(formulas, directory, map_size: int = 1099511627776, frame_idx_fmt: str = '012d', type_map: list[str] | None = None, write_batch_size: int = 1000, overwrite: bool = False, **kwargs)[source]#

Write multiple dpdata systems to one LMDB.

Parameters:
formulaslist[str]

One handle per system (the value is not used on disk).

directorystr

Output LMDB directory.

map_sizeint, optional

Maximum LMDB size in bytes. Default is 1 TiB (sparse).

frame_idx_fmtstr, optional

Format used for the per-frame integer key. Default "012d".

type_maplist[str], optional

Global element table. If None, the element list of the first system written is used (for a MultiSystems this is the union of all systems’ elements).

write_batch_sizeint, optional

Number of frames committed per LMDB write transaction.

overwritebool, optional

Whether to replace an existing destination after the new database has been written and validated. This option is supported on POSIX systems only. The default is False.

**kwargsdict

other parameters

Yields:
tuple

(self, formula) handle consumed by to_system().

to_system(data, file_name, **kwargs)[source]#

Save a single unlabeled System to an LMDB database.

Parameters:
datadict

System data to write.

file_namestr or os.PathLike

Destination LMDB directory.

**kwargsdict

Writer options accepted by to_multi_systems(), including map_size, frame_idx_fmt, type_map, write_batch_size, and overwrite.

exception dpdata.formats.deepmd.lmdb.format.LMDBFrameError[source]#

Bases: LMDBError

Frame data not found in LMDB.

exception dpdata.formats.deepmd.lmdb.format.LMDBMetadataError[source]#

Bases: LMDBError

Metadata not found or malformed in LMDB.

dpdata.formats.deepmd.lmdb.format.dump_systems(systems, file_name, type_map: list[str] | None = None, map_size: int = 1099511627776, frame_idx_fmt: str = '012d', write_batch_size: int = 1000, overwrite: bool = False)[source]#

Write an ordered sequence of systems to one LMDB, preserving identity.

Each element of systems is stored as a distinct source system, numbered 0, 1, 2, ... in iteration order, and recorded in the frame_system_ids metadata. In contrast to MultiSystems.to('deepmd/lmdb', ...), systems are not merged by formula, so the system partition used by DeePMD-kit’s prob_sys_size is kept.

Parameters:
systemsIterable[System]

An ordered iterable of dpdata.System / dpdata.LabeledSystem objects or raw system data dictionaries. Do not pass a dpdata.MultiSystems, whose frames are already merged by formula.

file_namestr

Output LMDB directory.

type_maplist[str], optional

Global element table. If None, the union of the elements of all systems is used. Provide it explicitly to stream a generator without materialising the whole sequence.

map_sizeint, optional

Maximum LMDB size in bytes. Default is 1 TiB (sparse).

frame_idx_fmtstr, optional

Format used for the per-frame integer key. Default "012d".

write_batch_sizeint, optional

Number of frames committed per LMDB write transaction.

overwritebool, optional

Whether to replace an existing destination after the new database has been written and validated. This option is supported on POSIX systems only. The default is False.

Examples

>>> import dpdata
>>> from dpdata.formats.deepmd.lmdb import dump_systems
>>> systems = [
...     dpdata.LabeledSystem(d, fmt="deepmd/npy") for d in directories
... ]
>>> dump_systems(systems, "data.lmdb", type_map=["H", "C", "N", "O"])