deepmd.pt.model.network.init

Functions

_no_grad_uniform_(tensor, a, b[, generator])
_no_grad_normal_(tensor, mean, std[, generator])
_no_grad_trunc_normal_(tensor, mean, std, a, b[, ...])
_no_grad_zero_(tensor)
_no_grad_fill_(tensor, val)
calculate_gain(nonlinearity[, param])	Return the recommended gain value for the given nonlinearity function.
_calculate_fan_in_and_fan_out(tensor)
_calculate_correct_fan(tensor, mode)
zeros_(→ torch.Tensor)	Fill the input Tensor with the scalar value 0.
ones_(→ torch.Tensor)	Fill the input Tensor with the scalar value 1.
constant_(→ torch.Tensor)	Fill the input Tensor with the value \(\text{val}\).
normal_(→ torch.Tensor)	Fill the input Tensor with values drawn from the normal distribution.
trunc_normal_(→ torch.Tensor)	Fill the input Tensor with values drawn from a truncated normal distribution.
kaiming_uniform_(tensor[, a, mode, nonlinearity, ...])	Fill the input Tensor with values using a Kaiming uniform distribution.
kaiming_normal_(tensor[, a, mode, nonlinearity, generator])	Fill the input Tensor with values using a Kaiming normal distribution.
xavier_uniform_(→ torch.Tensor)	Fill the input Tensor with values using a Xavier uniform distribution.
xavier_normal_(→ torch.Tensor)	Fill the input Tensor with values using a Xavier normal distribution.

Module Contents

deepmd.pt.model.network.init._no_grad_uniform_(tensor, a, b, generator=None)

deepmd.pt.model.network.init._no_grad_normal_(tensor, mean, std, generator=None)

deepmd.pt.model.network.init._no_grad_trunc_normal_(tensor, mean, std, a, b, generator=None)

deepmd.pt.model.network.init._no_grad_zero_(tensor)

deepmd.pt.model.network.init._no_grad_fill_(tensor, val)

deepmd.pt.model.network.init.calculate_gain(nonlinearity, param=None)

Return the recommended gain value for the given nonlinearity function.

The values are as follows:

nonlinearity	gain
Linear / Identity	\(1\)
Conv{1,2,3}D	\(1\)
Sigmoid	\(1\)
Tanh	\(\frac{5}{3}\)
ReLU	\(\sqrt{2}\)
Leaky Relu	\(\sqrt{\frac{2}{1 + \text{negative\_slope}^2}}\)
SELU	\(\frac{3}{4}\)

Warning

In order to implement Self-Normalizing Neural Networks , you should use nonlinearity='linear' instead of nonlinearity='selu'. This gives the initial weights a variance of 1 / N, which is necessary to induce a stable fixed point in the forward pass. In contrast, the default gain for SELU sacrifices the normalization effect for more stable gradient flow in rectangular layers.

Parameters:

• nonlinearity – the non-linear function (nn.functional name)

• param – optional parameter for the non-linear function

deepmd.pt.model.network.init._calculate_fan_in_and_fan_out(tensor)

deepmd.pt.model.network.init._calculate_correct_fan(tensor, mode)

deepmd.pt.model.network.init.zeros_(tensor: torch.Tensor) → torch.Tensor

Fill the input Tensor with the scalar value 0.

Parameters:

tensor – an n-dimensional torch.Tensor

Examples

>>> w = torch.empty(3, 5)
>>> nn.init.zeros_(w)

deepmd.pt.model.network.init.ones_(tensor: torch.Tensor) → torch.Tensor

Fill the input Tensor with the scalar value 1.

Parameters:

tensor – an n-dimensional torch.Tensor

Examples

>>> w = torch.empty(3, 5)
>>> nn.init.ones_(w)

deepmd.pt.model.network.init.constant_(tensor: torch.Tensor, val: float) → torch.Tensor

Fill the input Tensor with the value \(\text{val}\).

Parameters:

• tensor – an n-dimensional torch.Tensor

• val – the value to fill the tensor with

Examples

>>> w = torch.empty(3, 5)
>>> nn.init.constant_(w, 0.3)

deepmd.pt.model.network.init.normal_(tensor: torch.Tensor, mean: float = 0.0, std: float = 1.0, generator: torch.Generator | None = None) → torch.Tensor

Fill the input Tensor with values drawn from the normal distribution.

\(\mathcal{N}(\text{mean}, \text{std}^2)\).

Parameters:

• tensor – an n-dimensional torch.Tensor

• mean – the mean of the normal distribution

• std – the standard deviation of the normal distribution

• generator – the torch Generator to sample from (default: None)

Examples

>>> w = torch.empty(3, 5)
>>> nn.init.normal_(w)

deepmd.pt.model.network.init.trunc_normal_(tensor: torch.Tensor, mean: float = 0.0, std: float = 1.0, a: float = -2.0, b: float = 2.0, generator: torch.Generator | None = None) → torch.Tensor

Fill the input Tensor with values drawn from a truncated normal distribution.

The values are effectively drawn from the normal distribution \(\mathcal{N}(\text{mean}, \text{std}^2)\) with values outside \([a, b]\) redrawn until they are within the bounds. The method used for generating the random values works best when \(a \leq \text{mean} \leq b\).

Parameters:

• tensor – an n-dimensional torch.Tensor

• mean – the mean of the normal distribution

• std – the standard deviation of the normal distribution

• a – the minimum cutoff value

• b – the maximum cutoff value

• generator – the torch Generator to sample from (default: None)

Examples

>>> w = torch.empty(3, 5)
>>> nn.init.trunc_normal_(w)

deepmd.pt.model.network.init.kaiming_uniform_(tensor: torch.Tensor, a: float = 0, mode: str = 'fan_in', nonlinearity: str = 'leaky_relu', generator: torch.Generator | None = None)

Fill the input Tensor with values using a Kaiming uniform distribution.

The method is described in Delving deep into rectifiers: Surpassing human-level performance on ImageNet classification - He, K. et al. (2015). The resulting tensor will have values sampled from \(\mathcal{U}(-\text{bound}, \text{bound})\) where

\[\text{bound} = \text{gain} \times \sqrt{\frac{3}{\text{fan\_mode}}}\]

Also known as He initialization.

Parameters:

• tensor – an n-dimensional torch.Tensor

• a – the negative slope of the rectifier used after this layer (only used with 'leaky_relu')

• mode – either 'fan_in' (default) or 'fan_out'. Choosing 'fan_in' preserves the magnitude of the variance of the weights in the forward pass. Choosing 'fan_out' preserves the magnitudes in the backwards pass.

• nonlinearity – the non-linear function (nn.functional name), recommended to use only with 'relu' or 'leaky_relu' (default).

• generator – the torch Generator to sample from (default: None)

Examples

>>> w = torch.empty(3, 5)
>>> nn.init.kaiming_uniform_(w, mode="fan_in", nonlinearity="relu")

deepmd.pt.model.network.init.kaiming_normal_(tensor: torch.Tensor, a: float = 0, mode: str = 'fan_in', nonlinearity: str = 'leaky_relu', generator: torch.Generator | None = None)

Fill the input Tensor with values using a Kaiming normal distribution.

The method is described in Delving deep into rectifiers: Surpassing human-level performance on ImageNet classification - He, K. et al. (2015). The resulting tensor will have values sampled from \(\mathcal{N}(0, \text{std}^2)\) where

\[\text{std} = \frac{\text{gain}}{\sqrt{\text{fan\_mode}}}\]

Also known as He initialization.

Parameters:

• tensor – an n-dimensional torch.Tensor

• a – the negative slope of the rectifier used after this layer (only used with 'leaky_relu')

• mode – either 'fan_in' (default) or 'fan_out'. Choosing 'fan_in' preserves the magnitude of the variance of the weights in the forward pass. Choosing 'fan_out' preserves the magnitudes in the backwards pass.

• nonlinearity – the non-linear function (nn.functional name), recommended to use only with 'relu' or 'leaky_relu' (default).

• generator – the torch Generator to sample from (default: None)

Examples

>>> w = torch.empty(3, 5)
>>> nn.init.kaiming_normal_(w, mode="fan_out", nonlinearity="relu")

deepmd.pt.model.network.init.xavier_uniform_(tensor: torch.Tensor, gain: float = 1.0, generator: torch.Generator | None = None) → torch.Tensor

Fill the input Tensor with values using a Xavier uniform distribution.

The method is described in Understanding the difficulty of training deep feedforward neural networks - Glorot, X. & Bengio, Y. (2010). The resulting tensor will have values sampled from \(\mathcal{U}(-a, a)\) where

\[a = \text{gain} \times \sqrt{\frac{6}{\text{fan\_in} + \text{fan\_out}}}\]

Also known as Glorot initialization.

Parameters:

• tensor – an n-dimensional torch.Tensor

• gain – an optional scaling factor

• generator – the torch Generator to sample from (default: None)

Examples

>>> w = torch.empty(3, 5)
>>> nn.init.xavier_uniform_(w, gain=nn.init.calculate_gain("relu"))

deepmd.pt.model.network.init.xavier_normal_(tensor: torch.Tensor, gain: float = 1.0, generator: torch.Generator | None = None) → torch.Tensor

Fill the input Tensor with values using a Xavier normal distribution.

The method is described in Understanding the difficulty of training deep feedforward neural networks - Glorot, X. & Bengio, Y. (2010). The resulting tensor will have values sampled from \(\mathcal{N}(0, \text{std}^2)\) where

\[\text{std} = \text{gain} \times \sqrt{\frac{2}{\text{fan\_in} + \text{fan\_out}}}\]

Also known as Glorot initialization.

Parameters:

• tensor – an n-dimensional torch.Tensor

• gain – an optional scaling factor

• generator – the torch Generator to sample from (default: None)

Examples

>>> w = torch.empty(3, 5)
>>> nn.init.xavier_normal_(w)