Conda/pip environments
Note
The following instructions are for the Linux operating system. Most DeepModeling projects fully support the Linux OS but lack support for other OS. If you use the Windows OS, you can use Windows Subsystem for Linux (WSL).
How to setup a conda/pip base environment?
Note
The following instructions are for Miniforge but should also work for the offline packages provided by the DeepModeling projects.
An environment is a directory that contains a specific collection of packages that you have installed. Different environments are isolated from each other. A base environment is required before setting up multiple environments.
First of all, download and execute conda-forge installer:
wget https://github.com/conda-forge/miniforge/releases/latest/download/Miniforge3-Linux-x86_64.sh -O miniforge.sh
bash miniforge.sh -b -p $HOME/miniforge3
This will install the conda base environment into $HOME/miniforge3
. You can install the environment into a different path.
After installation, you can activate the base environment using
source $HOME/miniforge3/bin/activate
In order to initialize the environment whenever starting the bash shell, run
conda init --all
Now you can use conda install
or pip install
command to install conda or pip packages.
What is the difference between conda and pip?
The largest difference is that pip can only install Python packages, but conda can install any packages including C, C++, and Fortran packages without Python involved. We suggest reading the Anaconda blog for more information.
How can I check what packages have been installed?
Run conda list
and pip list
, which show all conda/pip packages that have been installed in the current environment.
How to create a new conda/pip environment?
To create a conda/pip environment named my_env
, run
conda create -n my_env python=3.11
Now you can activate the environment using
conda activate my_env
If you want to activate the environment in a shell script, use source
instead:
source activate my_env
Why do I need multiple environments instead of one?
Due to multiple reasons, dependencies of packages often conflict with each other. For example, Python 3.11 and 3.12 cannot be installed in the same environment; CUDA 11 and CUDA 12 packages cannot be installed in the same environment; Open MPI and MPICH cannot be installed in the same environment; packages built against zlib 1.2 and 1.3 cannot be installed in the same environment. While conda-forge has tried its best to reduce package conflicts, it may still happen when installing multiple packages. Installing different packages into different environments will reduce the risk of meeting the conflict packages.
How to use conda to install CUDA packages without CUDA installed?
Usually, conda
detects the CUDA driver on the system to install the compatible CUDA packages. You don’t need to worry about installing the CUDA Toolkit in advance. However, sometimes, you may want to install CUDA packages into a machine that doesn’t install CUDA drivers (for example, the login node of an HPC cluster). In this case, you must manually set the environment variable CONDA_OVERRIDE_CUDA
to tell conda which packages are compatible.
For example, you would like to use the CUDA packages on another machine with installed CUDA drivers. Run nvidia-smi
on that machine,
+---------------------------------------------------------------------------------------+
| NVIDIA-SMI 545.29.06 Driver Version: 545.29.06 CUDA Version: 12.3 |
|-----------------------------------------+----------------------+----------------------+
The output shows that the CUDA driver version is 12.3
. Then, on the machine for performing installation, you may set the environment variable export CONDA_OVERRIDE_CUDA=12.3
before running conda install
.
How to install conda/pip packages without the Internet access?
To install conda/pip packages without Internet access, you need to have a machine that can connect to the Internet, and use conda
or pip
to install all packages in an environment. Then, install conda-pack:
conda install conda-pack
Next, running conda pack
to pack an environment, saying my_env
:
conda pack -n my_env -o my_env.tar.gz
Transfer my_env.tar.gz
to the machine that doesn’t have the Internet access. On that machine, you can unpack the environment:
mkdir -p my_env
tar -xzf my_env.tar.gz -C my_env
source my_env/bin/activate
conda-unpack
How to speed up conda/pip in China?
When you are in China, we suggest adding Tsinghua University tuna mirrors to your conda/pip configs to speed up downloading conda/pip packages.
In order to set up Anaconda mirrors, append the following lines to the $HOME/.condarc
file (if it doesn’t exist, you can create one):
default_channels:
- https://mirrors.tuna.tsinghua.edu.cn/anaconda/pkgs/main
- https://mirrors.tuna.tsinghua.edu.cn/anaconda/pkgs/r
custom_channels:
conda-forge: https://mirrors.tuna.tsinghua.edu.cn/anaconda/cloud
In order to set up PyPI mirrors, run
pip config set global.index-url https://pypi.tuna.tsinghua.edu.cn/simple