portable conda environment as a binary tarball

Question

I am trying to build a portable conda environment. So we can tarball and distribute it to another server or many servers later. Place where we build environment and where it'll be deployed later are two different places. I noticed that conda create and conda install both hard-code environment location into shebang of all the installed scripts under environment_path\bin - is there is a way to override that?

We have to use deployment location and not build location for that environment in shebang bin/ scripts.

Also created https://github.com/conda/conda/issues/7861

Disclaimer: I'm aware of option of rebuilding environment using exported yaml file, but this is not what we're looking here. We want to make conda environment redistributable / portable as a binary tarball - deployment location is known, but it's not the same as environment build location.

Answer 1

I just found conda-pack that seems address this issue directly

https://github.com/conda/conda-pack

conda-pack is a command line tool for creating relocatable conda environments. This is useful for deploying code in a consistent environment, potentially in a location where python/conda isn't already installed.

Documentation: https://conda.github.io/conda-pack/

Use Cases:

• Bundling an application with its environment for deployment
• Packaging a conda environment for usage with Apache Spark when deploying on YARN (see here for more information).
• Packaging a conda environment for deployment on Apache YARN. One way to do this is to use Skein.
• Archiving an environment in a functioning state.

Update: Some other of our PySpark applications are using a conda environment on a location that's available on all Hadoop nodes (NFS mount) and it works very well for some conda environments that don't have a ton of dependencies.

Answer 2

Packages are saved in <conda_root>/pkgs. You can copy these files (as well as the anaconda installer script) to your deployment server, install anaconda, and running conda install <path_to_pkgs>/*.bz2.