Activate and switch Anaconda environment in Dockerfile during build

Question

I've been trying for hours and can't figure out how to activate and switch anaconda environments in a Dockerfile during the build process

Here's the initial code:

FROM nvidia/cuda:10.1-cudnn7-devel-ubuntu16.04

# Set user
ENV SETUSER myuser

RUN useradd -m $SETUSER
USER $SETUSER
WORKDIR /home/$SETUSER

# Install Anaconda
RUN wget https://repo.anaconda.com/archive/Anaconda3-2019.03-Linux-x86_64.sh
RUN bash Anaconda3-2019.03-Linux-x86_64.sh -b
RUN rm Anaconda3-2019.03-Linux-x86_64.sh

# Set path to conda
ENV CONDA_ENV_NAME mynewenv
RUN /home/$SETUSER/anaconda3/condabin/conda create -q --name $CONDA_ENV_NAME python=3.6 && \
    /home/$SETUSER/anaconda3/condabin/conda clean --yes --all
RUN /home/$SETUSER/anaconda3/condabin/conda activate base #Just for testing anaconda environment

At first, anaconda in Docker will complain that the shell is not setup properly, so after the conda create command I added:

RUN /home/$SETUSER/anaconda3/condabin/conda init bash
RUN /bin/bash -c "source /home/$SETUSER/.bashrc"
RUN /home/$SETUSER/anaconda3/condabin/conda activate base

Running the 3 commands after building the docker image works (i.e. running interactively after calling docker run container-name), but for some reason it does not work when building the container. I figured out that the $PATH variable was not being updated during the build, so comparing my $PATH when building and after building.

ENV PATH /home/$SETUSER/anaconda3/envs/$CONDA_ENV_NAME/bin:$PATH
ENV PATH /home/$SETUSER/anaconda3/condabin:$PATH
ENV PATH /home/$SETUSER/anaconda3/bin:$PATH
RUN conda init bash
RUN /bin/bash -c "source /home/$SETUSER/.bashrc"
RUN conda activate base

Now, the Docker $PATH when building and the $PATH when modified interactively when running the container after-building is the same, but I'm still getting the shell not properly setup error.

CommandNotFoundError: Your shell has not been properly configured to use 'conda activate'. To initialize your shell, run $ conda init Currently supported shells are: - bash - fish - tcsh - xonsh - zsh - powershell See 'conda init --help' for more information and options. IMPORTANT: You may need to close and restart your shell after running 'conda init'.

Why is this not working???

I've seen that there may be workaround using a miniconda docker template, but I cannot use that. How do I create and switch anaconda environment during the Docker building process? Thanks!

Answer 1

You've got way too many RUN commands in your Dockerfile. It's not just that each RUN creates a new layer in the image. It's also that each RUN command starts a new shell, and conda activate applies only to the current shell.

You should combine logical groups of actions into a single RUN command. Use && to combine commands, and \ to break lines for readability:

RUN conda activate <myenv> \
 && conda install <whatever> \
 && ...

Keep in mind: at the end of that RUN command, the shell will be gone. So if you want to do something else to that conda environment afterwards, you've got to run conda activate again, or else use -n <myenv> to put something into an environment without activating it first.

When you start a container from the image, you will also have to call conda activate inside the container.

Answer 2

Assuming you want to run something in the conda environment, would the conda run command work for you as below? It works for me to launch a Python process in a specific conda environment:

FROM continuumio/miniconda3:latest
# RUN apt-get update && apt-get -y install gcc  # For compiling a pkg like "regex"
WORKDIR myappdir
COPY environment.yml .
RUN conda update -n base -c defaults conda setuptools && \
    conda env create -n mycondaenv  # Installs environment.yml
COPY myapppkg myapppkg
ENTRYPOINT ["conda", "run", "-n", "mycondaenv", "python", "-m", "myapppkg"]

Note that conda run is experimental as of the date of this answer. For usage help, use conda run --help.

Answer 3

I haven't tested it with the nvidia image, but multi-stage Docker builds should help you out, which will probably look something like:

# get Miniconda docker image to get a installed conda env; WARNING: That image is Debian based
FROM continuumio/miniconda3:latest AS miniconda


# your Docker commands
FROM nvidia/cuda:10.1-cudnn7-devel-ubuntu16.04

# Set user
ENV SETUSER myuser

RUN useradd -m $SETUSER
USER $SETUSER
WORKDIR /home/$SETUSER


# Miniconda: get necessary files from build
COPY --from=miniconda /opt/conda /opt/conda
# Set correct permissions
RUN chown -R $SETUSER: /opt/conda
#   New terminals will have conda active
# If nvidia's Docker image has no .bashrc
# COPY --from=miniconda /home/$SETUSER/.bashrc
# else
RUN echo ". /opt/conda/etc/profile.d/conda.sh" >> ~/.bashrc && \
    echo "conda activate base" >> ~/.bashrc

# switch shell sh (default in Linux) to bash
SHELL ["/bin/bash", "-c"]

# give bash access to Anaconda, then normal anaconda commands, e.g. (-q: quiet, -y: answer yes)
RUN source /home/$SETUSER/.bashrc \
 && conda create -q --name testy \
 && conda activate testy \
 && conda install -y your_package

Inspiration from this GitHub issue: https://github.com/ContinuumIO/docker-images/issues/89