26
votes

I need to activate environment in docker and run a command in this environment. I create the environment, but then I try to activate this environment and run the command in this way:

CMD [ "source activate mro_env && ipython kernel install --user --name=mro_env" ]

but when I ran docker I get an error:

[FATAL tini (8)] exec source activate mro_env && ipython kernel install 
--user --name=mro_env failed: No such file or directory

This is the whole Dockerfile:

FROM continuumio/miniconda3

ADD /src/mro_env.yml /src/mro_env.yml
RUN conda env create -f /src/mro_env.yml

# Pull the environment name out of the mro_env.yml
RUN echo "source activate $(head -1 /src/mro_env.yml | cut -d' ' -f2)" > ~/.bashrc
ENV PATH /opt/conda/envs/$(head -1 /src/mro_env.yml | cut -d' ' -f2)/bin:$PATH

CMD [ "source activate mro_env && ipython kernel install --user --name=mro_env" ]
9
Please show us your Dockerfile; it looks like you didn't actually wind up creating the mro_env environment. Please also check out the official documentation for the CMD instruction. - Jordan Singer
@JordanSinger I added the Dockerfile - Yahav Festinger
Are you sure your environment creation succeeded? - FabienP
@FabienP Yes. the environment installed successfully. the problem is the integration with jupyter notebook. my goal is to install the environment on jupyter, and launch jupyter with this environment. Do you have any idea how could I do this? - Yahav Festinger
The way I do this is usually to install Jupyter in the environment, then launch it from there. Otherwise you could have a look to nb_conda_kernels to reach this environment from your Jupyter server. - FabienP

9 Answers

15
votes

Followed this tutorial and it worked. Example Dockerfile:

FROM continuumio/miniconda
WORKDIR /usr/src/app
COPY ./ ./
RUN conda env create -f environment.yml

# Make RUN commands use the new environment:
SHELL ["conda", "run", "-n", "myenv", "/bin/bash", "-c"]

EXPOSE 5003
# The code to run when container is started:
ENTRYPOINT ["conda", "run", "-n", "myenv", "python3", "src/server.py"]
7
votes

You can set CONDA_DEFAULT_ENV

Like this:

FROM continuumio/miniconda3

ARG conda_env=mro_env

ADD /src/environment.yml /src/environment.yml
RUN conda env create -f /src/environment.yml

ENV PATH /opt/conda/envs/$conda_env/bin:$PATH
ENV CONDA_DEFAULT_ENV $conda_env

CMD [ "python", "test.py" ]

UPDATE:

Better use activate. Work for me:

FROM continuumio/miniconda3

ADD /src/environment.yml /src/environment.yml

RUN conda env create -f /src/environment.yml
ENV PATH /opt/conda/envs/mro_env/bin:$PATH
RUN /bin/bash -c "source activate mro_env"

CMD [ "python", "test.py" ]
5
votes

As the user merv points out in one of the comments above (sorry don't have enough rep to vote up the comment) conda run buffers all stdout/stderr, thus making it not workable for applications interacting or even just displaying logs over I/O.

I noticed there was no accepted answer, so I just post what has worked very well for me:

You can use an entrypoint script to activate the conda enviroment and let it take over further input from the Dockerfile such that the python script can be executed within the activated conda environment

Example Dockerfile:

FROM continuumio/miniconda3

# make docker use bash instead of sh
SHELL ["/bin/bash", "--login", "-c"]

# copy all necessary files
COPY environment.yml .
COPY ownchain/mypyscript.py .
COPY entrypoint.sh /usr/local/bin/

# make entrypoint script executable
RUN chmod u+x /usr/local/bin/entrypoint.sh
# create environment
RUN conda env create -f environment.yml

ENTRYPOINT ["/usr/local/bin/entrypoint.sh"]
CMD ["python", "mypyscript.py"]

Where the entrypoint.sh looks like this:

#!/bin/bash --login
set -e

# activate conda environment and let the following process take over
conda activate myenv
exec "$@"

All credit to David R. Pugh from this post that has more details, in particular with regards to Jupyter.

5
votes

Here is how to run a script in a conda environment without buffering all standard input/output.

The --no-capture-output option is available since conda version 4.9.

FROM continuumio/miniconda3

COPY ./environment.yml ./environment.yml

RUN conda env create && conda clean --all -f --yes

ENTRYPOINT ["conda", "run", "--no-capture-output", "-n", "the_environment_name", "python", "myscript.py"]
4
votes

The problem for me was that running the command conda activate env inside docker after installing caused conda to ask me to use the conda init bash command. However, this command asks you to restart the shell, which we don't want to do inside docker. So the solution is to realize that the reason conda is asking you to restart the shell is because it has modified and wants to reload the contents of ~/.bashrc. We can do this manually and forego the need for restarting the shell using:

. ~/.bashrc

Here's the full dockerfile for those who want it:

FROM ubuntu:18.04
# update apt and get miniconda
RUN apt-get update \
    && apt-get install -y wget \
    && wget https://repo.anaconda.com/miniconda/Miniconda3-latest-Linux-x86_64.sh


# install miniconda
ENV PATH="/root/miniconda3/bin:$PATH"
RUN mkdir /root/.conda && bash Miniconda3-latest-Linux-x86_64.sh -b

# create conda environment
RUN conda init bash \
    && . ~/.bashrc \
    && conda create --name test-env python=3.7 \
    && conda activate test-env \
    && pip install ipython

Edit - deal with comment

The commenter is correct in that the above example doesn't work for switching conda environments within docker. More recently, I figured out that to get this to work, you should start every RUN command with conda init bash && .~/.bashrc && conda activate env.

Here's another example:


FROM quay.io/pypa/manylinux2014_x86_64

RUN yum install -y wget
RUN wget https://repo.anaconda.com/miniconda/Miniconda3-py39_4.9.2-Linux-x86_64.sh
RUN bash Miniconda3-py39_4.9.2-Linux-x86_64.sh -b -p /Miniconda3

RUN /Miniconda3/bin/conda create -y --name py37 python=3.7 pytest
RUN /Miniconda3/bin/conda create -y --name py38 python=3.8 pytest
RUN /Miniconda3/bin/conda create -y --name py39 python=3.9 pytest

RUN /Miniconda3/bin/conda init && bash ~/.bashrc && . ~/.bashrc

ENV conda /Miniconda3/bin/conda
ENV bashrc /root/.bashrc

# install numpy in each env
RUN $conda init && . $bashrc && conda activate py37 && pip install numpy 
RUN $conda init && . $bashrc && conda activate py38 && pip install numpy
RUN $conda init && . $bashrc && conda activate py39 && pip install numpy

3
votes

I understand that there is no one solution fits all but this is what I have been using with my Flask applications:

FROM continuumio/miniconda3

COPY environment.yml .
RUN conda env create -f environment.yml

COPY app /app
WORKDIR /app

CMD ["conda", "run", "-n", "my_env", "python", "app.py"]

The logic is very simple. First, env file is copied followed env creation. Next, application files (this is where all my Flask application files are located) are copied and using CMD the application is started by pointing to the respective env.

This is the project directory structure I used with the Dockerfile:

-- project
    -- app
        -- app.py
    -- environment.yaml
    -- Dockerfile
3
votes

For me, the solution introduced here worked seemlessly:

FROM continuumio/miniconda3
RUN conda create -n env python=3.6
RUN echo "source activate env" > ~/.bashrc
ENV PATH /opt/conda/envs/env/bin:$PATH
2
votes

I stumbled upon this question while trying to activate an environment and then installing some packages inside it. The SHELL solution did not work for me (maybe because I was attempting this on an older version of conda - 4.5.4 to be precise).

The solution is to activate the environment and run all required commands inside a new shell. Also, remember that each RUN will start a new shell that will not remember anything from the previous RUN.

FROM continuumio/miniconda3

ADD /src/environment.yml /src/environment.yml

RUN conda env create -f /src/environment.yml
ENV PATH /opt/conda/envs/mro_env/bin:$PATH
RUN /bin/bash -c "source activate mro_env \
    && conda config --add channels conda-forge \
    && conda install Jupiter \
    && conda env list"

CMD [ "python", "test.py" ]

Note every command is within "" of the same /bin/bash -c.

2
votes

Similar to other answers but with SHELL which looks more cleaner

RUN wget \
https://repo.anaconda.com/miniconda/Miniconda3-latest-Linux-x86_64.sh \
&& mkdir /root/.conda \
&& bash Miniconda3-latest-Linux-x86_64.sh -b \
&& rm -f Miniconda3-latest-Linux-x86_64.sh \
&& echo "source activate base" > ~/.bashrc

SHELL ["/bin/bash", "-c"]
ARG CONDA_ENV=env_name 
# Create env
RUN conda --version \
  && conda create -n ${CONDA_ENV} python=3.7 \
  && source activate ${CONDA_ENV}