I am trying to create a multistage docker image for a python conda project using conda-pack with the following Dockerfile on MacOS:
### build stage
FROM continuumio/miniconda3 AS build
COPY environment.yaml .
# Create the conda environment and install conda-pack
RUN conda env create -f environment.yaml
RUN conda install -c conda-forge conda-pack
# Use conda-pack to create a standalone enviornment in /venv:
RUN conda-pack -n test -o /tmp/env.tar && \
mkdir /venv && cd /venv && tar xf /tmp/env.tar && \
rm /tmp/env.tar
RUN /venv/bin/conda-unpack
### runtime stage
# Use the official Miniconda3 image as the base image
FROM continuumio/miniconda3
# Copy /venv from the previous stage:
COPY --from=build /venv /venv
And the following environment.yaml file:
name: test
dependencies:
- python=3.8.10
- pip=20.3
- pip:
- setuptools==59.5.0
As a result, I am getting the following error:
> [build 5/6] RUN conda-pack -n test -o /tmp/env.tar && mkdir /venv && cd /venv && tar xf /tmp/env.tar && rm /tmp/env.tar:
#9 2.287 Collecting packages...
#9 2.287 CondaPackError:
#9 2.287 Files managed by conda were found to have been deleted/overwritten in the
#9 2.287 following packages:
#9 2.287
#9 2.287 - setuptools 65.6.3:
#9 2.287 lib/python3.8/site-packages/pkg_resources/_vendor/__pycache__/zipp.cpython-38.pyc
#9 2.287 lib/python3.8/site-packages/pkg_resources/_vendor/importlib_resources/__init__.py
#9 2.287 lib/python3.8/site-packages/pkg_resources/_vendor/importlib_resources/__pycache__/__init__.cpython-38.pyc
#9 2.287 + 187 others
#9 2.287
#9 2.287 This is usually due to `pip` uninstalling or clobbering conda managed files,
#9 2.287 resulting in an inconsistent environment. Please check your environment for
#9 2.287 conda/pip conflicts using `conda list`, and fix the environment by ensuring
#9 2.287 only one version of each package is installed (conda preferred).
It looks like there is an issue with the version 59.5.0 of setuptools which I am trying to install with pip... I only get the error when using conda-pack, but at the same time it is the way I have found how I should be able to reduce the image size.
I have tried to reinstall conda (current conda version is 23.1.0) but I am still getting the error. How can I solve it? My goal is to be able to build a multistage docker image of my Conda project and use conda-pack to be able to lower down the size of the Docker image.
The point of the multi-stage docker build for your conda approach is to no longer need conda OR python installed in your runtime image.
However, you are using conda as the runtime image which will reinstall all the libraries you are trying to unpack. My guess is if you switch your runtime image to something lighter (ubuntu, debian, alpine, etc.) it will run just fine.
As for the issue you see above, adding a --ignore-missing-files flag to your conda-pack command may do the trick.