Docker: keep python packages consistent

Question

I want to run docker in my homelab for any python related projects. To save diskspace and keep installed python packages across dockers consistent, I am looking for a way to share the installed python packages across the different services.

Services:

• Juypterlab (to create python scripts)
• Kestra (for automated execution of scripts)
• Couple of Flask apps

Expectation:
If I install pip install pandas in Jupyterlab, this package will also be available immediately in the Kestra container

I am quite new to docker and docker-compose.

Answer 1

You can use Docker Volumes

Volumes can be more safely shared among multiple containers.

You can install your libraries on Volume.

You can define Volumes on dockerfile, docker-compose file or on cli

But to share volumes, you should define named volumes.

You can do so using cli or docker-compose file.

cli

docker volume create --name shared-volume
docker run -d --name container1 -v shared-volume:/path/in/container1 my-image1
docker run -d --name container2 -v shared-volume:/path/in/container2 my-image2

docker-compose file

services:
 service1:
    image: my-image1
    volumes:
      - shared-volume:/path/in/container1
 service2:
    image: my-image2
    volumes:
      - shared-volume:/path/in/container2
volumes:
 shared-volume

A named volume is declared for the path /path/in/container(1/2). Docker will create a volume, named shared-volume, (folder on your pc) and mount it to /path/in/container(1/2) inside the container. This means that any data written to /path/in/container(1/2) inside the container will be stored in the volume and persist even after the container is stopped or removed.

image from docker docs link

Beware containers might overwrite each other's data stored on the volume.