Run docker image as prefect task or flow

Question

My question is effectively a duplicate of this question however the only answer is related to Prefect 1. I would like to know if anyone is aware of how to do such a thing in Prefect 2.

I am able to execute Python in a Docker image however I'm unable to find any examples of how to simply execute a docker image as a task or flow similar to executing a docker run.

Thank!

Answer 1

1. Create a docker image with your python flows inside it.

FROM prefecthq/prefect:2.11.5-python3.11

WORKDIR /root/flows
COPY pyproject.toml .
RUN pip install .

COPY src ./src
RUN pip install .

ENV PREFECT_API_URL=http://server:4200/api

CMD ["python", "src/my_flows.py"]

2. Build it using docker build -t my-flows .
3. Use a docker compose similar to this. Note the usage my-flows image you built in step 2.

# https://github.com/rpeden/prefect-docker-compose
version: "3.9"
services:

 ### Prefect Database
 database:
   image: postgres:15.2-alpine
   restart: always
   environment:
     - POSTGRES_USER=postgres
     - POSTGRES_PASSWORD=postgres
     - POSTGRES_DB=prefect
   expose:
     - 5432
   volumes:
     - db:/var/lib/postgresql/data
   profiles: ["server"]


 ### Prefect Server API and UI
 server:
   image: prefecthq/prefect:2.11.5-python3.11
   restart: always
   volumes:
     - prefect:/root/.prefect
   entrypoint: ["/opt/prefect/entrypoint.sh", "prefect", "server", "start"]
   environment:
     - PREFECT_UI_URL=http://127.0.0.1:4200/api
     - PREFECT_API_URL=http://127.0.0.1:4200/api
     # If you want to access Prefect Server UI from anywhere other than the Docker host machine, you will need to change
     # PREFECT_UI_URL and PREFECT_API_URL to match the external hostname/IP of the host machine. For example:
     #- PREFECT_UI_URL=http://external-ip:4200/api
     #- PREFECT_API_URL=http://external-ip:4200/api
     - PREFECT_SERVER_API_HOST=0.0.0.0
     - PREFECT_API_DATABASE_CONNECTION_URL=postgresql+asyncpg://postgres:postgres@database:5432/prefect
     # Uncomment the following line if you want to use the 'S3 Bucket' storage block instead of the older 'S3' storage
     # - EXTRA_PIP_PACKAGES=prefect-aws
   ports:
     - 4200:4200
   depends_on:
     - database
   profiles: ["server"]

 ## Prefect Agent
 agent:
   image: prefecthq/prefect:2.11.5-python3.11
   restart: always
   entrypoint: ["/opt/prefect/entrypoint.sh", "prefect", "agent", "start", "-q", "YOUR_WORK_QUEUE_NAME"]
   environment:
     - PREFECT_API_URL=http://server:4200/api
#       Use PREFECT_API_KEY if connecting the agent to Prefect Cloud
#     - PREFECT_API_KEY=YOUR_API_KEY
   profiles: ["agent"]

 ### Prefect Flow CLI
 flows:
   image: "my-flows"
   entrypoint: ["python", "src/my_flows.py"]
   working_dir: "/root/flows"
   volumes:
     - "/data/:/data/"
   environment:
     - PREFECT_API_URL=http://server:4200/api
#       Use PREFECT_API_KEY to use the CLI to interact with Prefect Cloud
#     - PREFECT_API_KEY=YOUR_API_KEY
   profiles: ["flows"]

volumes:
 prefect:
 db:
networks:
 default:
   name: prefect-network

4. Now bring up the server, agent, db etc using docker-compose --profile server up
5. Next you can run the cli container with your flows using docker-compose run flows