Integrating Airbyte as a multi-container app in the main docker-compose.yml

Question

I am building a data pipeline with Airbyte, PostgreSQL and dbt. PostgreSQL and DBT I can easily set up via my main docker-compose.yml but with Airbyte I am not sure. Airbyte itself is a multi-container app so it has its own docker-compose.yml.

To deploy Airbyte, I follow the steps here and run it with the script ./run-ab-platform.sh. I am not sure though how to integrate this into the main docker-compose.yml.

Current docker-compose.yml structure:

version: "3.8"

services:
  db:
    container_name: postgres
    image: postgres:latest
    restart: always
    environment:
      - POSTGRES_USER=postgres
      - POSTGRES_PASSWORD=mysecretpassword
      - POSTGRES_DB=training
    ports:
      - '5432:5432'
    volumes:
      - ./postgres:/var/lib/postgresql/data
    healthcheck:
      test: ["CMD-SHELL", "pg_isready -U postgres"]
      interval: 60s
      timeout: 30s
      retries: 5
   
  dbt:
    image: fishtownanalytics/dbt:latest
    environment:
      DBT_PROFILES_DIR: /dbt/profile
    volumes:
      - ./my_dbt_profiles:/dbt/profile
    depends_on:
      - postgres

I already tried to use build or extend but can't seem to get it right.

Answer 1

Probably you can use extends as below:

services:
  web:
    extends:
      file: common-services.yml
      service: webapp

Here you can find relevant docs.

Answer 2

I guess the best way for you would be using Airbyte's connectors for PostgreSQL, thus configuring them to move data between your PostgreSQL databases. However, I'm not so sure of which way you are using to set up your Airbyte containerized app. Is it in your local environment (https://docs.airbyte.com/deploying-airbyte/local-deployment). If so, you should get the way to connect Airbyte with your other services.

As I remember, Airbyte also uses a Postgres service. So, you will likely get some port conflict using port 5432.