Pulling Marketo Data to S3 using Airflow

Question

I am looking for a process to pull Marketo data using Airflow. It seems that Airflow has no native Marketo webhook, did I understand that correctly ?

Or is there a way to use an existing Airflow operator/webhook to pull data from Marketo? Thanks

Answer 1

Most Operators for 3rd parties just wrap around APIs, Marketo has a REST api. You could use a PythonOperator with requests pull your data.

An example DAG could be:

import pendulum
from airflow.decorators import dag, task
from airflow.providers.amazon.aws.operators.s3 import S3CreateObjectOperator


@dag(
    schedule_interval=None,
    start_date=pendulum.yesterday(tz="Europe/London"),
    catchup=False,
)
def marketo_api_dag():
    @task.virtualenv(system_site_packages=False, requirements=["requests"])
    def pull_from_marketo():
        import requests
        import json

        response = requests.get()  # pull from marketo API using requests
        return json.dumps({})  # response formatted how you need it

    marketo_api_response_data = pull_from_marketo()

    save_marketo_to_s3 = S3CreateObjectOperator(
        task_id="save_marketo_to_s3", s3_bucket="s3bucket", s3_key="/path/to/save", data=marketo_api_response_data
    )

    marketo_api_response_data >> save_marketo_to_s3


marketo_api_dag_run = marketo_api_dag()

• requires the Amazon provider apache-airflow-providers-amazon.