The issue I am encountering is how to create an SSH tunnel using IAP for AlloyDB export in Airflow hosted on Google Composer.
I am replacing a CloudSQL connection with AlloyDB. The composer environment is on a different network than my AlloyDB instance, so I know I need to use a bastion VM as listed here.
On local, I am able to port forward to my alloydb-proxy-vm and run my Airflow process. I am portforwarding in my Terminal. Thus, my VM should be properly configured.
To build the functionality into my Airflow, I tried the BashOperator like this:
SSH_START = BashOperator(
task_id='create_ssh_tunnel',
bash_command="""
gcloud auth activate-service-account --key-file=/opt/airflow/np-serviceaccount.json
gcloud config set project np-myproject
gcloud compute ssh alloydb-psql-client \
--tunnel-through-iap \
--zone=us-central1-b \
--ssh-flag="-T -L 0.0.0.0:5432:localhost:5432"
""",
dag=dag
)
SSH_START >> ALLOYDB_EXPORT (using PostgresTOGCSOperator)
However, the issue with this is that I am simply opening and closing a tunnel as a task, I am not enabling the task that needs the connection to use it.
Should I be looking to use the ComputeEngineSSHHook or something similar instead?
Any advice would be greatly appreciated.