cloud function update oracle onprem db for large volume of data

Question

Our application has the usecase to upload 20TB of files in GCP bucket. Once its uploaded to GCP bucket, the corresponding bucket path should be updated to a onprem oracle db. will the GCP cloud function or GCP pub sub will support this volume of data ?

Need some solution to update the GCP path in oracle on prem db for large volume of data

Answer 1

A such volume of data can't be handle in a single piece by Cloud Functions (max 32Gb of memory) or PubSub (max 10Mb per message).

But Cloud Functions/PubSub can be use has orchestrator/notifier

• Your application create a 20Tb file in Cloud Storage
• Cloud Functions is invoked on the Finalized event on Cloud Storage
• The cloud function create a PubSub message
• You have a OnPrem environment that pull the PubSub subscription (push is also possible, but often more complex with your sec team) and when a message is detected, the 20Tb file downloaded locally (you can imagine that the Cloud Functions create a signed URL and put it in the message to allow your OnPrem environment downloading the file without authentication), and then load it on Oracle.