While mounting S3 Bucket on the DataBricks whic is on GCP environment getting error "Key based mount points are not supported"

Question

Here is the code:

access_key = dbutils.secrets.get(scope = "dll-gcp", key = "aws-access-key")
secret_key = dbutils.secrets.get(scope = "dll-gcp", key = "aws-secret-key")
encoded_secret_key = secret_key.replace("/", "%2F")
aws_bucket_name = "ls-ind-export"
mount_name = "s3"

dbutils.fs.mount(f"s3a://{access_key}:{encoded_secret_key}@{aws_bucket_name}", f"/mnt/{mount_name}")
display(dbutils.fs.ls(f"/mnt/{mount_name}"))

ExecutionError: An error occurred while calling o353.mount. : java.rmi.RemoteException: java.lang.UnsupportedOperationException: Key based mount points are not supported.; nested exception is:

Answer 1

It's a known limitation of Databricks on GCP - in contrast to Azure & AWS, it doesn't support mounts using AWS keys, Azure service principals, etc. Instead it's recommended to access data directly via s3a://.../... URL. Example from documentation:

access_key = dbutils.secrets.get(scope = "aws", key = "aws-access-key")
secret_key = dbutils.secrets.get(scope = "aws", key = "aws-secret-key")
sc._jsc.hadoopConfiguration().set("fs.s3a.access.key", access_key)
sc._jsc.hadoopConfiguration().set("fs.s3a.secret.key", secret_key)

# If you are using Auto Loader file notification mode to load files, provide the AWS Region ID.
aws_region = "aws-region-id"
sc._jsc.hadoopConfiguration().set("fs.s3a.endpoint", "s3." + aws_region + ".amazonaws.com")

myRDD = sc.textFile("s3a://%s/.../..." % aws_bucket_name)
myRDD.count()