Why is my minio python is not able to download non-text file types?

28 views Asked by At

I have a self-hosted Minio.

Problem: Unable to download .csv and .jpg files. Able to download .txt and .md files.

This is strange and I have been bashing my head in.

My bucket policies are extremely simple, pretty much open to any operations.

No where in my minio setup/settings, I do not have file type restrictions.

I am able to download any file type through the web portal. Unable to download the .csv, .jpg, and .tar files programmatically.

I tried manually uploading the files via the web portal and then running it programmatically, doesnt work. Only works for .md and .txt files.

I tried programmatically uploading and then downloading from the same access key, doesnt work. Only works for .md and .txt files.

I am at my wits end. Been at this for hours, cant take it anymore. Exhausted all of chatgpt help. Stackoverflow is my last attempt - trust me, I would rather be elsewhere.

Code:

class MinioUploader:

    def __init__(self):
        self.minioClient = Minio(
            endpoint="myminio.website.com",
            region="us-east-1",
            access_key="myaccesskeywhichiwillnotputhere",
            secret_key="mysecretketywhichiwillnotputhere",
            secure=True
        )
    def download_all_files(self, bucket_name, local_path="./storage/local_storage/"):
        if self.minioClient is None:
            logs_sys.error("Minio client is not initialized.")
            return

        if not self.minioClient.bucket_exists(bucket_name):
            logs_sys.error(f"Bucket: {bucket_name} does not exist")
            return

        local_path = os.path.join(local_path, bucket_name)
        if not os.path.exists(local_path):
            os.makedirs(local_path)
        print(f"local_path: {local_path}")

        objects = self.minioClient.list_objects(bucket_name, recursive=True)
        for obj in objects:
            try:
                print(obj.object_name)
                # Decode any percent-encoded characters in the object name
                decoded_object_name = unquote(obj.object_name)
                # Create a safe, absolute file path
                safe_local_path = os.path.join(local_path, *decoded_object_name.split('/'))
                # Ensure the directory for the file exists
                safe_local_path = safe_local_path.replace("\\", "/")
                os.makedirs(os.path.dirname(safe_local_path), exist_ok=True)
                # Download the object
                print(f"Downloading {obj.object_name} to {safe_local_path}")
                self.minioClient.fget_object(bucket_name=bucket_name, object_name=obj.object_name, file_path=safe_local_path)
            except Exception as e:
                logs_sys.error(f"Error: {e}")
                print(f"Error: {e}")

Print Out:

D:\TRADING_RESTORE_STRATEGIES>python -u "d:\TRADING_RESTORE_STRATEGIES\storage\minio_s3\minio_client.py"
local_path: ./storage/local_storage/testdownloadbucket
11.md
Downloading 11.md to ./storage/local_storage/testdownloadbucket/11.md
3-7-2024 3-11-49 AM.jpg
Downloading 3-7-2024 3-11-49 AM.jpg to ./storage/local_storage/testdownloadbucket/3-7-2024 3-11-49 AM.jpg
Error: S3 operation failed; code: AccessDenied, message: Access denied, resource: /testdownloadbucket/3-7-2024%203-11-49%20AM.jpg, request_id:
 17BACF23766A4250, host_id: dd9025af9251148b658df7ac2e3e8, bucket_name: testdownloadbucket, object_name: 3-
7-2024 3-11-49 AM.jpg
asad.txt
Downloading asad.txt to ./storage/local_storage/testdownloadbucket/asad.txt
new.csv
Downloading new.csv to ./storage/local_storage/testdownloadbucket/new.csv
Error: S3 operation failed; code: AccessDenied, message: Access denied, resource: /testdownloadbucket/new.csv, request_id: 17BACF23839DECB7, h
ost_id: dd9025af9251148b658df7ac2e3e8, bucket_name: testdownloadbucket, object_name: new.csv
something1.csv
Downloading something1.csv to ./storage/local_storage/testdownloadbucket/something1.csv
Error: S3 operation failed; code: AccessDenied, message: Access denied, resource: /testdownloadbucket/something1.csv, request_id: 17BACF238744
C16C, host_id: dd9025af9251148b658df7ac2e3e8, bucket_name: testdownloadbucket, object_name: something1.csv
test.txt
Downloading test.txt to ./storage/local_storage/testdownloadbucket/test.txt
<minio.api.Minio object at 0x00000248C9B0B130>
0

There are 0 answers