Azure Data Factory Get Metadata activity returning "(404) not found" error when getting column count

Question

I am trying to implement a Get Metadata activity to return the column count of files I have in a single blob storage container.

Get Metadata activity is returning this error: Error

I'm fairly new to Azure Data Factory and cannot solve this. Here's what I have:

Dataset:Source dataset

Name- ten_eighty_split_CSV

Connection- Blob storage

Schema- imported from blob storage file

Parameters- "FileName"; string; "@pipeline().parameters.SourceFile"

Pipeline:

Name: ten eighty split

Parameters: "SourceFile"; string; "@pipeline().parameters.SourceFile"

Settings: Concurrency: 1

Get Metadata activity: Get Metadata

Only argument is "Column count"

Throws the error upon debugging. I am not sure what to do, (404) not found is so broad I could not ascertain a specific solution. Thanks!

Answer 1

• The error occurs because you have given incorrect file name (or) name of a file that does not exist.

• Since you are trying to use blob created event trigger to find the column count, you can use the procedure below:

• After configuring the get metadata activity, create a storage event trigger. Go to Add trigger -> choose trigger -> Create new.

• Click on continue. You will get a Trigger Run Parameters tab. In this, give the value as @triggerBody().fileName.

• Complete the trigger creation and publish the pipeline. Now whenever the file is uploaded into your container (on top of which you created storage event trigger), it will trigger the pipeline automatically (no need to debug). If the container is empty and you try to debug by giving some value for sourceFile parameter, it would give the same error.
• Upload a sample file to your container. It will trigger the pipeline and give the desired result.

The following is the trigger JSON that I created for my container:

{
    "name": "trigger1",
    "properties": {
        "annotations": [],
        "runtimeState": "Started",
        "pipelines": [
            {
                "pipelineReference": {
                    "referenceName": "pipeline1",
                    "type": "PipelineReference"
                },
                "parameters": {
                    "sourceFile": "@triggerBody().fileName"
                }
            }
        ],
        "type": "BlobEventsTrigger",
        "typeProperties": {
            "blobPathBeginsWith": "/data/blobs/",
            "blobPathEndsWith": ".csv",
            "ignoreEmptyBlobs": true,
            "scope": "/subscriptions/b83c1ed3-c5b6-44fb-b5ba-2b83a074c23f/resourceGroups/<user>/providers/Microsoft.Storage/storageAccounts/blb1402",
            "events": [
                "Microsoft.Storage.BlobCreated"
            ]
        }
    }
}