Azure Data Factory: Copy file mentioned in JSON

Question

I have a JSON where one the parts is a filename.

{
    "dial_target_file_row_count": 2440,
    "dial_target_file_endpoint": "consumerzone@dialconsumingp01dls",
    "dial_target_file_full_path": "abfss://*****/data/snapshotdate=2023-03-25",
    "dial_metadata_file_location": "abfss://*****/metadata/",
    "dial_metadata_file_name": "dsapp_goldenDataset_2023-03-25_20230326075008_metadata.json",
}

Now I want to copy the file 'dail_target_file_full_path' onto my blob storage. How do I define the source and sink (the sink name should be a part of the original filename).

I don't see any option to have a dynamic filename

Answer 1

Add a lookup activity to your pipeline to get the content of the JSON with the file details. If this is one file, check "first row only". Add a copy data activity, and create a dataset with parameters:

In the copy data source, you can now add values for this parameters, which should be the result of the first JSON: Here are the values I provided in dynamic content for this parameters:

@activity('Lookup1').output.firstRow.dial_metadata_file_location
@activity('Lookup1').output.firstRow.dial_metadata_file_name

You will need to get just a part of the file path for the folder (without the url part)

Answer 2

Adding to @Chen Hirsh, you can extract the container name, folder path and file name using variables like below from the lookup output. Here I have not checked first row only in the lookup.

For this, I have copied a sample file to the above path(it means I am doing for the copy activity sink and you can change this as per your requirement).

Use the below dynamic content for the container name from dial_target_file_full_path abfss path.

@substring(activity('Lookup1').output.value[0].dial_target_file_full_path, add(indexOf(activity('Lookup1').output.value[0].dial_target_file_full_path, '//'), 2),sub(indexOf(activity('Lookup1').output.value[0].dial_target_file_full_path, '@'),add(indexOf(activity('Lookup1').output.value[0].dial_target_file_full_path, '//'), 2)))

For folder path:

@split(activity('Lookup1').output.value[0].dial_target_file_full_path,'.net')[1]

For file name:

@activity('Lookup1').output.value[0].dial_metadata_file_name

Create dataset parameters for container name, folder path and file name and give those in dataset path. Then use these in copy activity dataset parameters.(Here I have used it for sink).

Result: