Copying single csv with multiple schema in ADF

Question

Hi I am trying to sink csv file to databse using ADF and Data Flow CSV file structure as below r1c1,r1c2,r1c3,r1c4

r2c1,r2c2,r2c3,r2c4

r3c1,r3c2,r3c3,r3c4

r4c1,r4c2,r4c3,r4c4,r4c5,r4c6

r5c1,r5c2,r5c3,r5c4,r5c5,r5c6

so 2 schemas one with 4 columns and second with 6 columns. My goal is to copy first 3 rows in one table and last 2 in second table. Please guide me to acheive this in ADF.

Answer 1

Your useCase is typical for ConditionalSplit transformation. here is a simple demo that i created to demonstrate your usecase.

1. add 2 schemas as datasets in ADF
2. create a pipeline with Data Flow activity
3. in DataFlow : add your souce (CSV data) , i put Null values in the first 3 rows in columns 5 and 6 as you requested.
4. in Conditional split , i added this condition : "isNull(Column_5) && isNull(Column_6)" - see screenshot below.
5. Rows that matches the condition will be saved in the matching sink.

Data Flow Activities:

Conditional Split Activity:

Rows 1 - 3 output :

Rows 5 to 6 output:

Read more about it here : https://learn.microsoft.com/en-us/azure/data-factory/data-flow-conditional-split