How to Connect ADO REST API connection using with Azure data factory pipeline

Question

How to connect ADO REST API connection using with Azure data factory pipeline linked service and dataset.

I'm trying to create Dataset for REST connect using azure data factory, unable to get preview the data, getting error with Rest call failed with client error,status code 401 Unauthorised,please check your activity settings.

Answer 1

In ADF pipeline, you can use the Web activity to call the Azure DevOps REST API. The "401 Unauthorized" error should be caused that you provided an invalid authentication credentials for the request.

---

If the Azure DevOps organization you want to access via the REST API has been connect to the AAD (Microsoft Entra) tenant, it is recommended selecting "Service Principal" or "Managed Identity" as the authentication method on the Settings of the Web activity.

Below are the steps to use "Service Principal" as the authentication:

1. Prerequisites in AAD (Microsoft Entra) tenant.

   • Go to Microsoft Entra ID > App registrations to create a Service Principal in the tenant if you do not have one.

   • Open the Service Principal, go to Certificates & secrets > Client secrets tab to create a client secret for the Service Principal if there is no existing valid client secret. Copy and remember the value of the client secret, because the value will be masked as secret (******) once you refresh/reopen the page. You can store it as a secret in Azure Key Vault.

2. Prerequisites in Azure DevOps organization.

   • Go to Organization Settings > Microsoft Entra, ensure the organization has connected to the AAD tenant which the Service Principal is in.

   • Go to Organization Settings > Users, search and add the Service Principal as a user into the organization.

   • After successfully adding the Service Principal into the organization, add it into a security group so that you can manage the permissions of the Service Principal via controlling the permissions of that group in the organization.

3. In the ADF pipeline, on the Settings tab of the Web activity, you can set like as below image shows.

• Tenant: The ID (Directory (tenant) ID) of the tenant.
• Service principal ID: The ID (Application (client) ID) of the Service Principal.
• Service principal key: the value of the client secret generated above for the Service Principal. You can load the value from Azure Key Vault if you have stored it as a secret in Azure Key Vault.
• Resource: The scope specifically for Azure DevOps resources. The value is fixed (499b84ac-1321-427f-aa17-267ca6975798).

---

If the Azure DevOps organization does not connect to any tenant, you can set "Basic" as the authentication method on the Settings of the Web activity.

• Username: The username of the user who have the access to the Azure DevOps organization. Ensure the user has the necessary permissions to access the required resources in the Azure DevOps organization.
• Password: The PAT (Personal Access Token) of the user generated in Azure DevOps. Ensure the PAT has the necessary scopes to access the required resources in the Azure DevOps organization. You can store the PAT as a secret in Azure Key Vault.

---