What connector in Azure Data Factory can we use to connect to Log Analytics Workspace? My requirement is to read the "AzureActivity" table & write to a storage account as a parquet file. The reason why I want to use ADF to read from Log Analytics Workspace tables rather than the Activity Log directly is that the Activity Log Json files are already parsed by the ETL process that runs in Log Analytics Workspace, and when we export the tables from there, the JSON output is not that raw & complex to read. Also, if I get the JSON file converted to Parquet, then I already have a Databricks pipeline which reads the parquet files and populates Delta Lake tables. On the top of Databricks Delta Lake tables, I will be producing reports on Azure Activity, and many more diagnostic logs in future.
Using Azure Data Factory to read from Log Analytics Tables
1.4k views Asked by Noor Basha Shaik At
1
There are 1 answers
Related Questions in AZURE-DATA-FACTORY
- Difficulty creating a data pipeline with Fabric Datafactory using REST
- Accessing REST API Status Codes using Azure Data Factory Copy Activity (or similar)?
- Use an activity output as the step name to get additional info in ADF
- Exit loop condition when running the synpase notebooks based on metadata dependencies
- Azure Data Factory Copy Activity Only Importing First Row of XML file
- ADF Copy Activity from Source Azure Synapse Analytics Target ADLSGen2 Storage account
- Parmeter values not resolving in ADF
- How to copy XML files in a folder F1 based on whether its content is present on folder F2 (disregarding file names)
- Can I move an Azure Data Factory Pipeline to Azure DevOps?
- tsql functions like REPLACE() failing in azure data factory pipeline connected to salesforce
- Get the URL from C# script used in ssis
- Reading Unstructured Text from the entire file in Azure Data Factory
- Unable to PUT JSON using ADF Dataflow, the error is "the JSON value could not be converted to System.Collections.Generic.List"
- Manipulating Json in Azure Data Factory activities
- Couchbase Connector in ADF
Related Questions in AZURE-DATABRICKS
- ingesting high volume small size files in azure databricks
- includeExistingFiles: false does not work in Databricks Autoloader
- Problem to add service principal permissions with terraform
- Filter 30 unique product ids based on score and rank using databricks pyspark
- Tools for fast aggregation
- How to find location of all tables under a schema in databricks
- extraneous input 'timestamp' expecting {<EOF>, ';'}(line 1, pos 54)
- How to avoid being struct column name written to the json file?
- Understanding least common type in databricks
- Azure DataBricks - Looking to query "workflows" related logs in Log Analytics (ie Name, CreatedBy, RecentRuns, Status, StartTime, Job)
- Updating a Delta Tables using a "change feed like" JSON every day
- Issue with Databricks Workspace Conversion: Python Files Automatically Converted to Notebooks upon Merge
- use the output of notebook1 in notebook2
- Unable to read data from ADLS gen 2 in Azure Databricks
- Combine SQL cell output in a markdown cell in Databricks Notebook
Related Questions in AZURE-LOG-ANALYTICS
- Is there a way to view traffic logs for Azure Storage for connections that got blocked by Firewall settings from Networking pane?
- Not able to view the logs for hosted WebApp in Azure
- Azure DataBricks - Looking to query "workflows" related logs in Log Analytics (ie Name, CreatedBy, RecentRuns, Status, StartTime, Job)
- Sent the Postgresql query result to Log Analytics workspace to create custom metric in Azure Monitor
- Databricks Log Analytics Logs missing log
- Azure hide not mine logs - see only my logs
- KQL - How to enrich an event by matching an IP address to an IP range from a Sentinel Watchlist?
- How to Plot Pre-Averaged Time Series Data in KQL Without Using Summarize?
- Logic Apps: Run query and visualize results Html Table displayed in Email
- How can I use kusto to show which permissions are being used by which users on the data plane
- Sending log analytics workspace logs to Sentinel
- Unable to receive metrics on Log Analytics workspace from a Windows VM in Azure
- Azure Heartbeat Table: _ResourceId is blank
- Log Analytics Workspace / Azure Watchlist: KQL Filtering on datetime
- How can I keep on logging after retirement of instrumentationkey and classic application insights?
Related Questions in ACTIVITYLOG
- How to retrieve vmid of a deleted azure VM
- How to lookup who created/edited a secret in Azure Portal's Keyvault?
- How can we add a add extra column in activity log table ( spatie-activitylog )
- How to find when my vm was resized in GCP?
- Mass Update spatie/ActivityLog Laravel Better Solution?
- Activity log for specific azure resource
- How to retrieve the most recent modified (activity)time and the ResourceName(Id) of an AzResource that has been active for the last 90 days PowerShell
- Finding the times that most people were in a chat room
- Using Azure Data Factory to read from Log Analytics Tables
- Azure Error "The template deployment failed because of policy violation. Please see details for more information."
- Laravel Spatie - Custom Attribute value when audit log via modal
- Is there a way for the Azure Activity Log to track changes to Tags?
- How sensitive is the data in Data Access and Access Transparency Cloud Audit Logs?
- Laravel Activity Log won't work on Update
- GCP IAM User Deletion Log
Popular Questions
- How do I undo the most recent local commits in Git?
- How can I remove a specific item from an array in JavaScript?
- How do I delete a Git branch locally and remotely?
- Find all files containing a specific text (string) on Linux?
- How do I revert a Git repository to a previous commit?
- How do I create an HTML button that acts like a link?
- How do I check out a remote Git branch?
- How do I force "git pull" to overwrite local files?
- How do I list all files of a directory?
- How to check whether a string contains a substring in JavaScript?
- How do I redirect to another webpage?
- How can I iterate over rows in a Pandas DataFrame?
- How do I convert a String to an int in Java?
- Does Python have a string 'contains' substring method?
- How do I check if a string contains a specific word?
Popular Tags
Trending Questions
- UIImageView Frame Doesn't Reflect Constraints
- Is it possible to use adb commands to click on a view by finding its ID?
- How to create a new web character symbol recognizable by html/javascript?
- Why isn't my CSS3 animation smooth in Google Chrome (but very smooth on other browsers)?
- Heap Gives Page Fault
- Connect ffmpeg to Visual Studio 2008
- Both Object- and ValueAnimator jumps when Duration is set above API LvL 24
- How to avoid default initialization of objects in std::vector?
- second argument of the command line arguments in a format other than char** argv or char* argv[]
- How to improve efficiency of algorithm which generates next lexicographic permutation?
- Navigating to the another actvity app getting crash in android
- How to read the particular message format in android and store in sqlite database?
- Resetting inventory status after order is cancelled
- Efficiently compute powers of X in SSE/AVX
- Insert into an external database using ajax and php : POST 500 (Internal Server Error)
There is not direct/native connector to read data from Log Analytics. Possible option is to retrieve data from the Log Analytics REST APIs by using REST connector in ADF.
Here is an article by a community volunteer which has detailed steps on how you can get data from Log analytics tables using ADF.
Article: Retrieving Log Analytics Data with Data Factory
In case if you have complex JSON from your Log Analytics tables, then you may have to use Mapping Data flow instead of Copy activity to flatten the complex/nested JSON data.