How to integrate log analytics workspace with Azure Databricks notebook for monitoring databricks notebook(Custom logging)?

Question

I created notebooks in Azure databricks workspace and want to monitor my notebooks by using Log Analytics workspace but I could not see any log metrics for databricks notebook because databricks is a third-party tool. Like as we create log metrics monitoring for Azure SQL database by using log analytics with SQL analytics. After following few docs and blogs I got to know there is no inbuilt feature in log analytics for monitoring databricks notebook, I have to write custom code(using Python) for monitoring databricks notebook. Team, could you please help me here if anyone has implemented the same and have some idea. I followed this link:

https://docs.microsoft.com/en-us/azure/databricks/administration-guide/account-settings/azure-diagnostic-logs

Answer 1

Have you tried the steps mentioned in this document: “Diagnostic logging in Azure Databricks”?

By configuring Diagnostic setting, you can configure the collection of following data:

{ dbfs, clusters, accounts, jobs, notebook,ssh, workspace, secrets, sqlPermissions, instancePools}

STEP1: Make sure you have configured the diagnostic setting.

STEP2: After configuring the diagnostic setting, you can go to Log Analytics Workspace => Logs => Log Management => You will find Databricks Notebook => Run the below query to get the details about the notebook.

DatabricksNotebook 

| where TimeGenerated > ago(24h) 

| limit 10

STEP3: You can select any one from the results and check all the details regarding the notebook.