Azure DevOps pipeline integration with Databricks + how to print Databricks notebook result on pipeline result screen

Question

Am trying to integrate Azure Databricks within DevOps pipeline and used following URL: https://menziess.github.io/howto/run/databricks-notebooks-from-devops/

1. At Azure Databricks, created a job on top of my notebook
2. Used a separate VM with preinstalled Databricks-CLI and set it as a self-hosted pool agent
3. Prepared a YAML script in pipeline, which calls the above databricks job on my self hosted agent pool

My Pipeline and databricks jobs are executing successfully.

But I want to extract the output generated by notebook inside pepeline and print on console. In this way anyone can trigger the pipeline by passing their parameters and view the result on console screen. Notebook mostly returns output as table data (e.g. select * from table).

Could someone help me with this.

Answer 1

we can try to use the powerShell script to set the result as the variable, then we can output the data to a file. This can be a work around. At present, in the pipeline we can not pass their parameters and view the result on console screen.