How to dynamically pass ADLS gen2 folder/filename to Databricks Notebook in ADF or Databricks

0
votes

I am using a Databricks Notebook Activity in ADF to transform files in ADLS gen2 folder. This folder is dynamic and a new folder is created on daily basis with daynumber. So I want my Databricks Notebook Activity to pickup foldername dynamically for each day to process files in that folder.

Can we do this ADF or within Databricks Notebook Activity?

1
azureazure-data-factory-2azure-databricks

1 Answers

2
votes

Have you tried to add dynamic content to the Notebook path?
I've made a test according to the tutorial and it works well:

  1. First, I declare a parameter name. enter image description here

  2. Switch to the Setting tab, I add the expression @concat('/adftutorial/',string(dayOfMonth(utcnow()))) to the Notebook path text enter image description here

  3. Then I run debug and enter /path/filename to the parameter enter image description here

  4. It will read files in the dynamic path I specified before.
    enter image description here

In my case, the file doesn't exist so it pops up an error.