Azure Data Explorer - Continuous data export AAD token generation process

Question

We need to set up continuous data export from Kusto to an external table for data older than 60 days into ADLS. Based on documentation it looks like we should use AAD token to Productionize, however, the documentation does not clearly specify the process to generate AAD token.

We also followed the Microsoft Documentation to acquire the access token and registered an application, generated client secret key.

Need some help/suggestion regarding the process to generate the token

Append ;token=AadToken to the URI, with AadToken being a base-64 encoded AAD access token (make sure the token is for the resource https://storage.azure.com/).

.create external table logs (ing_dt:date,record:string)
kind=adl
partition by bin(ing_dt, 1d)
dataformat=json
(
h@'abfss://filesystem@<storageaccountname>.dfs.core.windows.net/input;token=*****)
with 
(
   docstring = "External Table",
   folder = "External",
   nameprefix= "ext"
 )

 .create-or-alter continuous-export exp_logs 
to table logs 
with 
(intervalBetweenRuns=1h) 
<| source
| extend ing_dt=format_datetime(ingestion_time(),'yyyy-MM-dd')
| project todatetime(ing_dt), record
| where todatetime(ing_dt) < ago(60d)

yifats yifats · Accepted Answer · 2020-06-19T21:58:01

The documentation you linked to references AAD client libraries to generate access token (see this section). For example, AcquireToken method if you're using .NET. However, AAD access tokens have a very short expiry (can be extended up to 1 day, I think) so it's not recommended to use those for continuous export, as ADX cannot renew the token and the export will start failing once the token has expired. The recommended authentication method for continuous export is either using an account key, or a SAS key (with a very long expiry). Both are documented here.

Azure Data Explorer - Continuous data export AAD token generation process

1 Answers