Terraform and Azure Data Explorer

Question

I am looking for recipes on how to standup a complete Azure Data Explorer stack using Terraform. So far I have found that Terraform has support to create clusters, apply access control and create databases.

I am yet to find guidance/recipes on creation of tables and table mappings.

How are others provisioning complete Azure Data Explorer stacks? What are some of the best practices to integrate create tables/table-mappings sql calls with Terraform scripts?

Thanks in advance.

bff1292 bff1292 · Accepted Answer · 2020-02-19T18:54:04

We found much the same. Supposedly, support for Tables and Ingestion Mappings via ARM (or perhaps APIs accessible to Terraform) will be supported at some time.

However, the ADX team within Azure refers to tables and ingestion mappings as being part of the "data plane" -- not the "control plane." So their preferred approach is to use the management commands from a Kusto client to create tables and ingestion mappings.

We wound up creating a mini-migration tool in house to handle this work for us. It basically just reads through a list of files in a directory and applies any new migrations to the database that haven't previously been run (recording successful migrations to avoid re-running them). I would love to open source this tool, but we'd need sign-off from legal. Unlike other database technology you may be familiar with, I don't believe that ADX supports transactional updates to table structure, so we didn't even bother making rollback functionality for our tool.

There are some tools you can use that don't require writing up a whole migration tool of your own: https://docs.microsoft.com/en-us/azure/kusto/tools/#source-control-integration-tools

Terraform and Azure Data Explorer

2 Answers