Stream Analytics Job deployed as Azure Resource Manager (ARM) template

Question

I am trying to setup an output EventHub for a Stream Analytics Job defined as a JSON template. Without the output bit the template is successfully deployed, however when adding the output definition it fails with:

Deployment failed. Correlation ID: <SOME_UUID>. {
    "code": "BadRequest",
    "message": "The JSON provided in the request body is invalid. Property 'eventHubName' value 'parameters('eh_name')' is not acceptable.",
    "details": {
        "code": "400",
        "message": "The JSON provided in the request body is invalid. Property 'eventHubName' value 'parameters('eh_name')' is not acceptable.",
        "correlationId": "<SOME_UUID>",
        "requestId": "<SOME_UUID>"
    }
}

I've defined the ARM template as:

{
    "$schema": "http://schema.management.azure.com/schemas/2015-01-01/deploymentTemplate.json#",
    "contentVersion": "1.0.0.0",
    "parameters": {
        "location": {
            "type": "string",
            "defaultValue": "westeurope"
        },
        "hubName": {
            "type": "string",
            "defaultValue": "fooIotHub"
        },
        "eh_name": {
            "defaultValue": "fooEhName",
            "type": "String"
        },
        "eh_namespace": {
            "defaultValue": "fooEhNamespace",
            "type": "String"
        },
        "streamAnalyticsJobName": {
            "type": "string",
            "defaultValue": "fooStreamAnalyticsJobName"
        }
    },
    "resources": [{
        "type": "Microsoft.StreamAnalytics/StreamingJobs",
        "apiVersion": "2016-03-01",
        "name": "[parameters('streamAnalyticsJobName')]",
        "location": "[resourceGroup().location]",
        "properties": {
            "sku": {
                "name": "standard"
            },
            "outputErrorPolicy": "Drop",
            "eventsOutOfOrderPolicy": "adjust",
            "eventsOutOfOrderMaxDelayInSeconds": 0,
            "eventsLateArrivalMaxDelayInSeconds": 86400,
            "inputs": [{
                "Name": "IoTHubInputLable",
                "Properties": {
                    "DataSource": {
                        "Properties": {
                            "iotHubNamespace": "[parameters('hubName')]",
                            "sharedAccessPolicyKey": "[listkeys(resourceId('Microsoft.Devices/IotHubs/IotHubKeys',parameters('hubName'), 'iothubowner'),'2016-02-03').primaryKey]",
                            "sharedAccessPolicyName": "iothubowner",
                            "endpoint": "messages/events"
                        },
                        "Type": "Microsoft.Devices/IotHubs"
                    },
                    "Serialization": {
                        "Properties": {
                            "Encoding": "UTF8"
                        },
                        "Type": "Json"
                    },
                    "Type": "Stream"
                }
            }],
            "transformation": {
                "name": "Transformation",
                "properties": {
                    "streamingUnits": 1,
                    "query": "<THE SQL-LIKE CODE FOR THE JOB QUERY>"
                }
            },
            "outputs": [{
                "name": "EventHubOutputLable",
                "properties": {
                    "dataSource": {
                        "type": "Microsoft.ServiceBus/EventHub",
                        "properties": {
                            "eventHubName": "parameters('eh_name')",
                            "serviceBusNamespace": "parameters('eh_namespace')",
                            "sharedAccessPolicyName": "RootManageSharedAccessKey"
                        }
                    },
                    "serialization": {
                        "Properties": {
                            "Encoding": "UTF8"
                        }
                    }
                }
            }]
        }
    }]
}

Checking here https://docs.microsoft.com/en-us/azure/templates/microsoft.streamanalytics/streamingjobs it looks like the structure of the JSON for the output is as the expected one (with the properties field along with the type).

I've figured out those "Event Hub properties" from the Chrome browser using Developer Tools and checking the details of the HTTP request "GetOutputs", otherwise I am not sure where I could see how to specify those properties? The structure looks quite similar to the one for the input IoT Hub (which is working), in that case using different lables for the properties related to the IoT Hub details.

Checking this blog post https://blogs.msdn.microsoft.com/david/2017/07/20/building-azure-stream-analytics-resources-via-arm-templates-part-2-the-template/ the output part is related to PowerBI and it looks like a different structure is used for the properties: outputPowerBISource, so I've tried to use for the Event Hub the field outputEventHubSource (from the checks using Chrome Developer Tools) instead of properties, but then I get this error:

Deployment failed. Correlation ID: <SOME_UUID>. {
    "code": "BadRequest",
    "message": "The JSON provided in the request body is invalid. Required property 'properties' not found in JSON. Path '', line 1, position 1419.",
    "details": {
        "code": "400",
        "message": "The JSON provided in the request body is invalid. Required property 'properties' not found in JSON. Path '', line 1, position 1419.",
        "correlationId": "<SOME_UUID>",
        "requestId": "<SOME_UUID>"
    }
}

The command I am using to deploy this template is the Azure CLI (from a Linux Debian machine):

az group deployment create \
  --name "deployStreamAnalyticsJobs" \
  --resource-group "MyRGName" \
  --template-file ./templates/stream-analytics-jobs.json

How do I specify an output in an Azure Resource Manager (ARM) template for a Stream Analytics Job?

Answer 1

Any property that contains a parameter (or any expression that needs to be evaluated must contain square brackets, e.g.

"eventHubName": "[parameters('eh_name')]",
"serviceBusNamespace": "[parameters('eh_namespace')]",

Otherwise the literal value in the quotes is used.

That help?

Answer 2

I found out all parameters need to be wrapped in square brakets (as pointed out in the other answer to this question).

Also to dynamically retrieve the shared access policy key (or any other parameter for an existing resource like the Event Hub) a combination of functions like listKeys and resourceId etc must be used, see below for a full example of an Event Hub described as an output for a Stream Analytics Job.

In details:

• the parameters defined for eventHubName and serviceBusNamespace must be evaluated using square brackets (see how I defined those parameters in the JSON example in the body of the question I asked above),
• the shared access policy could be either an hardcoded string (or a parameter as before) like sharedAccessPolicyName or dynamically retrieved using "sharedAccessPolicyKey": "[listKeys(resourceId('Microsoft.EventHub/namespaces/eventhubs/authorizationRules', parameters('eh_namespace'), parameters('eh_name'), 'RootManageSharedAccessKey'),'2017-04-01').primaryKey]" for the sharedAccessPolicyKey (this is sensitive data and it should be protected avoiding hardcoding information as a plain string)

The following JSON configuration shows an existing Event Hub defined as an output defined for the Stream Analytics Job:

        "outputs": [{
            "Name": "EventHubOutputLable",
            "Properties": {
                "DataSource": {
                    "Type": "Microsoft.ServiceBus/EventHub",
                    "Properties": {
                        "eventHubName": "[parameters('eh_name')]",
                        "serviceBusNamespace": "[parameters('eh_namespace')]",
                        "sharedAccessPolicyKey": "[listKeys(resourceId('Microsoft.EventHub/namespaces/eventhubs/authorizationRules', parameters('eh_namespace'), parameters('eh_name'), 'RootManageSharedAccessKey'),'2017-04-01').primaryKey]",
                        "sharedAccessPolicyName": "RootManageSharedAccessKey"
                    }
                },
                "Serialization": {
                    "Properties": {
                        "Encoding": "UTF8"
                    },
                    "Type": "Json"
                }
            }
        }]