Azure data explorer automatic ingestion fails but manual works

1
votes

I am using Azure data explorer and event hub to ingest JSON from blob storage.

The steps I am following are:

  1. A logicapp creates the blob JSON, in the format {"id":"12345","message":"base64String"}
  2. In Azure Data Explorer I created a table, then used "Ingest data from blob container". This successfully imported the data I already had in the container.
  3. I then clicked the link to set up the auto-ingest. This auto-ingesting is failing to ingest with the following exception:
.show ingestion failures

"OperationId": f533e94f-58f5-4c88-837e-e1b9117850fa,
"Database": redacteddb,
"Table": redactedtbl,
"FailedOn": 2021-04-28T14:19:25.2781929Z,
"IngestionSourcePath": https://redactedstore.blob.core.windows.net/attachments/12345.txt.json,
"Details": Stream_NoDataToIngest: The input stream produced 0 bytes. This usually means that the input JSON stream was ill formed.,
"FailureKind": Permanent,
"RootActivityId": guid...,
"OperationKind": DataIngestPull,
"OriginatesFromUpdatePolicy": 0,
"ErrorCode": Stream_NoDataToIngest,
"Principal": dstsapp=KustoIngestionProd,
"ShouldRetry": 0,
"User": ,
"IngestionProperties": [Format=Json/mandatory, IngestionMapping=[{"column":"id","datatype":"","Path":"$['id']"},{"column":"message","datatype":"","Path":"$['message']"}], ValidationPolicy=[Options=ValidateCsvInputConstantColumns, Implications=BestEffort, IsDetailedErrorReportingEnabled=False], Tags=[ToStringEmpty], IngestIfNotExists=[ToStringEmpty], ZipPattern=[null]],

If I run the ingest manually again through https://dataexplorer.azure.com/ it works.

Can anyone help me know how to troubleshoot next? What have i done wrong with setting up my ingestion?

1
azureazure-data-explorer

1 Answers

2
votes

Try re-configuring your data connection to use the multijson / multilinejson format, instead of json.

see: https://docs.microsoft.com/en-us/azure/data-explorer/ingest-json-formats?tabs=kusto-query-language#the-json-format

The JSON format Azure Data Explorer supports two JSON file formats:

  • json: Line separated JSON. Each line in the input data has exactly one JSON record.
  • multijson: Multi-lined JSON. The parser ignores the line separators and reads a record from the previous position to the end of a valid JSON.

For more information, see JSON Lines.