Kusto data ingestion from an Azure Function App ends with a 403

1
votes

I try to ingest data from azure function app into a ADX database. I followed the instruction found in the the article here.

The difference is, I'd like to insert data into the table. I struggle with a 403 error "Principal 'aadapp=;' is not authorized to access table"

What I did: I have created a AAD App with the following API permissions: AAD App configured permission

I configured the database via Kusto Explorer:

.add database myDB ingestors ('aadapp=;') 'theAADAppname'

.add table PressureRecords ingestors ('aadapp=;') 'theAADAppname'

.add table TemperatureRecords ingestors ('aadapp=;') 'theAADAppname'

My code:

 var kcsbDM = new KustoConnectionStringBuilder($"https://ingest-{serviceNameAndRegion}.kusto.windows.net:443/").WithAadApplicationKeyAuthentication(
            applicationClientId: "<my AD app Id>",
            applicationKey: "<my App Secret from Certificates & secrets>",
            authority: "<my tenant Id>");

        using (var ingestClient = KustoIngestFactory.CreateQueuedIngestClient(kcsbDM))
        {

            var ingestProps = new KustoQueuedIngestionProperties(databaseName, tableName);
            ingestProps.ReportLevel = IngestionReportLevel.FailuresAndSuccesses;
            ingestProps.ReportMethod = IngestionReportMethod.Queue;
            ingestProps.JSONMappingReference = mappingName;
            ingestProps.Format = DataSourceFormat.json;

            using (var memStream = new MemoryStream())
            using (var writer = new StreamWriter(memStream))
            {
                var messageString = JsonConvert.SerializeObject(myObject); // maps to the table / mapping 
                writer.WriteLine(messageString);
                writer.Flush();
                memStream.Seek(0, SeekOrigin.Begin);

                // Post ingestion message
                ingestClient.IngestFromStream(memStream, ingestProps, leaveOpen: true);
            }
2
azurefunctionazure-data-exploreringestadx
the steps you've taken seem to be ok. It's impossible to debug this without additional context (e.g. the full exception / error message you get) - if you don't want to include those on this SO thread, please consider opening a support ticket for your cluster via the Azure portal – Yoni L.

2 Answers

0
votes

The issue is that the mapping you are using in this ingestion command does not match the existing table schema (it has additional columns). In these cases Azure Data Explorer (Kusto) attempts to add the additional columns it finds in the mappings. Since the permission that the app has is 'ingestor', it cannot modify the table structure and thus the ingestion fails.

In your specific case, your table has a column that is written in a specific casing and in the ingestion mapping the same column has a different casing (for one character) so it is treated as a new column.

We will look into providing a better error message in this case.

0
votes

Update: the issue is fixed in the system and now it works as expected.

Avnera thanks for your hint, potential it is an issue because of the Real vs double translation. In one of my first try I used double in the table and that worked. That is not longer possible, looks the supported data types changed.

My current configuration:

.create table PressureRecords ( Timestamp:datetime, DeviceId:guid, Pressure:real )

.create-or-alter table PressureRecords ingestion json mapping "PressureRecords"
'['
'{"column":"TimeStamp","path":"$.DateTime","datatype":"datetime","transform":null},'
'{"column":"DeviceId","path":"$.DeviceId","datatype":"guid","transform":null},'
'{"column":"Pressure","path":"$.Pressure","datatype":"real","transform":null}'
']'

public class PressureRecord
{
    [JsonProperty(PropertyName = "Pressure")]
    public double Pressure { get; set; }
    [JsonProperty(PropertyName = "DateTime")]
    public DateTime DateTime { get; set; } = DateTime.Now;
    [JsonProperty(PropertyName = "DeviceId")]
    [Key]
    public Guid DeviceId { get; set; }
}