How to execute purge command on Kusto SDK

Question

I'm trying to run purge command for testing purposes using Kusto C# SDK

I'm executing .show purges as I understand if I can execute this command then I also can execute other purge command

I've use the connection string with the ingest prefix (as in this answer https://github.com/Azure/azure-kusto-python/issues/165), yet I'm unable to use purge command

public AdxConnection()
{
    // https://ingest-mycluster.myregion.kusto.windows.net
    var connStringBuilder = new KustoConnectionStringBuilder(_connectionString)
        .WithAadApplicationKeyAuthentication(_clientId, _clientSecret, _tenantId);

    _adminProvider = KustoClientFactory.CreateCslAdminProvider(connStringBuilder);
}

public async Task Purge()
{
    var purgeCommand = CslCommandGenerator.GenerateShowPurgeCommand(_database);
    await _adminProvider.ExecuteControlCommandAsync(_database, purgeCommand);
}

Here is the response from Kusto

Syntax error: Query could not be parsed: {
    "error": {
        "code": "BadRequest_SyntaxError",
        "message": "Request is invalid and cannot be executed.",
        "@type": "Kusto.Data.Exceptions.SyntaxException",
        "@message": "Syntax error: Query could not be parsed: . Query: '.show database environment1 purge operations'",
        "@context": {
            "timestamp": "2020-02-21T11:04:52.4907001Z",
            "serviceAlias": "INGEST-DATAMTE",
            "machineName": "KDataMana000000",
            "processName": "Kusto.WinSvc.DM.Svc",
            "processId": 9420,
            "threadId": 4368,
            "appDomainName": "Kusto.WinSvc.DM.Svc.exe",
            "clientRequestId": "KD2RunCommand;7ee3cb62-5c82-4a6d-9885-bc56a2339a15",
            "activityId": "4058bf27-af5c-4bab-8d6f-c7c41473c89a",
            "subActivityId": "3689c292-91a6-4595-8e60-663a4a59a370",
            "activityType": "P.WCF.Service.ExecuteControlCommandInternal..IAdminClientServiceCommunicationContract",
            "parentActivityId": "1a9d19bf-15fe-4c43-a154-9e47e3fa7a26",
            "activityStack": "(Activity stack: CRID=KD2RunCommand;7ee3cb62-5c82-4a6d-9885-bc56a2339a15 ARID=4058bf27-af5c-4bab-8d6f-c7c41473c89a > DN.Admin.Client.ExecuteControlCommand/1a9d19bf-15fe-4c43-a154-9e47e3fa7a26 > P.WCF.Service.ExecuteControlCommandInternal..IAdminClientServiceCommunicationContract/3689c292-91a6-4595-8e60-663a4a59a370)"
        },
        "@permanent": true
    }
}. Query: '.show database environment1 purge operations'

*Edit: Replacing GenerateShowPurgeCommand with GenerateDmPurgesShowCommand solve the problem. However, when running GenerateDmPurgeTableRecordsFirstPhase I've encounter another error

InternalServiceError (520-UnknownError): {
    "error": {
        "code": "Internal service error",
        "message": "Request aborted due to an internal service error.",
        "@type": "Kusto.Common.Svc.Exceptions.AdminCommandInvalidOperationException",
        "@message": "An admin command cannot be executed due to an invalid state: State='Operation PurgeTableRecords requires feature flag EnabledForPurge set on Engine'",
        "@context": {
            "timestamp": "2020-02-24T01:58:01.5764072Z",
            "serviceAlias": "INGEST-DATAMTE",
            "machineName": "KDataMana000000",
            "processName": "Kusto.WinSvc.DM.Svc",
            "processId": 5760,
            "threadId": 6100,
            "appDomainName": "Kusto.WinSvc.DM.Svc.exe",
            "clientRequestId": "KD2RunCommand;64439168-29e4-420e-b04a-07dccf11c5a4",
            "activityId": "855d59ba-3de2-44b4-9cac-62fbe5f7a5a3",
            "subActivityId": "5872fb41-7483-49bd-922c-1ae6104771fc",
            "activityType": "DM.MonitoredCommand",
            "parentActivityId": "c31fbc9a-8ae5-4cbf-9070-db5e39b8d5fa",
            "activityStack": "(Activity stack: CRID=KD2RunCommand;64439168-29e4-420e-b04a-07dccf11c5a4 ARID=855d59ba-3de2-44b4-9cac-62fbe5f7a5a3 > DN.Admin.Client.ExecuteControlCommand/bf23f2c3-d82b-4cf0-9f2b-0bfd32d55bc5 > P.WCF.Service.ExecuteControlCommandInternal..IAdminClientServiceCommunicationContract/c31fbc9a-8ae5-4cbf-9070-db5e39b8d5fa > DM.MonitoredCommand/5872fb41-7483-49bd-922c-1ae6104771fc)"
        },
        "@permanent": false
    }
}

Do I need to contact Kusto team to set EnabledForPurge to true, or I can do that using a command myself? This is not something I find in the docs.

Ravit D Ravit D · Accepted Answer · 2020-02-21T14:27:13

The operation .show database environment1 purge operations is a command that should be executed on the "engine" cluster, meaning DATAMTE (and not ingest-DATAMTE) in your case.

In your case, you may find the following command on the "DM" (INGEST-DATAME) more useful: https://docs.microsoft.com/en-us/azure/kusto/concepts/data-purge#track-purge-operation-status

This command shows purge operation status by specifying the operation Id within the requested time period

Update answer for updated question: you need to contact Kusto to whitelist your cluster for purge.

How to execute purge command on Kusto SDK

1 Answers