Elastic Search - Raw JSON ( unstructured data ) handling with elastic C# API

Question

We are working on an application, where we receive data in json format from different sources and different format and user can also add such sources there own, So we don't know json format what properties will be there

Our application presents that that data to a grid format and grid provides filtering, sorting, paging, grouping etc. kind of standard operation on that data.

We decided to use elastic search to store such large and unstructured data.On the backend, we use .NET ( C#).

Started to use bulk API to index data. Here is sample json: myJson = @"{""index"":{""_index"":""abc"",""_type"":""abc"",""_id"":1}}, {""Name"":""ApplicationFrameHost"",""CPU"":1.25,""Company"":null,""Product"":null,""Path"":null} {""index"":{""_index"":""abc"",""_type"":""abc"",""_id"":2}}, {""Name"":""audiodg"",""CPU"":1.5625,""Company"":null,""Product"":null,""Path"":null} {""index"":{""_index"":""abc"",""_type"":""abc"",""_id"":3}}, {""Name"":""Calculator"",""CPU"":0.5,""Company"":null,""Product"":null,""Path"":null} {""index"":{""_index"":""abc"",""_type"":""abc"",""_id"":4}}, {""Name"":""chrome"",""CPU"":144.109375,""Company"":null,""Product"":null,""Path"":null} {""index"":{""_index"":""abc"",""_type"":""abc"",""_id"":5}}, {""Name"":""chrome"",""CPU"":3384.609375,""Company"":null,""Product"":null,""Path"":null}

";

On C# code side:

 var connectionSettings = new ConnectionSettings(new Uri("http://localhost:9200/"));
    connectionSettings.DisableDirectStreaming(true);
    var client2 = new ElasticClient(connectionSettings);

var jsonPostData = new PostData<object>(myJson);
    var bulkRequestParameters = new BulkRequestParameters
    {

    };

    Func<BulkRequestParameters, BulkRequestParameters> convert = delegate (BulkRequestParameters s)
    {
        s.ErrorTrace(true);
        return s.Refresh(Refresh.True);
    };

    ElasticsearchResponse<VoidResponse> response = client2.LowLevel.Bulk<VoidResponse>("abc", "abc", jsonPostData, convert);

It does successfully insert data to elastic.

Just wanted to confirm is this the optimal way to perform this?

Problem we are facing We need grouping on any field on index data ( user can group on any field while presenting data, So we need aggregated data at runtime. How can this be achieved with C# API?

Also, How filtering can be done on multiple fields with C# API?

I am new to elastic search so not have much idea, what capabilities it has and what are the ways to achieve this with C# API though I have played a bit with Kibana console on grouping, I am getting following error: Fielddata is disabled on text fields by default. Set fielddata=true on [Name] in order to load fielddata in memory by uninverting the inverted index. Note that this can however use significant memory. Alternatively use a keyword field instead.

Hatim Stovewala Hatim Stovewala · Accepted Answer · 2017-10-28T12:07:15

Inserting data into ES using bulk API is a good choice to index your data. Regarding, presenting and showcasing your data, you need to be more specific what you need. ES have a powerful Aggregation framework to provide you with Aggregated data & Analytics. Share your use case so we can help you better.

Elastic Search - Raw JSON ( unstructured data ) handling with elastic C# API

1 Answers