ELK stack data mapping features

Question

So currenyly I am wondering whether there is a feature in ELK stack that can do lookup and mapping for the data from different sources.

For example: I have log for

'computer.log' -> {computer_id: 123456, internet_connected: 345612}
'phone.log' ->    {phone_id: 234561}
'internet.log' -> {internet_id: 345612, phone_push: 234561}

So we have 3 streams of logs being sent to filebeat -> logstash -> elasticsearch -> kibana

When we need to trace which phone is connected to the computer, I want to search "computer_id: 123456" and then have all these 3 logs popped up as a result. I know that we can search specific index for the data in elastic search and parse the raw logs in logstash. But I wonder how do I trace or do the mapping if we receive these 3 logs separately (within 5ms).

Will logstash have this kind of data tracing feature or do I have to write a program to handle with the mapping and insert the conversion id into the specific log before streaming into logstash?

I do not know the exact name of this feature, could you please tell me if there is one in ELK stack?

nitzien nitzien · Accepted Answer · 2018-09-24T02:39:21

If internet.log was coming last with a guaranteed gap of at least couple of seconds we could have enriched index containing internet.log.

I suggest following

While indexing internet.log, add a document_id as a uuid - https://www.elastic.co/guide/en/logstash/current/plugins-filters-uuid.html and add a status - 'unprocessed'
Add another logstash
- Input - Using elasticsearch input plugin queries internet.log with status as unprocessed
- Filter 1 - Using elasticsearch filter plugin queries computer.log to get computer_id and other fields, if there, in events.
- Filter 2 - Using elasticsearch filter plugin queries phone.log to get fields related to phone in events.
- If Filter 1 and 2 successfully returned data, update status as processed.
- Output - Reindex internet.log document with same document_id (uuid)

You may find some example here - * https://www.elastic.co/guide/en/logstash/current/lookup-enrichment.html * https://www.elastic.co/guide/en/logstash/current/plugins-filters-elasticsearch.html

Now, you have all data in single index (index.log) to search and query.

ELK stack data mapping features

1 Answers