Elastic search document storing

0
votes

Basic usecase that we are trying to solve is for users to be able to search from the contents of the log file .

Lets say a simple situation where user searches for a keyword and this is present in a log file which i want to render it back to the user.

We plan to use ElasticSearch for handling this. The idea that i have in mind is to use elastic search as a mechanism to store the indexed log files.

Having this concept in mind, i went through https://www.elastic.co/guide/en/elasticsearch/reference/current/index.html

Couple of questions i have, 1) I understand the input provided to elastic search is a JSON doc. It is going to scan this JSON provided and create/update indexes. So i need a mechanism to convert my input log files to JSON??

2) Elastic search would scan this input document and create/update inverted indexes. These inverted indexes actually point to the exact document. So does that mean, ES would store these documents somewhere?? Would it store them as JSON docs? Is it purely in memory or on file sytem/database?

3) No when user searches for a keyword , ES returns back the document which contains the searched keyword. Now do i need to have the ability to convert back this JSON doc to the original log document that user expects??

Clearly im missing something.. Sorry for asking questions this silly , but im trying to improve my skills and its WIP.

Also , i understand that there is ELK stack out there. For some reasons we just want to use ES and not the LogStash and Kibana part of the stack..

Thanks

1
elasticsearch

1 Answers

1
votes
  1. Logs needs to be parsed to JSON before they can be inserted into Elasticsearch
  2. All documents are stored on the filesystem and some data is kept in memory but all data is persistent.
  3. When you search Elasticsearch you get back matching JSON documents. If you want to display the original error message, you can store that original message in one of the JSON fields and display just that.

So if you just want to store log messages and not break them into fields or anything, you can simply take each row and send it to Elasticsearch like so:

{ "message": "This is my log message" }

To parse logs, break them into fields and add some logic, you will need to use some sort of app, like Logstash for example.