We are trying to implement Elasticsearch into our big data environment. Currently we are running Apache Hadoop 2.7, include Hive and Spark. Data store as Parquest format in Hadoop.
When we implement ELK into our environment, Can we only store data into Hadoop HDFS ? Or we must to extract data from Hadoop and import into Elasticsearch so can create indexing, but we have duplicate dataset into system(Hadoop HDFS and ElasticSearch)
Thank you.
Sorry if this is a bit long, but I hope that would help!
Elasticsearch is a search engine. Period. Search Solution. Or rather a type of database or datastore which helps you organize your data in such a way which can help you perform activities like data discovery or build search applications for your organisation.
Although you can also do lot of analytical queries and build analytical solutions around it, there are certain limitations.
The nature of Elasticsearch and the datastructures used in it are so different, that you would need to push the data(ingest) into it in order to perform all search/data-discover/analytical activities. It has its own file system and data structures which manage/store the data specifically for efficient searching.
So yes there will be duplication of data.
It is not to be used as analytical solution, although it does come with lot of aggregation queries, it is not as expressive as processing engine like Apache Spark or data virtualisation tools like Denodo or Presto.
It is not a storage solution like HDFS or S3 and used as a data lake for the organization.
It is not to be used as a transactional database to be replaced with RDBMS solutions.
Most of the times, many organisations ingest data into ES from various different sources like OLAP, RDBMS, NoSQL database, CMS, Messaging Queues so that they can do searching of the data more efficiently.
In a way, for most of the times, ES is never a primary datasource.
searchable.So two most common usage of ES is searching and logging and monitoring activities. Basically real time activities.
Mostly organisations are increasing leveraging Hadoop for its file system i.e. HDFS to be used as a data store while utilising Spark or Hive for data processing. Mostly to do heavy data analytical solutions for which ES has limitation.
Hadoop has the capability to store all file formats(of course you need to make use of parquet or other formats for efficient storage) however Elasticsearch only makes use of JSON. This makes Hadoop a default industry standard along with S3 and other FS for data-lake or data-archival storage tool.
If you are storing data in Hadoop, you probably would have to make do with other frameworks to do efficient data processing like Spark or Giraph or Hive to transform data and do complex analytical processing for which ES has a limitation. ES in its core is a full-text retrieval/search engine.
You probably need to run Map-Reduce or Spark Jobs and write tons of pattern-matching algorithm to find the documents or folders with any keyword you want to search. Every search would result into one such job. Which would not be practical.
Even if you transform and organise them in such a way for you to leverage Hive, still it would not be as efficient as Elasticsearch for text processing.
Here is a link that can help you understand core data structure used in Elasticsearch and why text search is faster/different/efficient.
Perhaps the below diagram as mentioned in this link could be useful.
Basically you can set up ingestion pipelines, process raw data from Hadoop, transform and thereby index them in Elasticsearch so that you can make use of its search capabilities.
Take a look this link to understand how you can make use of Spark with Elasticsearch and have two-way communication achieved between them.
Hope that helps!
