Where to create staging data table in BigData environment?

0
votes

I am currently having Hadoop-2, PIG, HIVE and HBASE. I have an inputdata. I have loaded that data in HDFS. I want to create staging data in this environment.

My query is -

In which BigData component, I should create Staging Table(Pig/HIVE/HBASE) ; this will have data coming in based on a condition? Later, we might want to run MapReduce Jobs with complex logic on it.

Please assist

2
hivehbaseapache-pighadoop2staging-table
Anywhere you want. Pig is not an option as it does not have a metastore. Hive if you want SQL Like queries. HBase based on your access patterns. – Venkat
Hello Venkat. Thanks for the reply.If I create it in HIVE and then i want to run mapReduce programs on top of it. How will I achieve that? And one more thing, what do you exactly means by access patterns? Any example? – user3343543

2 Answers

0
votes

Anywhere you want. Pig is not an option as it does not have a metastore. Hive if you want SQL Like queries. HBase based on your access patterns.

When you run a Hive query on top of data it is converted into MR.

When you create it in Hive use Hive Queries & not MR. If you are using MR then use Pig. You will not benefit creating a Hive table on top of data.

0
votes

Hive: If you have OLAP kind of workload and dont need realtime read/write.

HBase: If you have OLTP kind of workload. You need to do realtime/streaming read/write. Some batch or OLAP processing can be done by using MapReduce. SQL-like querying is possible by using Apache Phoenix.

You can run MapReduce job on HIVE and HBase both.