How do I read from Hive using Apache Beam?

Question

How to read from Hive using Apache Beam / how to use Hive as a source in Apache Beam ?

Post an answer instead of putting the the answer together with the quesion — David דודו Markovitz

ya_java_dev ya_java_dev · Accepted Answer · 2017-05-23T16:30:37

HadoopInputFormatIO can be used to read from Hive as below :

Configuration conf = new Configuration();
conf.setClass("mapreduce.job.inputformat.class", HCatInputFormat.class, 
InputFormat.class);
conf.setClass("key.class", LongWritable.class, WritableComparable.class);
conf.setClass("value.class", DefaultHCatRecord.class, Writable.class);
conf.set("hive.metastore.uris", "...");
HCatInputFormat.setInput(hiveConf, "myDatabase", "myTable", "myFilter");


PCollection<KV<LongWritable, DefaultHCatRecord>> data =
p.apply(HadoopInputFormatIO.<Long, 
DefaultHCatRecord>read().withConfiguration(conf));

How do I read from Hive using Apache Beam?

2 Answers