MongoDB Hadoop Integration

3
votes

I have installed Robomongo on my desktop.But i an mot able to ingest data into a hive table from Robomongo. I have applied the following steps:-

  1. downloaded the required jars- mongo-java-driver-2.13.3.jar, mongo-hadoop-core-1.4.0.jar,mongo-hadoop-hive-1.4.0.jar,mongodb-driver-3.2.1-javadoc.jar.

  2. I have placed the jar files in a temporary folder.

  3. In the hive script i have added these jar files. The script i used is as follows:-

ADD JAR /tmp/mongodb/jarfiles/mongo-java-driver-2.13.3.jar;

ADD JAR /tmp/mongodb/jarfiles/mongo-hadoop-core-1.4.0.jar;

ADD JAR /tmp/mongodb/jarfiles/mongo-hadoop-hive-1.4.0.jar;

ADD JAR /tmp/mongodb/jarfiles/mongodb-driver-3.2.1-javadoc.jar

CREATE TABLE individuals

( id STRING,

name STRING,

age STRING,

nationality STRING

)

STORED BY 'com.mongodb.hadoop.hive.MongoStorageHandler'

WITH SERDEPROPERTIES('mongo.columns.mapping'='{"id":"_id","name":"Name","age":"Age","nationality":"Nationality"}')

TBLPROPERTIES('mongo.uri'='mongodb://localhost:port/admin.test_1');

In the local host i have given the ip address and in the port i have given the port number. admin is the database name and test_1 is the collection that i am trying to ingest. Every time i run this code i get the following error:-

Error: Error while processing statement: java.net.URISyntaxException: Relative path in absolute URI: SERDEPROPERTIES('mongo.columns.mapping'='{"id":%22_id%22,%22name%22:%22Name%22,%22age%22:%22Age%22,%22nationality%22:%22Nationality%22%7D') (state=,code=1)

When i use SERDEPROPERTIES('mongo.columns.mapping'='{}') in the above code keeping everything else intact i get the following error :-

Error: Error while processing statement: java.net.URISyntaxException: Illegal character in scheme name at index 13: TBLPROPERTIES('mongo.uri'='mongodb://localhost:port/admin.test_1') (state=,code=1)

I am using CDH 5.4. Can anyone tell me how i can resolve this issue?

1
mongodbhadoop
I have also downloaded and added mongo-java-driver-3.2.2 and mongodb-driver-3.2.2.jar, but i still get the same error. Help needed ! – riz
I am not sure where the state=,code=1 line comes from. Do you have fields related to state or code ? if so, maybe post a sample document. – Wan Bachtiar
I have fields named- name, age and nationality. I don't have field s named state or code. The exact script that i have used is present here. I think state=,code=1 is irrespective of the fields because when i keep the mongo.columns.mapping empty i.e. When i use SERDEPROPERTIES('mongo.columns.mapping'='{}') , even then i get the state=,code=1 error which is the same case when i mention the fields in mongo.column.mapping – riz
Hi, 1) which MongoDB version are you using ? 2) What's the topology of your deployment? i.e. standalone, replica set, etc. 3) You don't need mongodb-driver, only mongo-java-driver. 4) What operating system are you running? thanks. – Wan Bachtiar
I am using 3.2.3 version of MongoDB. Standalone deployment. 64 bit WIndows 7. – riz

1 Answers

1
votes

As mentioned in mongo-hadoop Hive installation, the connector requires at least version 3.0.0 of the driver "uber" jar (called "mongo-java-driver.jar"). You seems to be using v2.13.3 which may not have the support for Hive yet.

You can download v3+ java uber driver from MongoDB Java Driver page. Make sure you select mongo-java-driver and specific version before clicking on the download button. The jar file name should be similar to mongo-java-driver-3.x.x.jar.