Which version of sqoop should I use for hadoop 3.3.0?

0
votes

I am trying to install sqoop 1.4.7 in windows 10 on hadoop 3.3.0 ,

on using ./configure-sqoop on GIT bash I get following o/p:

Warning: C:\sqoop_data\sqoop-1.4.7.bin__hadoop-2.6.0/../hbase does not exist! HBase imports will fail. Please set $HBASE_HOME to the root of your HBase installation. Warning: C:\sqoop_data\sqoop-1.4.7.bin__hadoop-2.6.0/../hcatalog does not exist! HCatalog jobs will fail. Please set $HCAT_HOME to the root of your HCatalog installation. Warning: C:\sqoop_data\sqoop-1.4.7.bin__hadoop-2.6.0/../accumulo does not exist! Accumulo imports will fail. Please set $ACCUMULO_HOME to the root of your Accumulo installation. Warning: C:\sqoop_data\sqoop-1.4.7.bin__hadoop-2.6.0/../zookeeper does not exist! Accumulo imports will fail. Please set $ZOOKEEPER_HOME to the root of your Zookeeper installation.

on verifying the installation using sqoop.cmd version , I get:

Warning: HBASE_HOME and HBASE_VERSION not set. Warning: HCAT_HOME not set Warning: HCATALOG_HOME does not exist HCatalog imports will fail. Please set HCATALOG_HOME to the root of your HCatalog installation. Warning: ACCUMULO_HOME not set. Warning: ZOOKEEPER_HOME not set. Warning: HBASE_HOME does not exist HBase imports will fail. Please set HBASE_HOME to the root of your HBase installation. Warning: ACCUMULO_HOME does not exist Accumulo imports will fail. Please set ACCUMULO_HOME to the root of your Accumulo installation. Warning: ZOOKEEPER_HOME does not exist Accumulo imports will fail. Please set ZOOKEEPER_HOME to the root of your Zookeeper installation. The system cannot find the path specified.

Please help with a solution to this problem

1
windowshadoopsqoop

1 Answers

1
votes

it depends if you are working on a server-side or client-side.

If you are on the server-side do the following:

Copy the Sqoop artifact to the machine where you want to run Sqoop server. The Sqoop server acts as a Hadoop client, therefore Hadoop libraries (Yarn, Mapreduce, and HDFS jar files) and configuration files (core-site.xml, mapreduce-site.xml, ...) must be available on this node. You do not need to run any Hadoop related services - running the server on a “gateway” node is perfectly fine.

Decompress Sqoop distribution tarball

tar -xvf sqoop-<version>-bin-hadoop<hadoop-version>.tar.gz

Move decompressed content to any location

mv sqoop-<version>-bin-hadoop<hadoop version>.tar.gz /usr/lib/sqoop

Change working directory

cd /usr/lib/sqoop

And sqoopt needs the enviromantal variables to be poiting at Hadoop libraries. So set and export the variables like that:

# Export HADOOP_HOME variable
export HADOOP_HOME=/...

# Or alternatively HADOOP_*_HOME variables
export HADOOP_COMMON_HOME=/...
export HADOOP_HDFS_HOME=/...
export HADOOP_MAPRED_HOME=/...
export HADOOP_YARN_HOME=/...

Sqoop server will need to impersonate users to access HDFS, so edit the core-site.xml file:

<property>
  <name>hadoop.proxyuser.sqoop2.hosts</name>
  <value>*</value>
</property>
<property>
  <name>hadoop.proxyuser.sqoop2.groups</name>
  <value>*</value>
</property>

For client installation Just copy Sqoop distribution artifact on target machine and unzip it in desired location. You can start client with following command:

sqoop2-shell

for the path variable, All user and administrator facing shell commands are stored in bin/ directory. It’s recommended to add this directory to your $PATH for easier execution, for example:

PATH=$PATH:`pwd`/bin/