I'm getting following error while using sqoop import command: ERROR security.UserGroupInformation: PriviledgedActionException as:SqoopUser (auth:SIMPLE) cause:java.io.FileNotFoundException: File does not exist: hdfs://localhost:8020/home/SqoopUser/sqoop-1.4.3-cdh4.4.0/sqoop-1.4.3-cdh4.4.0.jar.
I'm using Cloudera hadoop and Sqoop.
Not able to understand , why it is looking for jar files in hdfs.
Regards, Big Data Developer
It is common for Hadoop services to look for jars in HDFS because all nodes in the cluster can access files in HDFS. This is important if the MapReduce job being kicked off by the Hadoop service, in this case Sqoop, has a dependence on those jars. Remember, the Mappers are running on a DataNode, not the NameNode even though you are (probably) running the Sqoop command from the NameNode. Putting the jars on HDFS is not the only possible solution to this problem, but it is a sensible one.
Now we can deal with the actual error. At least one, but probably all, of your Mappers are unable to find a jar they need. That means that either the jar does not exist or the user trying to access them does not have the required permissions. First check if the file exists by running hadoop fs -ls home/SqoopUser/sqoop-1.4.3-cdh4.4.0/sqoop-1.4.3-cdh4.4.0.jar by a user with superuser privileges on the cluster. If it does not exist, put it there with hadoop fs -put {jarLocationOn/NameNode/fileSystem/sqoop-1.4.3-cdh4.4.0.jar} /home/SqoopUser/sqoop-1.4.3-cdh4.4.0/sqoop-1.4.3-cdh4.4.0.jar. I haven't worked with Cloudera specifically, so you will have to track down the jar location on the NameNode yourself. If Cloudera is anything like Hortonworks, there will be occasional issues like this where cluster deployment scripts/documentation misses a couple of required steps to get everything up and running.
Now that we know the file exists, we can check to see if the user SqoopUser has permissions to the file. Again, run hadoop fs -ls home/SqoopUser/sqoop-1.4.3-cdh4.4.0/sqoop-1.4.3-cdh4.4.0.jar and look at the file permissions. Also check the permissions of the directories containing the jar. Explaining POSIX file permissions is outside the scope of this answer, so if you are not familiar you might need to read up on those. One important note is that HDFS does not have its own concept of groups, it bases that on the groups of the underlying OS. Just make sure the jar is readable by SqoopUser and all of the parent directories are executable by SqoopUser. Indiscriminate user of chmod 777 will take of this, ie hadoop fs -chmod 777 /home/SqoopUser/sqoop-1.4.3-cdh4.4.0/sqoop-1.4.3-cdh4.4.0.jar. But of course be more discerning about the permissions you grant if your environment requires it.
If you find file permissions are tripping you up more often than you would like, the nuclear option is to set dfs.permissions to false in hdfs-site.xml. This will let all users access all files on HDFS. This can be very useful for rapid development, but it is safer to leave dfs.permisssions on.
First, the answer:
You need to refresh the client config files. In CDH 5, this can be done from the front page of the Manager's UI by clicking on the circle-arrow symbol next to the service. I think the config that is causing your problem is the YARN service, but you might as well bring all your config files up to date.
Second, the background:
I just solved this issue for myself. I am using CDH 5.0.
Here is the command I ran and the error encountered:
$ sqoop import --connect jdbc:mysql://xxx/g2_2 --username xxx --password xxx --table client
...
14/05/27 22:11:54 WARN security.UserGroupInformation: PriviledgedActionException as:aortiz (auth:SIMPLE) cause:java.io.FileNotFoundException: File does not exist: hdfs://xxx:8020/opt/cloudera/parcels/CDH-5.0.1-1.cdh5.0.1.p0.47/lib/sqoop/lib/commons-io-1.4.jar
14/05/27 22:11:54 ERROR tool.ImportTool: Encountered IOException running import job: java.io.FileNotFoundException: File does not exist: hdfs://xxx:8020/opt/cloudera/parcels/CDH-5.0.1-1.cdh5.0.1.p0.47/lib/sqoop/lib/commons-io-1.4.jar
Please execute below commands which will resolve your problem. It will copy your files in HDFS system.
hdfs dfs -copyFromLocal /opt/hadoop/sqoop-1.4.6 hdfs://localhost:9000/opt/hadoop/sqoop-1.4.6
hdfs dfs -copyFromLocal /opt/hadoop/sqoop-1.4.6/lib hdfs://localhost:9000/opt/hadoop/sqoop-1.4.6/lib
hdfs dfs -copyFromLocal /opt/hadoop/sqoop-1.4.6/sqoop-1.4.6.jar hdfs://localhost:9000/opt/hadoop/sqoop-1.4.6/sqoop-1.4.6.jar
Similarly copy any file which is not able to get by HDFS system.
Note : In above command /opt/hadoop/sqoop-1.4.6 is my system sqoop installation location
