We are using Google Cloud Bigtable for our Big Data. When I'm running MapReduce job I assembly a jar and run it and now I'm getting this error:
Application application_1451577928704_0050 failed 2 times due to AM Container for appattempt_1451577928704_0050_000002 exited with exitCode: 1 For more detailed output, check application tracking page:http://censored:8088/cluster/app/application_1451577928704_0050Then, click on links to logs of each attempt. Diagnostics: Exception from container-launch. Container id: container_e02_1451577928704_0050_02_000001 Exit code: 1 Stack trace: ExitCodeException exitCode=1: at org.apache.hadoop.util.Shell.runCommand(Shell.java:545) at org.apache.hadoop.util.Shell.run(Shell.java:456) at org.apache.hadoop.util.Shell$ShellCommandExecutor.execute(Shell.java:722) at org.apache.hadoop.yarn.server.nodemanager.DefaultContainerExecutor.launchContainer(DefaultContainerExecutor.java:211) at org.apache.hadoop.yarn.server.nodemanager.containermanager.launcher.ContainerLaunch.call(ContainerLaunch.java:302) at org.apache.hadoop.yarn.server.nodemanager.containermanager.launcher.ContainerLaunch.call(ContainerLaunch.java:82) at java.util.concurrent.FutureTask.run(FutureTask.java:266) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617) at java.lang.Thread.run(Thread.java:745) Container exited with a non-zero exit code 1 Failing this attempt. Failing the application.
When I logged to see the logging of the workers node I saw this error:
2016-02-15 02:59:54,106 INFO [main] org.apache.hadoop.mapreduce.v2.app.MRAppMaster: Created MRAppMaster for application appattempt_1451577928704_0050_000001 2016-02-15 02:59:54,294 WARN [main] org.apache.hadoop.util.NativeCodeLoader: Unable to load native-hadoop library for your platform... using builtin-java classes where applicable 2016-02-15 02:59:54,319 INFO [main] org.apache.hadoop.mapreduce.v2.app.MRAppMaster: Executing with tokens: 2016-02-15 02:59:54,319 INFO [main] org.apache.hadoop.mapreduce.v2.app.MRAppMaster: Kind: YARN_AM_RM_TOKEN, Service: , Ident: (appAttemptId { application_id { id: 50 cluster_timestamp: 1451577928704 } attemptId: 1 } keyId: -******) 2016-02-15 02:59:54,424 INFO [main] org.apache.hadoop.mapreduce.v2.app.MRAppMaster: Using mapred newApiCommitter. 2016-02-15 02:59:54,755 WARN [main] org.apache.hadoop.hdfs.shortcircuit.DomainSocketFactory: The short-circuit local reads feature cannot be used because libhadoop cannot be loaded. 2016-02-15 02:59:54,855 INFO [main] org.apache.hadoop.mapreduce.v2.app.MRAppMaster: OutputCommitter set in config null 2016-02-15 02:59:54,911 INFO [main] org.apache.hadoop.service.AbstractService: Service org.apache.hadoop.mapreduce.v2.app.MRAppMaster failed in state INITED; cause: org.apache.hadoop.yarn.exceptions.YarnRuntimeException: java.lang.ClassCastException: org.apache.xerces.dom.DeferredElementNSImpl cannot be cast to org.w3c.dom.Text org.apache.hadoop.yarn.exceptions.YarnRuntimeException: java.lang.ClassCastException: org.apache.xerces.dom.DeferredElementNSImpl cannot be cast to org.w3c.dom.Text at org.apache.hadoop.mapreduce.v2.app.MRAppMaster$1.call(MRAppMaster.java:478) at org.apache.hadoop.mapreduce.v2.app.MRAppMaster$1.call(MRAppMaster.java:458) at org.apache.hadoop.mapreduce.v2.app.MRAppMaster.callWithJobClassLoader(MRAppMaster.java:1560) at org.apache.hadoop.mapreduce.v2.app.MRAppMaster.createOutputCommitter(MRAppMaster.java:458) at org.apache.hadoop.mapreduce.v2.app.MRAppMaster.serviceInit(MRAppMaster.java:377) at org.apache.hadoop.service.AbstractService.init(AbstractService.java:163) at org.apache.hadoop.mapreduce.v2.app.MRAppMaster$4.run(MRAppMaster.java:1518) at java.security.AccessController.doPrivileged(Native Method) at javax.security.auth.Subject.doAs(Subject.java:422) at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1657) at org.apache.hadoop.mapreduce.v2.app.MRAppMaster.initAndStartAppMaster(MRAppMaster.java:1515) at org.apache.hadoop.mapreduce.v2.app.MRAppMaster.main(MRAppMaster.java:1448) Caused by: java.lang.ClassCastException: org.apache.xerces.dom.DeferredElementNSImpl cannot be cast to org.w3c.dom.Text at org.apache.hadoop.conf.Configuration.loadResource(Configuration.java:2603) at org.apache.hadoop.conf.Configuration.loadResources(Configuration.java:2502) at org.apache.hadoop.conf.Configuration.getProps(Configuration.java:2405) at org.apache.hadoop.conf.Configuration.get(Configuration.java:981) at org.apache.hadoop.conf.Configuration.getTrimmed(Configuration.java:1031) at org.apache.hadoop.conf.Configuration.getBoolean(Configuration.java:1432) at org.apache.hadoop.hbase.HBaseConfiguration.checkDefaultsVersion(HBaseConfiguration.java:67) at org.apache.hadoop.hbase.HBaseConfiguration.addHbaseResources(HBaseConfiguration.java:81) at org.apache.hadoop.hbase.HBaseConfiguration.create(HBaseConfiguration.java:96) at org.apache.hadoop.hbase.HBaseConfiguration.create(HBaseConfiguration.java:105) at org.apache.hadoop.hbase.mapreduce.TableOutputFormat.setConf(TableOutputFormat.java:184) at org.apache.hadoop.util.ReflectionUtils.setConf(ReflectionUtils.java:76) at org.apache.hadoop.util.ReflectionUtils.newInstance(ReflectionUtils.java:136) at org.apache.hadoop.mapreduce.v2.app.MRAppMaster$1.call(MRAppMaster.java:474) ... 11 more
I tried an older jar and it's running perfectly fine and I'm not sure why the new jar won't work - Didn't change anything.
Please advise?
Thanks!
Update 1: Here is some more details: I setup the cluster with the dataproc. We are using the newest versions, here is the library dependencies:
val BigtableHbase = "com.google.cloud.bigtable" % "bigtable-hbase-1.1" % "0.2.2" val BigtableHbaseMapreduce = "com.google.cloud.bigtable" % "bigtable-hbase-mapreduce" % "0.2.2" val CommonsCli = "commons-cli" % "commons-cli" % "1.2" val HadoopCommon = "org.apache.hadoop" % "hadoop-common" % "2.7.1" val HadoopMapreduceClientApp = "org.apache.hadoop" % "hadoop-mapreduce-client-app" % "2.7.1" val HbaseCommon = "org.apache.hbase" % "hbase-common" % "1.1.2" val HbaseProtocol = "org.apache.hbase" % "hbase-protocol" % "1.1.2" val HbaseClient = "org.apache.hbase" % "hbase-client" % "1.1.2" val HbaseServer = "org.apache.hbase" % "hbase-server" % "1.1.2" val HbaseAnnotations = "org.apache.hbase" % "hbase-annotations" % "1.1.2"
libraryDependencies += BigtableHbase libraryDependencies += BigtableHbaseMapreduce libraryDependencies += CommonsCli libraryDependencies += HadoopCommon libraryDependencies += HadoopMapreduceClientApp libraryDependencies += HbaseCommon libraryDependencies += HbaseProtocol libraryDependencies += HbaseClient libraryDependencies += HbaseServer libraryDependencies += HbaseAnnotations
Java version:
openjdk version "1.8.0_66-internal" OpenJDK Runtime Environment (build 1.8.0_66-internal-b17) OpenJDK 64-Bit Server VM (build 25.66-b17, mixed mode)
Alpn version: alpn-boot-8.1.3.v20150130
hbase verison:
2016-02-15 20:45:42,050 INFO [main] util.VersionInfo: HBase 1.1.2 2016-02-15 20:45:42,051 INFO [main] util.VersionInfo: Source code repository file:///mnt/ram/bigtop/bigtop/output/ hbase/hbase-1.1.2 revision=Unknown 2016-02-15 20:45:42,051 INFO [main] util.VersionInfo: Compiled by bigtop on Tue Nov 10 19:09:17 UTC 2015 2016-02-15 20:45:42,051 INFO [main] util.VersionInfo: From source with checksum 42e8a1890c700d37485c69a44a3
hadoop version:
Hadoop 2.7.1 Subversion https://bigdataoss-internal.googlesource.com/third_party/apache/bigtop -r 2a194d4d838b79460c3ceb892f3c94 44218ba970 Compiled by bigtop on 2015-11-10T18:38Z Compiled with protoc 2.5.0 From source with checksum fc0a1a23fc1868e4d5ee7fa2b28a58a This command was run using /usr/lib/hadoop/hadoop-common-2.7.1.jar
