Failed to query external table via PXF

0
votes

I made a GPDB(greenplum-db-5.18.0) cluster with one master and 3 segment hosts. And following this link, I configured kerberos and hdfs core xmls, and PXF service startup successfully. While I run a query pointing to remote hdfs file, it keeps giving error log like :

SEVERE: The exception contained within MappableContainerException could not be mapped to a response, re-throwing to the HTTP container
java.lang.VerifyError: Bad return type
Exception Details:
  Location:
    org/apache/hadoop/hdfs/DFSClient.getQuotaUsage(Ljava/lang/String;)Lorg/apache/hadoop/fs/QuotaUsage; @160: areturn
  Reason:
    Type 'org/apache/hadoop/fs/ContentSummary' (current frame, stack[0]) is not assignable to 'org/apache/hadoop/fs/QuotaUsage' (from method signature)
  Current Frame:
    bci: @160
    flags: { }
    locals: { 'org/apache/hadoop/hdfs/DFSClient', 'java/lang/String', 'org/apache/hadoop/ipc/RemoteException', 'java/io/IOException' }

The PXF service is started with openjdk 1.8.0_131. And our hdfs cluster is based on hadoop mdh 2.6.0.4, so I put hadoop-*2.6.0-mdh2.6.0.4.jar into $PXF_CONF/lib, while GPDB 5.18 itself is based on hadoop 2.8.5. Is it a conflict for this error?

I found one issue alike in error but without definite solution:Exception in thread "main" java.lang.VerifyError: Bad type on operand stack

BTW, is there any other thing I missed for accessing hdfs file with PXF such as setting HADOOP VERSION via gpconfig, which is not mentioned in the guide doc.

2
greenplum

2 Answers

1
votes

PXF is compatible with Cloudera, Hortonworks Data Platform, MapR, and generic Apache Hadoop distributions. Microsoft's (Microsoft Distribution of Hadoop (MDH)) is not within the release notes.

https://gpdb.docs.pivotal.io/5180/pxf/client_instcfg.html

0
votes

Try cleaning out your $PXF_CONF/lib folder. In general, the libraries that we ship are sufficient to connect to most hadoop distributions. I would suggest you try the following:

$ rm -rf $PXF_CONF/lib/*
$ $PXF_HOME/bin/pxf restart

If you are on a multinode cluster, you will need to remove the jar files from $PXF_CONF/lib/ manually on each segment host, and then restart the PXF cluster.