Connection refused while connecting from polybase to hadoop

Question

While trying to create external table from sql server 2017 to Hadoop in Ubuntu 16.04 it throws the below error

Msg 105019, Level 16, State 1, Line 1 EXTERNAL TABLE access failed due to internal error: 'Java exception raised on call to HdfsBridge_IsDirExist. Java exception message: Call From DESKTOP-VE8KNAG/xxx.xxx.x.xxx to xxx.xxx.x.x:54310 failed on connection exception: java.net.ConnectException: Connection refused: no further information; For more details see: http://wiki.apache.org/hadoop/ConnectionRefused: Error [Call From DESKTOP-VE8KNAG/1xxx.xxx.x.xxx to xxx.xxx.x.x:54310 failed on connection exception: java.net.ConnectException: Connection refused: no further information; For more details see: http://wiki.apache.org/hadoop/ConnectionRefused] occurred while accessing external file.'

External source location is fetched from core-site.xml
Tmp folder is created and added permission to the user and added it in core-site.xml
All nodes are running in Hadoop 10625 DataNode 10869 SecondaryNameNode 17113 ResourceManager 17434 NodeManager 10490 NameNode 21566 Jps

sql query

create EXTERNAL DATA SOURCE [HDP2] 
WITH (TYPE = HADOOP,
 LOCATION = N'hdfs://xxx.xxx.x.x:54310',
 CREDENTIAL = [HDPUser])
GO

create EXTERNAL TABLE [dbo].CLASS_DIM_EXP (
    [CLASS_ID] [varchar](8) NOT NULL,
    [CLASS_DESC] [varchar](100) NULL,
    [INSERT_DATE] [datetime2](7) NOT NULL,
    [LAST_UPDATE_DATETIME] [datetime2](7) NOT NULL)
WITH   (LOCATION='/user/pdw_user',
DATA_SOURCE = HDP2,
FILE_FORMAT = TSV,
REJECT_TYPE = VALUE,
REJECT_VALUE = 0);

Core-site.xml

<property>
  <name>hadoop.tmp.dir</name>
  <value>/app/hadoop/tmp</value>
  <description>A base for other temporary directories.</description>
</property>

<property>
  <name>fs.default.name</name>
  <value>hdfs://localhost:54310</value>
  <description>The name of the default file system.  A URI whose
  scheme and authority determine the FileSystem implementation.  The
  uri's scheme determines the config property (fs.SCHEME.impl) naming
  the FileSystem implementation class.  The uri's authority is used to
  determine the host, port, etc. for a filesystem.</description>
</property>

Is that something needed to be changed?

Mani Mani · Accepted Answer · 2018-04-26T10:15:01

The problem here is cores-site.xml file contains hdfs://localhost:54310 need to replace this with the corresponding IP address hdfs://xxx.xxx.x.x:54310.

Connection refused while connecting from polybase to hadoop

1 Answers