ConnectException: connect error: No such file or directory when trying to connect to '50010' using importtsv on hbase

Question

I configured short-circuit settings on both hdfs-site.xml and hbase-site.xml. And I run importtsv on hbase to import data from HDFS to HBase on Hbase cluster. I look over the log on each datanode and all datanode have ConnectException i said to the title.

2017-03-31 21:59:01,273 WARN [main] org.apache.hadoop.hdfs.shortcircuit.DomainSocketFactory: error creating DomainSocket
java.net.ConnectException: connect(2) error: No such file or directory when trying to connect to '50010'
    at org.apache.hadoop.net.unix.DomainSocket.connect0(Native Method)
    at org.apache.hadoop.net.unix.DomainSocket.connect(DomainSocket.java:250)
    at org.apache.hadoop.hdfs.shortcircuit.DomainSocketFactory.createSocket(DomainSocketFactory.java:164)
    at org.apache.hadoop.hdfs.BlockReaderFactory.nextDomainPeer(BlockReaderFactory.java:753)
    at org.apache.hadoop.hdfs.BlockReaderFactory.createShortCircuitReplicaInfo(BlockReaderFactory.java:469)
    at org.apache.hadoop.hdfs.shortcircuit.ShortCircuitCache.create(ShortCircuitCache.java:783)
    at org.apache.hadoop.hdfs.shortcircuit.ShortCircuitCache.fetchOrCreate(ShortCircuitCache.java:717)
    at org.apache.hadoop.hdfs.BlockReaderFactory.getBlockReaderLocal(BlockReaderFactory.java:421)
    at org.apache.hadoop.hdfs.BlockReaderFactory.build(BlockReaderFactory.java:332)
    at org.apache.hadoop.hdfs.DFSInputStream.blockSeekTo(DFSInputStream.java:617)
    at org.apache.hadoop.hdfs.DFSInputStream.readWithStrategy(DFSInputStream.java:841)
    at org.apache.hadoop.hdfs.DFSInputStream.read(DFSInputStream.java:889)
    at org.apache.hadoop.hdfs.DFSInputStream.read(DFSInputStream.java:696)
    at java.io.DataInputStream.readByte(DataInputStream.java:265)
    at org.apache.hadoop.io.WritableUtils.readVLong(WritableUtils.java:308)
    at org.apache.hadoop.io.WritableUtils.readVIntInRange(WritableUtils.java:348)
    at org.apache.hadoop.io.Text.readString(Text.java:471)
    at org.apache.hadoop.io.Text.readString(Text.java:464)
    at org.apache.hadoop.mapred.MapTask.getSplitDetails(MapTask.java:358)
    at org.apache.hadoop.mapred.MapTask.runNewMapper(MapTask.java:751)
    at org.apache.hadoop.mapred.MapTask.run(MapTask.java:341)
    at org.apache.hadoop.mapred.YarnChild$2.run(YarnChild.java:163)
    at java.security.AccessController.doPrivileged(Native Method)
    at javax.security.auth.Subject.doAs(Subject.java:422)
    at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1656)
    at org.apache.hadoop.mapred.YarnChild.main(YarnChild.java:158)
2017-03-31 21:59:01,277 WARN [main] org.apache.hadoop.hdfs.shortcircuit.ShortCircuitCache: ShortCircuitCache(0x34f7234e): failed to load 1073750370_BP-642933002-"IP_ADDRESS"-1490774107737

EDIT

hadoop 2.6.4 hbase 1.2.3

hdfs-site.xml

<property>
    <name>dfs.namenode.dir</name>
    <value>/home/hadoop/hdfs/nn</value>
</property>
<property>
    <name>dfs.namenode.checkpoint.dir</name>
    <value>/home/hadoop/hdfs/snn</value>
</property>
<property>
    <name>dfs.datanode.data.dir</name>
    <value>file:///home/hadoop/hdfs/dn</value>
</property>
<property>
    <name>dfs.namenode.http-address</name>
    <value>hadoop1:50070</value>
</property>
<property>
    <name>dfs.namenode.secondary.http-address</name>
    <value>hadoop1:50090</value>
</property>
<property>
    <name>dfs.namenode.rpc-address</name>
    <value>hadoop1:8020</value>
</property>
<property>
    <name>dfs.namenode.handler.count</name>
    <value>50</value>
</property>
<property>
    <name>dfs.datanode.handler.count</name>
    <value>50</value>
</property>
<property>
    <name>dfs.client.read.shortcircuit</name>
    <value>true</value>
</property>
<property>
    <name>dfs.block.local-path-access.user</name>
    <value>hbase</value>
</property>
<property>
    <name>dfs.datanode.data.dir.perm</name>
    <value>775</value>
</property>
<property>
    <name>dfs.domain.socket.path</name>
    <value>_PORT</value>
</property>
<property>
    <name>dfs.client.domain.socket.traffic</name>
    <value>true</value>
</property>

hbase-site.xml

<property>
    <name>hbase.rootdir</name>
    <value>hdfs://hadoop1/hbase</value>
</property>
<property>
    <name>hbase.zookeeper.quorum</name>
        <value>hadoop1,hadoop2,hadoop3,hadoop4,hadoop5,hadoop6,hadoop7,hadoop8</value>
</property>
<property>
    <name>hbase.cluster.distributed</name>
    <value>true</value>
</property>
<property>
    <name>dfs.client.read.shortcircuit</name>
    <value>true</value>
</property>
<property>
    <name>hbase.regionserver.handler.count</name>
    <value>50</value>
</property>
<property>
    <name>hfile.block.cache.size</name>
    <value>0.5</value>
</property>
<property>
    <name>hbase.regionserver.global.memstore.size</name>
    <value>0.3</value>
</property>
<property>
    <name>hbase.regionserver.global.memstore.size.lower.limit</name>
    <value>0.65</value>
</property>
<property>
    <name>dfs.domain.socket.path</name>
    <value>_PORT</value>
</property>

franklinsijo franklinsijo · Accepted Answer · 2017-03-31T15:13:28

Short-circuit reads make use of a UNIX domain socket. This is a special path in the filesystem that allows the Client and the DataNodes to communicate. You will need to set a path (not port) to this socket. The DataNode should be able to create this path.

The parent directory of the path value (for ex: /var/lib/hadoop-hdfs/) must exist and should be owned by the hadoop superuser. Also make sure any user except the HDFS user or root has no access to this path.

mkdir /var/lib/hadoop-hdfs/
chown hdfs_user:hdfs_user /var/lib/hadoop-hdfs/
chmod 750 /var/lib/hadoop-hdfs/

Add this property to hdfs-site.xml on all datanodes and clients.

<property>
  <name>dfs.domain.socket.path</name>
  <value>/var/lib/hadoop-hdfs/dn_socket</value>
</property>

Restart the services after making the changes.

Note: Paths under /var/run or /var/lib are commonly used.

ConnectException: connect error: No such file or directory when trying to connect to '50010' using importtsv on hbase

1 Answers