HDFS put a local file to hdfs but got UnresolvedAddressException

Question

I want to put a 70G file to hdfs so I used 'put' command to do this. However, I got the following exception. I tried small size file with the same command, it works. Does anyone know what the problem could be? Thanks!

WARN  [DataStreamer for file /user/qzhao/data/sorted/WGC033800D_sorted.bam._COPYING_] hdfs.DFSClient (DFSOutputStream.java:run(628)) - DataStreamer Exception java.nio.channels.UnresolvedAddressException
    at sun.nio.ch.Net.checkAddress(Net.java:127)
    at sun.nio.ch.SocketChannelImpl.connect(SocketChannelImpl.java:644)
    at org.apache.hadoop.net.SocketIOWithTimeout.connect(SocketIOWithTimeout.java:192)
    at org.apache.hadoop.net.NetUtils.connect(NetUtils.java:529)
    at org.apache.hadoop.hdfs.DFSOutputStream.createSocketForPipeline(DFSOutputStream.java:1526)
    at org.apache.hadoop.hdfs.DFSOutputStream$DataStreamer.createBlockOutputStream(DFSOutputStream.java:1328)
    at org.apache.hadoop.hdfs.DFSOutputStream$DataStreamer.nextBlockOutputStream(DFSOutputStream.java:1281)
    at org.apache.hadoop.hdfs.DFSOutputStream$DataStreamer.run(DFSOutputStream.java:526)
put: java.nio.channels.ClosedChannelException
    at org.apache.hadoop.hdfs.DFSOutputStream.checkClosed(DFSOutputStream.java:1538)
    at org.apache.hadoop.fs.FSOutputSummer.write(FSOutputSummer.java:98)
    at org.apache.hadoop.fs.FSDataOutputStream$PositionCache.write(FSDataOutputStream.java:58)
    at java.io.DataOutputStream.write(DataOutputStream.java:107)
    at org.apache.hadoop.io.IOUtils.copyBytes(IOUtils.java:80)
    at org.apache.hadoop.io.IOUtils.copyBytes(IOUtils.java:52)
    at org.apache.hadoop.io.IOUtils.copyBytes(IOUtils.java:112)
    at org.apache.hadoop.fs.shell.CommandWithDestination$TargetFileSystem.writeStreamToFile(CommandWithDestination.java:395)
    at org.apache.hadoop.fs.shell.CommandWithDestination.copyStreamToTarget(CommandWithDestination.java:327)
    at org.apache.hadoop.fs.shell.CommandWithDestination.copyFileToTarget(CommandWithDestination.java:303)
    at org.apache.hadoop.fs.shell.CommandWithDestination.processPath(CommandWithDestination.java:243)
    at org.apache.hadoop.fs.shell.CommandWithDestination.processPath(CommandWithDestination.java:228)
    at org.apache.hadoop.fs.shell.Command.processPaths(Command.java:306)
    at org.apache.hadoop.fs.shell.Command.processPathArgument(Command.java:278)
    at org.apache.hadoop.fs.shell.CommandWithDestination.processPathArgument(CommandWithDestination.java:223)
    at org.apache.hadoop.fs.shell.Command.processArgument(Command.java:260)
    at org.apache.hadoop.fs.shell.Command.processArguments(Command.java:244)
    at org.apache.hadoop.fs.shell.CommandWithDestination.processArguments(CommandWithDestination.java:200)
    at org.apache.hadoop.fs.shell.CopyCommands$Put.processArguments(CopyCommands.java:259)
    at org.apache.hadoop.fs.shell.Command.processRawArguments(Command.java:190)
    at org.apache.hadoop.fs.shell.Command.run(Command.java:154)
    at org.apache.hadoop.fs.FsShell.run(FsShell.java:287)
    at org.apache.hadoop.util.ToolRunner.run(ToolRunner.java:70)
    at org.apache.hadoop.util.ToolRunner.run(ToolRunner.java:84)
    at org.apache.hadoop.fs.FsShell.main(FsShell.java:340)

any issues with your namenode? can you check for the hdfs-site.xml.core-site.xml the configurations — K S Nidhin
Is it a DNS or FireWall problem? check all your nodes' addresses. — magooup

PradeepKumbhar PradeepKumbhar · Accepted Answer · 2016-05-02T09:43:12

As the same command is working for small sized file, there could be a space issue on your HDFS. You can check the output of hdfs dfsadmin -report command to see if it's full.

Assuming that you are using command something similar to hdfs dfs -put /path/to/local.file /user/USERNAME/path/to/dir-or-file. If not, please share your command.

HDFS put a local file to hdfs but got UnresolvedAddressException

2 Answers