File copying in java (cannot create file) by Hadoop

Question

I currently want to copy a file from hdfs to local computer. I have finished most of the work by fileinputstream and fileoutputstream. But then I encounter the following issue.

JAVA I/O exception. Mkdirs fail to create file

I have do some research and figure out that as I am using filesystem.create()(hadoop function)

https://hadoop.apache.org/docs/current/api/org/apache/hadoop/fs/FileSystem.html#create(org.apache.hadoop.fs.Path,%20org.apache.hadoop.util.Progressable)

the reason is as follows:

if I set my path to a non-existing folder, a folder will be created and the file I download is inside.
if I set my path to existing folder (say current directory), the above I/O exception occur.

Say if I already get the path and fileinputstream right, what should I use (better in filesystem library) in order to go around this problem

my code

//src and dst are the path input and output
Configuration conf = new Configuration();
FileSystem inFS = FileSystem.get(URI.create(src), conf);
FileSystem outFS = FileSystem.get(URI.create(dst), conf);
FSDataInputStream in = null;
FSDataOutputStream out = null;

in = inFS.open(new Path(src));

out = outFS.create(new Path(dst),
            new Progressable() {
        /*
         * Print a dot whenever 64 KB of data has been written to
         * the datanode pipeline.
         */
             public void progress() {
             System.out.print(".");
           }
    });

Please be aware with permission also. I often face many issues when i create a file in system because of user,group and folder permission. — huy
So you're trying to rewrite the hdfs dfs -copyToLocal Java code? — OneCricketeer
yes it just do similar thing to hdfs dfs -copyToLocal and it is on linux — Ricky Ng

IssiahB IssiahB · Accepted Answer · 2018-10-12T04:28:45

In the "File" class there is a method called createNewFile() that will create a new file only if one doent exist.

File copying in java (cannot create file) by Hadoop

1 Answers