How to copy all files and directories except one from local drive to HDFS?

Question

I have a directory /home/my_directory/ that contains the following files and sub-directories:

anotherTest.txt  doNotCopy.txt   my_subdirectory/   subdirectory2/   test.txt      

How can I copy all files and directories on HDFS except doNotCopy.txt file?

Answer 1

--copyFromLocal does not give a way to exclude particular files. What you can do is the following:

ls /home/my_directory | grep -v 'doNotCopy.txt' | while read -r fileName ; do 
    eval "hdfs dfs -copyFromLocal -f $fileName /path/to/HDFS/" 
done

Now if you ls your /path/to/HDFS/ you should get the desired files in there:

hdfs dfs -ls -C /path/to/HDFS/

anotherTest.txt   my_subdirectory/   subdirectory2/   test.txt 

If you want to exclude multiple files or directories

ls /home/my_directory | grep -v 'doNotCopy.txt\|dirDoNotCopy\|anotherTextFile.txt' | while read -r fileName ; do 
    eval "hdfs dfs -copyFromLocal -f $fileName /path/to/HDFS/" 
done