How do you move files but not the directories in hdfs?

I've been trying to find the solution to this problem for a while. I have found that in a normal file system using shell you can use this command to move all files under a location but leave the directories alone.

    find . -maxdepth 1 -type f -exec mv {} destination_path \;

I was wondering if there is also a command to be able to do the same in hdfs.

So if I have a folder in hdfs called "folder1" which contains the files "copyThis.txt", "copyThisAsWell.txt" and "theFinalCopy.txt" and also contains a folder "doNotCopy" and I want to copy the files into a new folder called "folder2" but leave the folder "doNotCopy" behind, how can this be done in hdfs?

Thanks for any help you can provide.

Answers


There is no direct hdfs command to achieve this. But you can still do it by using combination of unix & hdfs commands. Something like this:

hdfs dfs -ls folder1 | grep "^-" | awk '{print $8}' | awk '{system("hdfs dfs -cp "$0" folder2")}'

Explanation:

hdfs dfs -ls folder1 : list the parent directory (not recursively)

grep "^-" : Exclude directories from the list

awk '{print $8}' : Print just the file name to be copied

awk '{system("hdfs dfs -cp "$0" folder2")}' : execute hdfs command with the file name just fetched

Note: This is just a workaround and will not work if you want to specify the depth like -maxdepth option of find command.


With xargs: hdfs dfs -ls folder1 | grep "^-" | awk '{print $8}' | xargs -n 1 -P 2 -I % hdfs dfs -mv % folder2


Need Your Help

Blur the background of the WPF container

.net wpf effects blur

Ultimately, what I want to achieve is a replication to some extend, of an Aero glass functionality of a WPF content control.