Moving huge local files to HDFS using java

I have about 20 millions files stored on my local file system, each file 5k represents a tweet.

This stored as the following:

/home/username/tweets/$tag/$year/$month/$day/$tweetid.txt

Example1 : /home/username/tweets/SCP/2014/04/11/9989443342233.txt

Example1 : /home/username/tweets/WDR/2014/02/08/5890321764568.txt

So is it possible to write a map reduce java program to move all tweets under a certain tag to a singe directory in HDFS based on the tag.

Any similar examples?

Answers


As seen in https://blog.cloudera.com/blog/2009/02/the-small-files-problem/,

Make the sequencefile first, then upload it to HDFS.


Need Your Help

Add sql database in mfc dialog based application

visual-c++ mfc

I have a MFC dialog based project,Now I want to add database with that project.

Panel Open close not working

javascript jquery html css jquery-mobile

Hi I have requirement where Jquery mobile panel will be opening when some conditions are met i have tried close,open panel but that was not working. Please help Thanks in advance