Hive/Beeline, how can I set the job .staging directory?

On the cluster I'm working on every user is given 60GB of Hadoop quota. Historically the project I'm working on generates a lot of Hive queries. In order for things to work faster I'm trying to parallel these queries (which are unrelated) but as a result the directory /user/{myusername}/.staging/ is being filled with job_{someid} directories which in turn are filled with the hive jars and consume these 60GB very fast. While I can limit the parallelization factor I would also like to see if I can ask Hive to put these jars on a different directory. Say /tmp/{myusername} where I have a lot more space.

Any idea how do I tell Hive/Beeline to create the .staging directory under /tmp/{myusername}?

Answers


Easiest way is on execution of your beeline session.

beeline --hive.exec.stagingdir=/tmp/{myusername}

Think you can do it via !set inside beeline but don't have the syntax to hand.


The above doesn't work.

We found the following working

beeline --hiveconf hive.exec.stagingdir=/tmp/{myusername}


Need Your Help

how to fetch facebook friends list by Ruby

ruby-on-rails ruby facebook-graph-api koala-gem

I'm trying to use facebook graph api to get a user's friend list. But my app can only list the first user's friend list. For example, when I open user 6's friend list 6, it will show the friend lis...