HTTPS Request in Mapreduce

I'm trying to make an HTTPS request from the Mapper class of my job. I'm using Java's HttpsURLConnection library, which requires me to set the following properties: javax.net.ssl.keyStore javax.net.ssl.keyStorePassword

There's no problem setting these properties in a Java application, but in MapReduce I'm having trouble setting these properties at the mapper level. I can't figure out how to "reach outside" of Hadoop and set these properties for every mapper.

Answers


It is not a good approach to create request from MapReduce. The intuition is: you use MapReduce to be able to scale your computation. You havent provided more info about what you are doing, so lets think about what happens when you would scale (orders of magnitude) the task.

  • If your goal is to connect to one place, you are effectively (D)DoSing it. The server on the other end would not probably scale up as well as your MapReduce (cluster).

  • If you are crawling many targets, consider using some crawling distributed framework (find some on Google).


Solution wasn't related to anything with the application code, but rather cluster configuration. The keystore file needed to be placed in the same location on each data node. Not exactly endorsing this as the safest solution in terms of security, but it worked for a quick POC.


Need Your Help

Can I start using HTML 5 for my smartphone app?

iphone android html5 smartphone

Most smartphones use modern browser engines that have implementing HTML 5 (or at least partially). Should I start using HTML 5 for my web application ? Where can I find a list of browser engines us...

Route audio from phone microphone to phone's microUSB port in android phones

android usb host microphone

Is there any API or is it possible to route audio or capture our voice through android phone's microphone and route it to direcly to the the phone's USB slot?