How to use JavaRDD class for extracting count with group by?

I want to extract username and count (how many times each event is performed by each user) using JavaRDD Class. I don't have any idea how to make a JavaRDD object.

Here is a snapshot from my data:

{
    "_id" : ObjectId("57b3e6d1cab823158a06cafe"),
    "app" : {
            "clientIp" : "111.0.0.1",
            "event" : {
                    "event_name" : "MAX_SEARCH",
                    "appId" : 1,
                    "userName" : "Alex"
                    }
                }
}

And the expected result:

Alex    MAX_SEARCH    5

How can I do this?

Answers


Consider you have multiple records with you in text file as shown below, and you wants to get the username, event name and event count.

{
  "_id": ObjectId("57b3e6d1cab823158a06cafe"),
  "app": {
    "clientIp": "111.0.0.1",
    "event": {
      "event_name": "MAX_SEARCH",
      "appId": 1,
      "userName": "Alex"
    }
  }
},
{
  "_id": ObjectId("57b3e6d1cab823158a06cafe"),
  "app": {
    "clientIp": "111.0.0.1",
    "event": {
      "event_name": "MAX_SEARCH",
      "appId": 1,
      "userName": "Alex"
    }
  }
}
{
  "_id": ObjectId("57b3e6d1cab823158a01cafe"),
  "app": {`enter code here`
    "clientIp": "111.0.0.1",
    "event": {
      "event_name": "MAX_SEARCH",
      "appId": 1,
      "userName": "Hokam"
    }
  }
},
{
  "_id": ObjectId("57b3e6d1cab823158a02cafe"),
  "app": {
    "clientIp": "111.0.0.1",
    "event": {
      "event_name": "MIN_SEARCH",
      "appId": 1,
      "userName": "Hokam"
    }
  }
}

The below code snippet helps you to read the data from the above file, create rdd out of it and generate the expected results.

import net.minidev.json.JSONObject;
import net.minidev.json.JSONValue;
SparkConf conf = new SparkConf().setAppName("UserEventLogger").setMaster("local[2]");
JavaSparkContext sc = new JavaSparkContext(conf);

String fileData = FileUtils.readFileToString(new File("/data/pocs/text-file.json"));
List<JSONObject> jsonObject = (List<JSONObject>) JSONValue.parse("[" + fileData + "]");

JavaRDD<JSONObject> jsonRdd = sc.parallelize(jsonObject);

jsonRdd.mapToPair(new PairFunction<JSONObject, String, Integer>() {

    @Override
    public Tuple2<String, Integer> call(JSONObject appObj) throws Exception {
        JSONObject app = (JSONObject) appObj.get("app");
        JSONObject event = ((JSONObject) app.get("event"));
        String username = event.getAsString("userName");
        String eventName = event.getAsString("event_name");

        return new Tuple2<String, Integer>(username + " " + eventName, 1);
    }
}).reduceByKey(new Function2<Integer, Integer, Integer>() {

    @Override
    public Integer call(Integer v1, Integer v2) throws Exception {
        return v1 + v2;
    }
}).foreach(new VoidFunction<Tuple2<String, Integer>>() {

    @Override
    public void call(Tuple2<String, Integer> t) throws Exception {
        System.out.println(t._1 + " " + t._2);

    }
});

sc.stop();

Once you execute the above code snippet you will see the following output:

Hokam MAX_SEARCH 1
Alex MAX_SEARCH 2
Hokam MIN_SEARCH 1

Need Your Help

devise password in confirm email

ruby-on-rails-3 email devise passwords

# in views/device/registration/new.html.erb: