Histogram in Spark (1) | Big Data Analytics with Spark

Histogram in Spark (1) | Big Data Analytics with Spark

Spark’s DoubleRDDFunctions provide a histogram function for RDD[Double]. However there are no histogram function for RDD[String]. Here is a quick exercise for doing it. We will use immutable Map in this exercise.

Create a dummy RDD[String] and apply the aggregate method to calculate histogram

val d=sc.parallelize((1 to 10).map(_ % 3).map("val"+_.toString))

d.aggregate(Map[String,Int]())(
     (m,c)=>m.updated(c,m.getOrElse(c,0)+1),
     (m,n)=>(m /: n){case (map,(k,v))=>map.updated(k,v+map.getOrElse(k,0))}
)

def mapadd[T](m:Map[T,Int],n:Map[T,Int])={
     (m /: n){case (map,(k,v))=>map.updated(k,v+map.getOrElse(k,0))}
}

mapadd(Map("a"->1,"b"->2),Map("a"->2,"c"->1))

d.aggregate(Map[String,Int]())(
     (m,c)=>m.updated(c,m.getOrElse(c,0)+1),
     mapadd(_,_)
)
Read full article from Histogram in Spark (1) | Big Data Analytics with Spark