Why Am I getting Nullpointer Exception

Question

I'm trying to implement something like this

Object Main extends app {
    val sc = new SparkContext(sparkConf) 
    val hiveContext = new org.apache.spark.sql.hive.HiveContext(sc)
    val data = hiveContext.sql("select * from person").map(mapData(_))
    println(data.collect())
    def mapData(row:Row){
        val data = new Person <- Java object
        data.setName(getStringTrimmedToNull(row.getAs("name")))
        data.setId(getStringTrimmedToNull(row.getAs("id")))
        var salaryQuery = "select * from salary where id = '${id}'"
        val query = salaryQuery. replace("${id}",data.getId)
        hiveContext.sql(query).map(mapSalData(_))
    }

    def mapSalData(row:Row){
        val salData = new Salary <= object for java class salary
        //set data into the object
    }

}

Above is the pseudo code of what I'm trying to do and it is returning null pointer error at this line

 hiveContext.sql(query).map(mapSalData(_))

Would any one have any idea on what am I doing wrong?

zsxwing zsxwing · Accepted Answer · 2017-03-07T20:44:41

This is a common error for Spark beginner. You cannot use HiveContext(SQLContext, SparkSession, or SparkContext) in a closure that will run in executors.

You can rewrite your SQL codes to use join like select * from person p join salary s where p.id = s.id.

Why Am I getting Nullpointer Exception

1 Answers