Scala: (_) in the SQLContext.scala of SparkSQL

Question

The Spark version is 1.3.0.

The source code from the SQLContext.scala (https://github.com/apache/spark/blob/master/sql/core/src/main/scala/org/apache/spark/sql/SQLContext.scala):

 @transient
  protected[sql] val ddlParser = new DDLParser(sqlParser.apply(_))

  @transient
  protected[sql] val sqlParser = {
    val fallback = new catalyst.SqlParser
    new SparkSQLParser(fallback(_))
  }

  protected[sql] def parseSql(sql: String): LogicalPlan = {
    ddlParser(sql, false).getOrElse(sqlParser(sql))
  }

I cannot really understand the code above.

how does the sqlParser.apply(_) work ?

what is (_) on earth? DDLParser's constructor needs a parameter parseQuery: String => LogicalPlan, but the return type of sqlParser.apply is LogicalPlan.

similarly, fallback(_) is also a apply call, how does (_) work?

Ben Reich Ben Reich · Accepted Answer · 2015-04-15T12:30:00

The underscore in this case denotes a partial function application, which is one way to convert a method type into a function type. So when you write sqlParser.apply(_), it is equivalent to x => sqlParser.apply(x). Since the DDLParser constructor needs an argument of type String => LogicalPlan, and sqlParser.apply returns a LogicalPlan, the compiler can deduce that x here is actually a String, and so sqlParser.apply(_) is of type String => LogicalPlan.

You have many options here. The following would be equivalent under most circumstances:

new DDLParser(x => sqlParser.apply(x))
new DDLParser(x => sqlParser(x))
new DDLParser(sqlParser.apply(_))
new DDLParser(sqlParser.apply _)
new DDLParser(sqlParser(_))
new DDLParser(sqlParser _)
new DDLParser(sqlParser.apply)
new DDLParser(sqlParser)

You can read more about the difference between methods types and function types here.

Another, closely related way of turning a method type into a function type is called eta expansion, which is used interchangeably in certain contexts. Read about partial function application, eta expansion, and all the other uses of underscore in Scala here.

Scala: (_) in the SQLContext.scala of SparkSQL

1 Answers