spark-shell “error: type mismatch” using :paste to define class/companion object

Question

Using spark-shell v1.6.

Big differences when I load a class def and its companion object depending on how it's done.

one line at a time - OK
via :paste - PROBLEM

First things first... since the repl finds it ambiguous to load a companion object with the same name, I give it an altered name. No problem there. Setup lookes like this...

import scala.util.Try

class Foo5(val i: Int)

object Foo5Companion {
  def apply(i: Int): Foo5 = new Foo5(i)
  def apply(d: Double): Foo5 = new Foo5(d.toInt)
  def applyTry(i: Int): Try[Foo5] = Try { apply(i) }
  def applyTry(d: Double): Try[Foo5] = Try { apply(d) }
}

Now let's do something simple with this class.

val ls_i: List[Int] = List(1,2,3)
val ls_foo: List[Foo5] = ls_i.map(Foo5Companion.apply)

If I've loaded the class def and companion object with :paste I get this error... to be clear, Foo5 has only been defined once in a new session. This is not an instance of the issue described here: "error: type mismatch" in Spark with same found and required datatypes

<console>:42: error: type mismatch;
found   : List[Foo5]
required: List[Foo5]
        val ls_foo: List[Foo5] = ls_i.map(Foo5Companion.apply)

BUT...

if i load the same defs in line-by-line (without using :paste)... it works fine...

ls_foo: List[Foo5] = List($iwC$$iwC$Foo5@66f1a93a, $iwC$$iwC$Foo5@39d53a3, $iwC$$iwC$Foo5@4dddf42f)

My question is... what's the difference? Why is :paste causing a problem and making the repl think Foo5 is ambiguous?

I think it may be the same underlying issue as the linked question; the advice there to "keep track of everything in your head" is not friendly; that's the REPL's job. — som-snytt

som-snytt som-snytt · Accepted Answer · 2018-05-23T20:25:20

Edit: this bug was fixed in 2.11.9, so latest 2.11.12 works.

Edit: since you're on Spark 1.6, I supposed you're stuck on 2.10. That's OK, I just picked up Homer again the other day, and in a very old version (Lattimore, 1951).

It looks like an old bug with how the Spark shell is handling imports from history under -Yrepl-class-based.

With -Xprint:typer:

  import scala.util.Try;
  import $line3.$read.INSTANCE.$iw.$iw.Foo5;
  private[this] val $line3$read: $line3.$read = $line3.$read.INSTANCE;
  <stable> <accessor> def $line3$read: $line3.$read = $iw.this.$line3$read;
  import $iw.this.$line3$read.$iw.$iw.Foo5Companion;

One import is via an aliased member, so the paths to the two Foo5 differ.

You can use :load instead of :paste in this case. Normally, you'd :paste companions.

This is no consolation, but it will be fixed when Spark upgrades to 2.12:

$ ~/scala-2.11.8/bin/scala -Yrepl-class-based 
Welcome to Scala 2.11.8 (OpenJDK 64-Bit Server VM, Java 1.8.0_171).
Type in expressions for evaluation. Or try :help.

scala> :paste foo5.scala
Pasting file foo5.scala...
import scala.util.Try
defined class Foo5
defined object Foo5Companion

scala> val ls_i: List[Int] = List(1,2,3)
ls_i: List[Int] = List(1, 2, 3)

scala> val ls_foo: List[Foo5] = List(1,2,3).map(Foo5Companion.apply)
<console>:15: error: type mismatch;
 found   : List[Foo5]
 required: List[Foo5]
       val ls_foo: List[Foo5] = List(1,2,3).map(Foo5Companion.apply)
                                               ^

scala> :quit
$ scala -Yrepl-class-based 

     ________ ___   / /  ___  
    / __/ __// _ | / /  / _ | 
  __\ \/ /__/ __ |/ /__/ __ | 
 /____/\___/_/ |_/____/_/ | | 
                          |/  version 2.12.6

scala> :paste foo5.scala
Pasting file foo5.scala...
import scala.util.Try
defined class Foo5
defined object Foo5Companion

scala> val ls_i: List[Int] = List(1,2,3)
ls_i: List[Int] = List(1, 2, 3)

scala> val ls_foo: List[Foo5] = List(1,2,3).map(Foo5Companion.apply)
ls_foo: List[Foo5] = List(Foo5@52354202, Foo5@6b1321b7, Foo5@342ee097)

spark-shell “error: type mismatch” using :paste to define class/companion object

1 Answers