Scala Iterator ++ blows the stack

Question

I recently notice this bug causing StackOverFlowError in Scala Iterator++ caused by lazy init. Here's the code to make the bug appear.

var lines = Source.fromFile("file").getLines()
var line = lines.next()
lines = Array(line).toIterator ++ lines
lines.foreach { println(_) }
System.exit(0)

What I get is

Exception in thread "main" java.lang.StackOverflowError
at scala.collection.Iterator$JoinIterator.hasNext(Iterator.scala:219)
at scala.collection.Iterator$JoinIterator.hasNext(Iterator.scala:219)
at scala.collection.Iterator$JoinIterator.hasNext(Iterator.scala:219)
at scala.collection.Iterator$JoinIterator.hasNext(Iterator.scala:219)
...

It should be caused by this line in scala source (scala.collection.Iterator.scala:208)

lazy val rhs: Iterator[A] = that.toIterator

As rhs is a lazy init val, when the iterator is used what the name "lines" refers to already changed, and caused a loop reference, which leads to the error.

I noticed this post talks about the problem in 2013. However it seems it has not been fully repaired. I am running Scala 2.11.8 from Maven Repo.

My Question: I can rename the iterator, e.g. "lines2" to avoid this bug, but is this the only way to solve the problem? I feel like using the name "lines" is more natural and don't want to forsake it if possible.

jwvh jwvh · Accepted Answer · 2017-02-06T03:28:06

If you want to reload an Iterator using the same var, this appears to work. [Tested on 2.11.7 and 2.12.1]

scala> var lines = io.Source.fromFile("file.txt").getLines()
lines: Iterator[String] = non-empty iterator

scala> var line = lines.next()
line: String = this,that,other,than

scala> lines = Iterator(line +: lines.toSeq:_*)
lines: Iterator[String] = non-empty iterator

scala> lines.foreach(println)
this,that,other,than
here,there,every,where

But it might make more sense to use a BufferedIterator where you can call head on it to peek at the next element without consuming it.

explanation

lines.toSeq <-- turn the Iterator[String] into a Seq[String] (The REPL will show this as a Stream but that's because the REPL has to compile and represent each line of input separately.)

line +: lines.toSeq <-- create a new Seq[String] with line as the first element (i.e. prepended)

(line +: lines.toSeq:_*) <-- turns a single Seq[T] into a parameter list that can be passed to the Iterator.apply() method. @som-snytt has cleverly pointed out that this can be simplified to (line +: lines.toSeq).iterator

BufferedIterator example

scala> var lines = io.Source.fromFile("file.txt").getLines.buffered
lines: scala.collection.BufferedIterator[String] = non-empty iterator
                        ^^^^^^^^^^^^^^^^^^^^^^^^ <-- note the type

scala> lines.head
res5: String = this,that,other,than

scala> lines foreach println
this,that,other,than
here,there,every,where

Scala Iterator ++ blows the stack

2 Answers