Spark MLLIB questions

2
votes

I am looking into the source code of Spark-mllib, and I noticed all the source code are in scala, not java.

Wouldn't scala in general use more resources than Java? Why the ML engine here is written in scala, not Java? Am I missing anything here?

Also, Eclipse Luna seems to have trouble coordinate with the latest version of scala? Which IDE would people recommend for build the spark code with scala modules?

Thank you very much!

1
scalaapache-sparkapache-spark-mllib
Why do you think scala would use more resources in general? Scala code can be optimized where needed to be just as fast. - Michael Zajac
Just based on my study, scala uses up more memories than java. For example: stackoverflow.com/questions/5901452/… Or am I missing anything? - Edamame
Did you read the first answer by Rex Kerr in the linked question? It argues the opposite of what you're saying. - Michael Zajac
Yes, I read Rex Kerr's comment. And the following comment of his is exactly what I am worried about: val bigEnough = array.filter(_.length > 2).flatMap(mapping.get) Easy! But, unless you're fairly familiar with how the collections work, what you might not realize is that this way of doing this created an extra intermediate array (with filter), and an extra object for every element of the array (with mapping.get, which returns an option). It also creates two function objects (one for the filter and one for the flatMap), though that is rarely a major issue since function objects are small. - Edamame
Keep reading - it looks like you didn't get to the last two paragraphs of his answer. - The Archetypal Paul

1 Answers

0
votes
  1. The source code are in scala, java and python at most places in the latest Spark version(1.4). (https://spark.apache.org/docs/latest/index.html)
  2. I recommend you to use the IntelliJ IDEA. According most people's experience, it is most intelligent IDE for scala development.