Spark, ADAM and Zeppelin

0
votes

Trying to attempt genomic analysis using ADAM and Zeppelin. I'm not sure if I'm doing this right but running into below issue. 

%dep
z.reset()
z.addRepo("Spark Packages Repo").url("http://dl.bintray.com/spark-packages/maven")
z.load("com.databricks:spark-csv_2.10:1.2.0")   
z.load("mysql:mysql-connector-java:5.1.35")  
z.load("org.bdgenomics.adam:adam-core_2.10:0.20.0")
z.load("org.bdgenomics.adam:adam-cli_2.10:0.20.0")
z.load("org.bdgenomics.adam:adam-apis_2.10:0.20.0")

%spark

import org.bdgenomics.adam.rdd.ADAMContext._
import org.bdgenomics.adam.rdd.ADAMContext
import org.bdgenomics.adam.projections.{ AlignmentRecordField, Projection }
import org.apache.spark.SparkContext
import org.apache.spark.SparkConf
import org.bdgenomics.adam.rdd.ADAMContext
import org.bdgenomics.adam.rdd.ADAMContext._
import org.bdgenomics.adam.projections.Projection
import org.bdgenomics.adam.projections.AlignmentRecordField
import scala.io.Source
import org.apache.spark.rdd.RDD
import org.bdgenomics.formats.avro.Genotype
import scala.collection.JavaConverters._
import org.bdgenomics.formats.avro._
import org.apache.spark.SparkContext._
import org.apache.spark.mllib.linalg.{ Vector => MLVector, Vectors }
import org.apache.spark.mllib.clustering.{ KMeans, KMeansModel }

val ac = new ADAMContext(sc)

and I get the following output with a error

import org.bdgenomics.adam.rdd.ADAMContext._
import org.bdgenomics.adam.rdd.ADAMContext
import org.bdgenomics.adam.projections.{AlignmentRecordField, Projection}
import org.apache.spark.SparkContext
import org.apache.spark.SparkConf
import org.bdgenomics.adam.rdd.ADAMContext
import org.bdgenomics.adam.rdd.ADAMContext._
import org.bdgenomics.adam.projections.Projection
import org.bdgenomics.adam.projections.AlignmentRecordField
import scala.io.Source
import org.apache.spark.rdd.RDD
import org.bdgenomics.formats.avro.Genotype
import scala.collection.JavaConverters._
import org.bdgenomics.formats.avro._
import org.apache.spark.SparkContext._
import org.apache.spark.mllib.linalg.{Vector=>MLVector, Vectors}
import org.apache.spark.mllib.clustering.{KMeans, KMeansModel}
res7: org.apache.spark.SparkContext = org.apache.spark.SparkContext@62ec8142
<console>:188: error: constructor ADAMContext in class ADAMContext cannot be accessed in class $iwC
              new ADAMContext(sc)

Any idea where to look? Am I missing any dependencies? ^

2
apache-sparkapache-zeppelin

2 Answers

2
votes

According file ADAMContext.scala in the version you use. The constructor is private.

class ADAMContext private (@transient val sc: SparkContext) 
    extends Serializable with Logging {
    ...
}

You can instead use like this. 

import org.bdgenomics.adam.rdd.ADAMContext._

val adamContext: ADAMContext = z.sc

It will use the implicit conversion in object ADAMContext

object ADAMContext {
    implicit def sparkContextToADAMContext(sc: SparkContext): ADAMContext = 
        new ADAMContext(sc)
}
0
votes

It did work without using Z reference!!

val ac:ADAMContext  = sc
val genotypes: RDD[Genotype] = ac.loadGenotypes("/tmp/ADAM2").rdd

Output

ac: org.bdgenomics.adam.rdd.ADAMContext = org.bdgenomics.adam.rdd.ADAMContext@2c60ef7e

genotypes: 
org.apache.spark.rdd.RDD[org.bdgenomics.formats.avro.Genotype] = MapPartitionsRDD[3] at map at ADAMContext.scala:207

I had tried doing this at the adam-shell prompt and I don't recall having to use implicit conversion. It was using the 0.19 version of ADAM though.