Spark Did not find registered driver with class com.mysql.jdbc.Driver

2
votes

I'm using CDH 5.7.0 and PySpark. When I run the action such as RDD.count() ,It show errors :Did not find registered driver with class com.mysql.jdbc.Driver

Following is the step

pyspark --driver-class-path /usr/share/java/mysql-connector-java.jar (the /usr/share/java/mysql-connector-java.jar on every node) 

>>>url ="jdbc:mysql://host/spark?user=root&password=test"
>>> stock_data=sqlContext.read.format("jdbc").option("url",url).option("dbtable","StockPrices").load()
>>> stock_data.printSchema()
root
 |-- date: string (nullable = true)
 |-- open: double (nullable = true)
 |-- high: double (nullable = true)
 |-- low: double (nullable = true)
 |-- close: double (nullable = true)
 |-- volume: long (nullable = true)
 |-- adjclose: double (nullable = true)
>>> stock_data.count()
......
    at java.lang.reflect.Method.invoke(Method.java:606)
    at py4j.reflection.MethodInvoker.invoke(MethodInvoker.java:231)
    at py4j.reflection.ReflectionEngine.invoke(ReflectionEngine.java:381)
    at py4j.Gateway.invoke(Gateway.java:259)
    at py4j.commands.AbstractCommand.invokeMethod(AbstractCommand.java:133)
    at py4j.commands.CallCommand.execute(CallCommand.java:79)
    at py4j.GatewayConnection.run(GatewayConnection.java:209)
    at java.lang.Thread.run(Thread.java:745)
**Caused by: java.lang.IllegalStateException: Did not find registered driver with class com.mysql.jdbc.Driver**
    at org.apache.spark.sql.execution.datasources.jdbc.JdbcUtils$$anonfun$createConnectionFactory$2$$anonfun$3.apply(JdbcUtils.scala:58)
    at org.apache.spark.sql.execution.datasources.jdbc.JdbcUtils$$anonfun$createConnectionFactory$2$$anonfun$3.apply(JdbcUtils.scala:58)
    at scala.Option.getOrElse(Option.scala:120)
    at org.apache.spark.sql.execution.datasources.jdbc.JdbcUtils$$anonfun$createConnectionFactory$2.apply(JdbcUtils.scala:57)
    at org.apache.spark.sql.execution.datasources.jdbc.JdbcUtils$$anonfun$createConnectionFactory$2.apply(JdbcUtils.scala:52)
    at org.apache.spark.sql.execution.datasources.jdbc.JDBCRDD$$anon$1.<init>(JDBCRDD.scala:347)
    at org.apache.spark.sql.execution.datasources.jdbc.JDBCRDD.compute(JDBCRDD.scala:339)
    at org.apache.spark.rdd.RDD.computeOrReadCheckpoint(RDD.scala:306)
    at org.apache.spark.rdd.RDD.iterator(RDD.scala:270)
    at org.apache.spark.rdd.MapPartitionsRDD.compute(MapPartitionsRDD.scala:38)
2
mysqljdbcapache-sparkpysparkcloudera-cdh

2 Answers

1
votes

finally find...you need under conf,there is a default.conf file,add spark.executor.extraClassPath mysql.jar,so the executor can find the driver.

0
votes

That's failure when I use the link like:

--driver-class-path /usr/share/java/mysql-connector-java.jar

But that's ok when I use the full file path like:

--driver-class-path /usr/share/java/mysql-connector-java-5.1.28.jar