Starting thrift server in spark

3
votes

Can anyone help me with starting spark thrift server? I am running my script in standalone mode and I want to fetch data in my business intelligence tool.

In order to do that I need to start thrift server. I tried running shell script: 

$SPARK_HOME/sbin/start-thriftserver.sh

but I get an error:

error "ps unknown option --0"

2
apache-sparkapache-spark-sqlspark-thriftserver
Provide more debug info, start server with bash -x sbin/start-thriftserver.sh and attach log to the question. - Mariusz
--ps: unknown option -- o (NewLine)Try ps --help' for more information. starting org.apache.spark.sql.hive.thriftserver.HiveThriftServer2, logging to /e/Development/spark-2.0.1-bin-hadoop2.7/logs/spark--org.apache.spark.sql.hive.thriftserver.HiveThriftServer2-1-TEAMTAPPS.out After this below line keeps up repeating ps: unknown option -- o Try ps --help' for more information. - Bhanuday Birla
Possible duplicate of stackoverflow.com/questions/36593446/… - Mariusz
@mariusz ...the link provides solution to start master but i want to start thrift server. - Bhanuday Birla
Yes, but this applies to all server scripts, spark-thriftserver is one of them. - Mariusz

2 Answers

8
votes

In Spark 2.2.1

cd %SPARK_HOME%\bin
spark-class org.apache.spark.deploy.SparkSubmit --class org.apache.spark.sql.hive.thriftserver.HiveThriftServer2 spark-internal

this started the spark thrift server in windows. In another terminal run the below beeline command to open a jdbc connection

cd %SPARK_HOME%\bin
beeline -u jdbc:hive2://localhost:10000
5
votes

The sbin/ scripts work fine under linux, but they are not prepared to run in Windows (see Failed to start master for Spark in Windows). But spark thriftserver can be start in foreground independent from OS using command:

java -cp conf/:jars/* org.apache.spark.deploy.SparkSubmit --class org.apache.spark.sql.hive.thriftserver.HiveThriftServer2 spark-internal

Then you can test connection using beeline supplied with spark:

$ ./bin/beeline -u jdbc:hive2://localhost:10000
Connecting to jdbc:hive2://localhost:10000
16/11/22 13:09:57 INFO Utils: Supplied authorities: localhost:10000
16/11/22 13:09:57 INFO Utils: Resolved authority: localhost:10000
16/11/22 13:09:57 INFO HiveConnection: Will try to open client transport with JDBC Uri: jdbc:hive2://localhost:10000
Connected to: Spark SQL (version 2.0.1)
Driver: Hive JDBC (version 1.2.1.spark2)
Transaction isolation: TRANSACTION_REPEATABLE_READ
Beeline version 1.2.1.spark2 by Apache Hive
0: jdbc:hive2://localhost:10000> show databases;
+---------------+--+
| databaseName  |
+---------------+--+
| default       |
| elo           |
+---------------+--+
2 rows selected (0,26 seconds)