Cannot connect to Hive metastore from Spark application

Question

I am trying to connect to Hive-metastore from the Spark application but each time it gets stuck on trying to connect and crash with a timeout:

INFO  metastore:376 - Trying to connect to metastore with URI thrift://hive-metastore:9083
WARN  metastore:444 - set_ugi() not successful, Likely cause: new client talking to old server. Continuing without it.
org.apache.thrift.transport.TTransportException: java.net.SocketTimeoutException: Read timed out

The application crashes on the line where I create an external Hive table

I run Hive-metastore as well as Spark application (using Spark K8s operator) in Kubernetes cluster. I checked the accessibility of Hive-metastore service outside of the cluster using telnet (node ip: service node port) and curled the service inside of the cluster, the service seems to be assessable. What could be the reason of this error?

This is the configuration of Hive-metastore uri in Spark application

val sparkSession = SparkSession
  .builder()
  .config(sparkConf)
  .config("hive.metastore.uris", "thrift://hive-metastore:9083")
  .config("hive.exec.dynamic.partition", "true")
  .config("hive.exec.dynamic.partition.mode", "nonstrict")
  .enableHiveSupport()
  .getOrCreate()

Hive-metastore yaml configuration looks like this:

apiVersion: v1
kind: Service
metadata:
  name: hive-metastore-np
spec:
  selector:
    app: hive-metastore
  ports:
    - protocol: TCP
      targetPort: 9083
      port: 9083
      nodePort: 32083
  type: NodePort
---
apiVersion: apps/v1
kind: Deployment
metadata:
  name: hive-metastore
spec:
  replicas: 1
  selector:
    matchLabels:
      app: hive-metastore
  template:
    metadata:
      labels:
        app: hive-metastore
    spec:
      containers:
        - name: hive-metastore
          image: mozdata/docker-hive-metastore:1.2.1
          imagePullPolicy: Always
          env:
            - name: DB_URI
              value: postgresql
            - name: DB_USER
              value: hive
            - name: DB_PASSWORD
              value: hive-password
            - name: CORE_CONF_fs_defaultFS
              value: hdfs://hdfs-namenode:8020
          ports:
            - containerPort: 9083

UPDATE: When I try to curl hive-metastore:9083, the service is accessible but it returns an empty response which means there might be a problem with hive-metastore K8s definition

> GET / HTTP/1.1
> User-Agent: curl/7.35.0
> Host: hive-metastore:9083
> Accept: */*

I have a one node cluster with Minikube so I could check only one node if you mean it — Cassie

K.Naga K.Naga · Accepted Answer · 2019-02-25T17:37:09

This error occurs when there's a discrepancy between versions of hive jars in your cluster and the hive jars that Spark uses (which is usually consistent with the Spark version you're using). You need to determine the version of hive jars used in the cluster and add these jars into your Spark image. You can then make your SparkSession use those compatible hive jars by adding the following configurations to your SparkSession:

  .conf("spark.sql.hive.metastore.version", "<your hive metastore version>")
  .conf("spark.sql.hive.metastore.version", "<your hive version>")
  .conf("spark.sql.hive.metastore.jars", "<uri of all the correct hive jars>")

Cannot connect to Hive metastore from Spark application

1 Answers