In yarn-cluster mode, the Spark client uses the local Kerberos ticket to connect to Hadoop services and retrieve special auth tokens that are then shipped to the YARN container running the driver then the driver broadcasts the token to the executors

Kerberos authentication with Hadoop cluster from Spark stand alone cluster running on Kubernetes cluster This is a situation with two clusters.

I did restart all services on the cluster, but that didn't help. I don't understand what is meant with 'Spark will automatically obtain a token for Hive ' though. The user has contributor access to the blob storage (not sure if that is actually needed). Well, the user launching the application has valid ticket, as can be seen in the output of klist. This is normally done at launch time: in a secure cluster Spark will automatically obtain a token for the cluster’s HDFS filesystem, and potentially for HBase and Hive.

To start with the official documentation: it explains thatįor a Spark application to interact with HDFS, HBase and Hive, it must acquire the relevant tokens using the Kerberos credentials of the user launching the application -that is, the principal whose identity will become that of the launched Spark application.

spark-submit,Client cannot authenticate via:.

(I couldn't find similar documentation from Microsoft).

spark-submit failing to connect to metastore due to Kerberos : Caused by GSSException: No valid credentials provided.

Issue with Spark Java API, Kerberos, and Hive.

Spark on YARN + Secured hbase For HBase instead of Hive, but same conclusion.

It appears that many other answers/websites also suggest to specify principal/keytab explicitly:

I created a keytab for my user and specified that file instead, but that didn't help.

Spark.sql('CREATE DATABASE IF NOT EXISTS `.

Should we be using Kerberos to authenticate to Hive, and if yes, how? As detailed below, I think we have to specify keytab and principal, but I don't know what exactly. We recently enabled Kerberos authentication on our Spark cluster, but we found that when we submit Spark jobs in cluster mode, the code cannot connect to Hive.

Enable adobe xd viewer file