kerberos环境下spark读取kafka写hbase,Spark on YARN + Secured hbase

kerberos环境下spark读取kafka写hbase,Spark on YARN + Secured hbase

6            2

I am submitting a job to YARN (on spark 2.1.1 + kafka 0.10.2.1) which connects to a secured hbase cluster. This job, performs just fine when i am running in "local" mode (spark.master=local[*]).

However, as soon as I submit the job with master as YARN (and deploy mode as client), I see the following error message -

Caused by: javax.security.auth.login.LoginException: Unable to obtain password from user

I am following hortonworks recommendations for providing information to yarn cluster regarding the HBase and keytab etc. Followed this kb article -

Any pointers what could be going on ?

the mechanism for logging into HBase:

UserGroupInformation.setConfiguration(hbaseConf)
val keyTab = "keytab-location") 
val principal = "kerberos-principal"
val ugi = UserGroupInformation.loginUserFromKeytabAndReturnUGI(principal, keyTab)
UserGroupInformation.setLoginUser(ugi)
ugi.doAs(new PrivilegedExceptionAction[Void]() {

override def run: Void = {
  hbaseCon = Some(ConnectionFactory.createConnection(hbaseConf))
  null
}
})

Also, I tried the alternative mechanism for logging in, as:

UserGroupInformation.loginUserFromKeytab(principal, keyTab)
connection=ConnectionFactory.createConnection(hbaseConf)

please suggest.             follow

www.zeeklog.com  - kerberos环境下spark读取kafka写hbase,Spark on YARN + Secured hbase

258k6262 gold badges769769 silver badges829829 bronze badges                 asked May 30 '17 at 14:53

www.zeeklog.com  - kerberos环境下spark读取kafka写hbase,Spark on YARN + Secured hbase

1,89755 gold badges2424 silver badges4444 bronze badges

1 Answer                     12

You are not alone in the quest for Kerberos auth to HBase from Spark, cf.

A little-known fact is that Spark now generates Hadoop "auth tokens" for Yarn, HDFS, Hive, HBase on startup. These tokens are then broadcasted to the executors, so that they don't have to mess again with Kerberos auth, keytabs, etc.

The first problem is that it's not explicitly documented, and in case of failure the errors are hidden by default (i.e. most people don't connect to HBase with Kerberos, so it's usually pointless to state that the HBase JARs are not in the CLASSPATH and no HBase token was created... usually.)
To log all details about these tokens, you have to set the log level for org.apache.spark.deploy.yarn.Client to DEBUG.

The second problem is that beyond the properties, Spark supports many env variables, some documented, some not documented, and some actually deprecated.
For instance, SPARK_CLASSPATH is now deprecated, and its content actually injected in Spark properties spark.driver / spark.executor.extraClassPath.
But SPARK_DIST_CLASSPATH is still in use, and in the Cloudera distro for example, it is used to inject the core Hadoop libs & config into the Spark "launcher" so that it can bootstrap a YARN-cluster execution, before the driver is started (i.e. before spark.driver.extraClassPath is evaluated).
Other variables of interest are

  • HADOOP_CONF_DIR
  • SPARK_CONF_DIR
  • SPARK_EXTRA_LIB_PATH
  • SPARK_SUBMIT_OPTS
  • SPARK_PRINT_LAUNCH_COMMAND

The third problem is that, in some specific cases (e.g. YARN-cluster mode in the Cloudera distro), the Spark property spark.yarn.tokens.hbase.enabled is set silently to false -- which makes absolutely no sense, that default is hard-coded to true in Spark source code...!
So you are advised to force it explicitly to true in your job config.

The fourth problem is that, even if the HBase token has been created at startup, then the executors must explicitly use it to authenticate. Fortunately, Cloudera has contributed a "Spark connector" to HBase, to take care of this kind of nasty stuff automatically. It's now part of the HBase client, by default (cf. hbase-spark*.jar).

The fifth problem is that, AFAIK, if you don't have metrics-core*.jar in the CLASSPATH then the HBase connections will fail with puzzling (and unrelated) ZooKepper errors.


¤¤¤¤¤ How to make that stuff work, with debug traces

# we assume that spark-env.sh and spark-default.conf are already Hadoop-ready,
# and also *almost* HBase-ready (as in a CDH distro);
# especially HADOOP_CONF_DIR and SPARK_DIST_CLASSPATH are expected to be set
# but spark.*.extraClassPath / .extraJavaOptions are expected to be unset

KRB_DEBUG_OPTS="-Dlog4j.logger.org.apache.spark.deploy.yarn.Client=DEBUG -Dlog4j.logger.org.apache.hadoop.hbase.zookeeper.RecoverableZooKeeper=DEBUG -Dlog4j.logger.org.apache.hadoop.hbase.client.ConnectionManager$HConnectionImplementation=DEBUG -Dlog4j.logger.org.apache.hadoop.hbase.spark.HBaseContext=DEBUG -Dsun.security.krb5.debug=true -Djava.security.debug=gssloginconfig,configfile,configparser,logincontext"
EXTRA_HBASE_CP=/etc/hbase/conf/:/opt/cloudera/parcels/CDH/lib/hbase/hbase-spark.jar:/opt/cloudera/parcels/CDH/lib/hbase/lib/metrics-core-2.2.0.jar

export SPARK_SUBMIT_OPTS="$KRB_DEBUG_OPTS"
export HADOOP_JAAS_DEBUG=true
export SPARK_PRINT_LAUNCH_COMMAND=True

spark-submit --master yarn-client \
  --files "/etc/spark/conf/log4j.properties#yarn-log4j.properties" \
  --principal [email protected] --keytab /a/b/XX.keytab \
  --conf spark.yarn.tokens.hbase.enabled=true \
  --conf spark.driver.extraClassPath=$EXTRA_HBASE_CP \
  --conf spark.executor.extraClassPath=$EXTRA_HBASE_CP \
  --conf "spark.executor.extraJavaOptions=$KRB_DEBUG_OPTS -Dlog4j.configuration=yarn-log4j.properties" \
  --conf spark.executorEnv.HADOOP_JAAS_DEBUG=true \
  --class TestSparkHBase  TestSparkHBase.jar

spark-submit --master yarn-cluster --conf spark.yarn.report.interval=4000 \
  --files "/etc/spark/conf/log4j.properties#yarn-log4j.properties" \
  --principal [email protected] --keytab /a/b/XX.keytab \
  --conf spark.yarn.tokens.hbase.enabled=true \
  --conf spark.driver.extraClassPath=$EXTRA_HBASE_CP \
  --conf "spark.driver.extraJavaOptions=$KRB_DEBUG_OPTS -Dlog4j.configuration=yarn-log4j.properties" \
  --conf spark.driverEnv.HADOOP_JAAS_DEBUG=true \
  --conf spark.executor.extraClassPath=$EXTRA_HBASE_CP \
  --conf "spark.executor.extraJavaOptions=$KRB_DEBUG_OPTS -Dlog4j.configuration=yarn-log4j.properties" \
  --conf spark.executorEnv.HADOOP_JAAS_DEBUG=true \
  --class TestSparkHBase  TestSparkHBase.jar

PS: when using a HBaseContext you don't need /etc/hbase/conf/ in the executor's CLASSPATH, the conf is propagated automatically.

PPS: I advise you to set log4j.logger.org.apache.zookeeper.ZooKeeper=WARN in log4j.properties because it's verbose, useless, and even confusing (all the interesting stuff is logged at HBase level)

PPS: instead of that verbose SPARK_SUBMIT_OPTS var, you could also list statically the Log4J options in $SPARK_CONF_DIR/log4j.properties and the rest in $SPARK_CONF_DIR/java-opts; same goes for the Spark properties in $SPARK_CONF_DIR/spark-defaults.conf and env variables in $SPARK_CONF_DIR/spark-env.sh


¤¤¤¤¤ About the "Spark connector" to HBase

Excerpt from the , chapter 83 Basic Spark

At the root of all Spark and HBase integration is the HBaseContext. The HBaseContext takes in HBase configurations and pushes them to the Spark executors. This allows us to have an HBase Connection per Spark Executor in a static location.

What is not mentioned in the doc is that the HBaseContext uses automatically the HBase "auth token" (when present) to authenticate the executors.

Note also that the doc has an example (in Scala then in Java) of a Spark foreachPartition operation on a RDD, using a BufferedMutator for async bulk load into HBase.

Read more

🚀Zeek.ai一款基于 Electron 和 Vite 打造的跨平台(支持 Windows、macOS 和 Linux) AI 浏览器

🚀Zeek.ai一款基于 Electron 和 Vite 打造的跨平台(支持 Windows、macOS 和 Linux) AI 浏览器

是一款基于 Electron 和 Vite 打造的跨平台(支持 Windows、macOS 和 Linux) AI 浏览器。 集成了 SearXNG AI 搜索、开发工具集合、 市面上最流行的 AI 工具门户,以及代码编写和桌面快捷工具等功能, 通过模块化的 Monorepo 架构,提供轻量级、可扩展且高效的桌面体验, 助力 AI 驱动的日常工作流程。

By Ne0inhk
超快速,使用ChatGPT编写回归和分类算法

超快速,使用ChatGPT编写回归和分类算法

本文将使用一些 ChatGPT 提示,这些提示对于数据科学家在工作时非常重要。 微信搜索关注《Python学研大本营》,加入读者群,分享更多精彩 以下是一些示例ChatGPT 提示的列表以及数据科学家的响应。 ChatGPT 提示 为决策树回归算法生成 python 代码。 下面是使用scikit-learn在 Python 中进行决策树回归的示例代码: import numpy as np import matplotlib.pyplot as plt from sklearn.tree import DecisionTreeRegressor # Generate random data rng = np.random.default_rng() x = 5 * rng.random(100) y = np.sin(x) + 0.

By Ne0inhk
力扣每日一题:993.二叉树的堂兄弟节点 深度优先算法

力扣每日一题:993.二叉树的堂兄弟节点 深度优先算法

993.二叉树的堂兄弟节点 难度:简单 题目: 在二叉树中,根节点位于深度 0 处,每个深度为 k 的节点的子节点位于深度 k+1 处。 如果二叉树的两个节点深度相同,但 父节点不同 ,则它们是一对堂兄弟节点。 我们给出了具有唯一值的二叉树的根节点 root ,以及树中两个不同节点的值 x 和 y 。 只有与值 x 和 y 对应的节点是堂兄弟节点时,才返回 true 。否则,返回 false。 示例: 示例 1: 输入:root = [1,2,3,4], x = 4, y = 3 输出:false

By Ne0inhk