Unable to stop Kerberos debug logging

问题

I'm using a Kerberos enabled Spark cluster for running our Spark applications. The Kerberos has been setup previously by other members of the organization, and I have no idea how it works. In the initial days, we had used the Kerberos debug logs to understand the exception "Unable to obtain password from user" which was being raised due to absence of a JCE certificate in the cacerts folder of jre security. However, we no longer require the logs and thus, used the -Dsun.security.krb5.debug=false parameter to disable the logging. However, this did not have any effect. Is there any other parameter that could do the trick? Please help me.

回答1:

Excerpt from the GitBook "Hadoop and Kerberos: The Madness Beyond the Gate" by Steve Loughran, chapter Low-Level Secrets

JVM Library logging

You can turn Kerberos low-level logging on
-Dsun.security.krb5.debug=true

This doesn't come out via Log4J, or java.util logging; it just comes out on the console. Which is somewhat inconvenient —but bear in mind they are logging at a very low level part of the system. And it does at least log. If you find yourself down at this level you are in trouble. Bear that in mind.

If you want to debug what is happening in SPNEGO, another system property lets you enable this:
-Dsun.security.spnego.debug=true

You can ask for both of these in the HADOOP_OPTS environment variable
export HADOOP_OPTS="-Dsun.security.krb5.debug=true -Dsun.security.spnego.debug=true"

Hadoop-side JAAS debugging

Set the env variable HADOOP_JAAS_DEBUG to true and UGI will set the "debug" flag on any JAAS files it creates.

You can do this on the client, before issuing a hadoop, hdfs or yarn command, and set it in the environment script of a YARN service to turn it on there.
export HADOOP_JAAS_DEBUG=true

On the next Hadoop command, you'll see a trace like (.........)

Caveat: the Java properties starting with sun.security. apply to Sun/Oracle Java run-time, and also OpenJDK run-time and its variants. But not to IBM Java, etc.

Excerpt from the Java 8 documentation under Troubleshooting Security

If you want to monitor security access, you can set the java.security.debug System property.
(.......) Separate multiple options with a comma.

When troubleshooting Kerberos specifically, I personally use that combination:
-Djava.security.debug=gssloginconfig,configfile,configparser,logincontext

Excerpt from the Oracle JDK 9 Release Notes section tools/launcher

JDK 9 supports a new environment variable JDK_JAVA_OPTIONS to prepend options to those specified on the command line. The new environment variable has several advantages over the legacy/unsupported _JAVA_OPTIONS environment variable including the ability to include java launcher options (...)

These two env variables are a very dirty (and utterly difficult to detect) way to inject Java system properties without them appearing on the command line.

What does that mean for you? Well, you have to search for multiple Java system props and environment variables, which might be set

for env variables: globally (cf. /etc/profile.d/*.sh), or at account level (cf. ~/.bashrc and friends), or inside Hadoop "include files", or directly inside a shell script that runs your Spark job
for system props: in any shell-or-env variable that is later developed in a shell script (...) or any env var picked up by Java on startup, or in YARN configuration files (when using Spark-on-YARN), or directly on a Java command-line

Good luck.
I personally would run a dummy Spark job that just dumps all env variables and Java system props; then inspect the dump to detect what to search for; then run a brute-force find ... -exec grep ... on the Linux filesystem (repeat as needed).

来源：https://stackoverflow.com/questions/58762997/unable-to-stop-kerberos-debug-logging

标签

java

apache-spark

kerberos