need help for "Timeout to launched SPARK_SQL engine with $SPARK_HOME/bin/submit xxxxxx"

[xiaowuzi010599]

This problem occurs in the company environment. I checked all the configuration items to make sure that the configuration is normal. The same configuration can run kyuubi normally in the test cluster, and there are no other errors in the log. Try to increase "kyuubi.session.engine.initialize.timeout" to "PT10M" , but it does not work. I would like to ask if this error is a known problem, and what other troubleshooting suggestions are there?

[pan3793]

Kyuubi will redirect the spark-submit log to the file, have you checked that?

[xiaowuzi010599]

yes,this file only show
"WARN Utils: spark.executor.instances less than spark.dynamicAllocation.minExecutors is invalid,ignoring its setting xxxxx".
As far as I understand it, this warn instruction does not affect normal engine start-up.

[xiaowuzi010599]

yes,this file only show
"WARN Utils: spark.executor.instances less than spark.dynamicAllocation.minExecutors is invalid,ignoring its setting xxxxx".
As far as I understand it, this warn instruction does not affect normal engine start-up.

[pan3793]

it's unexpected that the log file only has one line message, please provide the whole spark-submit command logged in kyuubi log file, and the full spark-submit log file

[xiaowuzi010599]

sor,Please see the description below. I just forgot to click the reply button.

[xiaowuzi010599]

spark-submit :

"beeline -u xxx -n hadoop -e xxx " in the morning, the log is ：

22/09/21 11:27:59 WARN Utils: spark.executor.instances less than spark.dynamicAllocation.minExecutors is invalid, ignoring its setting, please update your configs.

---

But I just tested other users, and the log looks like this：

22/09/21 16:29:54 WARN Utils: spark.executor.instances less than spark.dynamicAllocation.minExecutors is invalid, ignoring its setting, please update your configs.
22/09/21 16:37:37 WARN RetryInvocationHandler: Exception while invoking class org.apache.hadoop.yarn.api.impl.pb.client.ApplicationClientProtocolPBClientImpl.getNewApplication over rm1. Not retrying because failovers (30) exceeded maximum allowed (30)
22/09/21 16:37:37 ERROR SparkContext: Error initializing SparkContext.
22/09/21 16:37:37 WARN YarnSchedulerBackend$YarnSchedulerEndpoint: Attempted to request executors before the AM has registered!
22/09/21 16:37:37 WARN MetricsSystem: Stopping a MetricsSystem that is not running
22/09/21 16:37:37 ERROR SparkSQLEngine: Failed to instantiate SparkSession: User: hadoop is not allowed to impersonate sqlquery1

---

When I upgraded the hive version from 1 to 2 before, I saw a similar error report, but the cluster has been configured and I don't understand why the error is reported here again.

[xiaowuzi010599]

only restart namenode and datanode

[pan3793]

rm should be restarted as well, and i'm not sure about nm.

[xiaowuzi010599]

After I restarted rm, the log file began to report some rights-related errors, which have now been solved. Why did the log file report no errors at all in the morning and began to report errors after switching users in the afternoon? but it doesn't seem to be the core problem. The core problem should be some directory permissions.

[pan3793]

I can not answer your question based on your brief description. Hadoop-related system is quite complex, Kyuubi relies on spark-submit to launch Spark applications, your should make sure you have a correct Spark setup before using Kyuubi, and how to setup Spark on Hadoop is something out of the scope of Kyuubi project.

[xiaowuzi010599]

Well, the Spark3 environment is a new official service environment that I introduced, and it has not been fully verified. This problem has not occurred in the test cluster, and the rm of the test cluster has not been restarted. As you said, the hadoop environment is somewhat complex, but it has been solved. Thank you.