How to increase Jetty's request/response header size of KyuubiRestFrontendService?

[minyk]

Hi,

We're starting to change from our own SparkUI proxy service to kyuubi's proxy /engine-ui.

Because kyuubi-server is sitting behind some gateways, there're too many(?) request headers like this:

2024-03-07 11:23:57.400 KyuubiRestFrontendService-1637 DEBUG org.eclipse.jetty.client.HttpSender: Request headers HttpRequest[GET /jobs/ HTTP/1.1]@52519fe9
x-b3-parentspanid: 31b132d98719ade4
sec-fetch-site: none
x-b3-sampled: 0
User-Agent: Mozilla/5.0 (Macintosh; Intel Mac OS X 10_15_7) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/122.0.0.0 Safari/537.36
Accept-Encoding: gzip, deflate, br, zstd
sec-fetch-user: ?1
Authorization: Bearer ...
sec-ch-ua-mobile: ?0
pinpoint-pappnamespace: com.pinpoint
pinpoint-spanid: 2056943685799885569
pinpoint-flags: 0
ndc-x-forwarded-host: ndapdevtest
upgrade-insecure-requests: 1
pinpoint-pspanid: -891978538187517349
Content-Length: 0
pinpoint-pappname: ndc-gateway
sec-fetch-mode: navigate
Cookie: ...
x-request-id: b7500163-4917-4c62-a700-4e5898445852
Accept: text/html,application/xhtml+xml,application/xml;q=0.9,image/avif,image/webp,image/apng,*/*;q=0.8,application/signed-exchange;v=b3;q=0.7
pinpoint-traceid: ndc-gateway^1709689013074^16231
X-Forwarded-Host: ndapdevtest-ndap-kyuubi.nexr.kr,ndapdevtest-ndap-kyuubi.nexr.kr
X-Forwarded-Proto: https,https
pinpoint-papptype: 1210
x-envoy-attempt-count: 1
sec-ch-ua: "Chromium";v="122", "Not(A:Brand";v="24", "Google Chrome";v="122"
pinpoint-host: ndapdevtest-ndap-ndap-kyuubi-server.ndapdevtest.svc.cluster.local.gateway.svc.cluster.local:10099
x-b3-traceid: f8b3fdee1f4075a1b5424a877f2b2773
x-b3-spanid: d3506856e67fe716
sec-ch-ua-platform: "macOS"
Forwarded: proto=https;host=ndapdevtest-ndap-kyuubi.nexr.kr;for="x.x.x.x:45975"

So I make a quick change like KyuubiTHttpFrontendService to set request/response header size: minyk@f4bbc6f, but had no lock.

Still too large header error:

2024-03-07 11:23:57.401 KyuubiRestFrontendService-1637 DEBUG org.eclipse.jetty.client.HttpSender: Generated headers (4096 bytes), chunk (-1 bytes), content (0 bytes) - HEADER_OVERFLOW/HttpGenerator@48e4e170{s=START}
2024-03-07 11:23:57.401 KyuubiRestFrontendService-1637 DEBUG org.eclipse.jetty.client.HttpSender: Request failure HttpRequest[GET /jobs/ HTTP/1.1]@52519fe9
org.eclipse.jetty.http.BadMessageException: 500: Request header too large
	at org.eclipse.jetty.client.http.HttpSenderOverHTTP$HeadersCallback.process(HttpSenderOverHTTP.java:235) [jetty-client-9.4.52.v20230823.jar:9.4.52.v20230823]
	at org.eclipse.jetty.util.IteratingCallback.processing(IteratingCallback.java:248) [jetty-util-9.4.52.v20230823.jar:9.4.52.v20230823]
	at org.eclipse.jetty.util.IteratingCallback.iterate(IteratingCallback.java:229) [jetty-util-9.4.52.v20230823.jar:9.4.52.v20230823]
	at org.eclipse.jetty.client.http.HttpSenderOverHTTP.sendHeaders(HttpSenderOverHTTP.java:65) [jetty-client-9.4.52.v20230823.jar:9.4.52.v20230823]
	at org.eclipse.jetty.client.HttpSender.send(HttpSender.java:212) [jetty-client-9.4.52.v20230823.jar:9.4.52.v20230823]
	at org.eclipse.jetty.client.http.HttpChannelOverHTTP.send(HttpChannelOverHTTP.java:84) [jetty-client-9.4.52.v20230823.jar:9.4.52.v20230823]

if header is smaller than 4000, confirm passed:

2024-03-07 11:27:51.760 KyuubiRestFrontendService-1615 DEBUG org.eclipse.jetty.client.HttpSender: Generated headers (3888 bytes), chunk (-1 bytes), content (0 bytes) - FLUSH/HttpGenerator@3edef09c{s=COMPLETING}

Am I miss something for jetty's configurations?

Thank you.

[pan3793]

we only expose a few Jetty configurations, do you know which vanilla Jetty configuration affects that? we may want to expose it to Kyuubi conf, or passthrough all Jetty configurations via a dedicated prefix in Kyuubi conf.

[minyk]

@pan3793 passthrough way is best, not I don't have an idea for the notation of configs. I'll try somethings.

[yaooqinn]

Hi @minyk, would you mind submitting a pull request for this? I think it would be beneficial if we made it configurable.

I have found a PR from apache zeppelin apache/zeppelin#2663

[yaooqinn]

apache/spark#23090

[minyk]

@yaooqinn I saw that PR, but I did not understand why they use cf.getHttpConfiguration().setRequestHeaderSize(requestHeaderSize). I'll try their way then I'll make a PR for this after solved.

[minyk]

Also, this change might affect to Trino Engine:

[minyk]

I found a way. request header size is not the problem, but HttpClient's request buffer size. Codes like below:

diff --git a/kyuubi-server/src/main/scala/org/apache/kyuubi/server/api/v1/ApiRootResource.scala b/kyuubi-server/src/main/scala/org/apache/kyuubi/server/api/v1/ApiRootResource.scala
--- a/kyuubi-server/src/main/scala/org/apache/kyuubi/server/api/v1/ApiRootResource.scala	(revision ddf36633b13b9baf1c2b6555137868a674b85a37)
+++ b/kyuubi-server/src/main/scala/org/apache/kyuubi/server/api/v1/ApiRootResource.scala	(revision d3041bf2d6894b966844aae14d0243026062cbfb)
@@ -86,6 +86,7 @@
   def getEngineUIProxyHandler(fe: KyuubiRestFrontendService): ServletContextHandler = {
     val proxyServlet = new EngineUIProxyServlet()
     val holder = new ServletHolder(proxyServlet)
+    holder.setInitParameter("requestBufferSize", "6144")
     val proxyHandler = new ServletContextHandler(ServletContextHandler.NO_SESSIONS)
     proxyHandler.setContextPath("/engine-ui")
     proxyHandler.addServlet(holder, "/*")

I'll make a PR for this change.

[minyk]

We have some candidates. In jetty's doc: https://eclipse.dev/jetty/javadoc/jetty-9/org/eclipse/jetty/proxy/AbstractProxyServlet.html#createHttpClient()

config	default	description
maxThreads	256	The max number of threads of HttpClient's Executor. If not set, or set to the value of "-", then the Jetty server thread pool will be used.
maxConnections	32768	The max number of connections per destination, see HttpClient.setMaxConnectionsPerDestination(int)
idleTimeout	30000	The idle timeout in milliseconds, see HttpClient.setIdleTimeout(long)
timeout	60000	The total timeout in milliseconds, see Request.timeout(long, java.util.concurrent.TimeUnit)
requestBufferSize	HttpClient's default	The request buffer size, see HttpClient.setRequestBufferSize(int)
responseBufferSize	HttpClient's default	The response buffer size, see HttpClient.setResponseBufferSize(int)

And there're two handlers in ApiRootResource. Both servlet need this configurations?

WDYT?