[DH-3] try using a test fork of the chp

[shaneknapp]

@consideRatio pointed me to this... there were a couple of memory leaks in node-http-proxy that haven't been rolled in to a release yet, so let's test this out and see if it helps.

jupyterhub/configurable-http-proxy#434 (comment)
http-party/node-http-proxy#1559
Jimbly/http-proxy-node16@56283e3
https://github.com/consideRatio/configurable-http-proxy/commits/main/

[ryanlovett]

This looks like valid chart yaml to me, based on https://github.com/jupyterhub/zero-to-jupyterhub-k8s/blob/main/jupyterhub/templates/proxy/deployment.yaml and https://z2jh.jupyter.org/en/stable/resources/reference.html#proxy-chp-extracommandlineflags.

At first I thought the timeouts were 1000 days, but I see the units are milliseconds.

[shaneknapp]

hmm, interesting. i did a manual deployment to test this out on a staging hub and got the following error in the chp pod: error: unknown option '--proxyTimeout=86400000'

timeout worked just fine... i'll comment out proxyTimeout and re-test my deployment.

[shaneknapp]

ok, commenting out proxyTimeout let me deploy, and timeout is showing up in the pod config:

Containers:
  chp:
    Container ID:  containerd://2400011a0b0b7fca551735393d81b8b6b1efc60b2080e3061ce669c526844098
    Image:         quay.io/jupyterhub/configurable-http-proxy:4.6.1-fork
    Image ID:      quay.io/jupyterhub/configurable-http-proxy@sha256:55345bfa2f47f4d0c995e25d8cd14a09ce20b0aaeed65df648ec818441629c87
    Ports:         8000/TCP, 8001/TCP
    Host Ports:    0/TCP, 0/TCP
    Command:
      configurable-http-proxy
      --ip=
      --api-ip=
      --api-port=8001
      --default-target=http://hub:$(HUB_SERVICE_PORT)
      --error-target=http://hub:$(HUB_SERVICE_PORT)/hub/error
      --port=8000
      --timeout=86400000

[shaneknapp]

ah, it's proxy-timeout! https://github.com/jupyterhub/configurable-http-proxy?tab=readme-ov-file#command-line-options

[shaneknapp]

there we go!

Containers:
  chp:
    Container ID:  containerd://99cab96888c3d6609d224e6b78bc9936156a6f98a12d08105f1d7f6396d224fb
    Image:         quay.io/jupyterhub/configurable-http-proxy:4.6.1-fork
    Image ID:      quay.io/jupyterhub/configurable-http-proxy@sha256:55345bfa2f47f4d0c995e25d8cd14a09ce20b0aaeed65df648ec818441629c87
    Ports:         8000/TCP, 8001/TCP
    Host Ports:    0/TCP, 0/TCP
    Command:
      configurable-http-proxy
      --ip=
      --api-ip=
      --api-port=8001
      --default-target=http://hub:$(HUB_SERVICE_PORT)
      --error-target=http://hub:$(HUB_SERVICE_PORT)/hub/error
      --port=8000
      --timeout=86400000
      --proxy-timeout=86400000

[shaneknapp]

@consideRatio fyi, this looks quite promising. i've been running the fork without timeouts on our two biggest/most active hubs since friday last week and have only had one chp oomkill between then and now.

here's the memory usage over the past 24 hours on one:

i'm keeping a close eye on these two hubs for the rest of the week and will roll out to our remaining hubs on monday.

since our core node has plenty of available ram, i'm also bumping the chp's memory limit from 1Gi to 1.5Gi. this should give is a little bit more headroom and minimize (hopefully) and additional oomkills.