Infrastructure Professionalization for the Managed JupyterHub Service v1

[choldgraf]

Summary

We need our Managed JupyterHub Service to reach a certain level of professionalization before we can consider it "production ready". This includes having a sound architectural design, robust deployment infrastructure, and a reliable experience in deploying all the kinds of hubs we wish to deploy.

Benefit

The reason to focus explicitly on professionalization is so that we do not lose time in the future doing random de-bugs and putting out fires because our infrastructure is not working as expected. It is an intentional investment of our time now in order to avoid unintentionally spending a lot of our time on operations work in the future.

Acceptance criteria

We'll need to decide as a team what "good enough" looks like here. I'd suggest something like:

Each member of the Engineering team is confident that they could deploy all of our hubs on their own, that the hubs would operate as-expected without issue, and that we feel confident telling Hub Administrators to use our hubs for important work.

Tasks to complete

• Automate & document full process of setting up a new hub #336
• Automate hub deployment for all our clusters #507
• Deploy pangeo-hubs automatically #577
• Improve the hub health check in deployer #586
• Fix in-notebook config for culling kernels #518
• Coalesce daskhub into basehub #502
• Backup JupyterHub databases #452
• Provide more information during CI failures #449
• Helm template validation in our CI/CD #279
• Automatically test a dask-gateway notebook for daskhubs #152
• Test YAML validity for each PR #165
• Fix our Grafana / Prometheus data connection #616
• Operate a staging cluster for our pilot hubs team-compass#228

[choldgraf]

I'm closing this as Yuvi included much of these in #610