potentially merging atmos and ocean hubs

[charlesbluca]

As we work on consolidating the many hubs we once had running, do we feel there is a need for a distinct hub dedicated to atmospheric science - i.e. a new Atmos deployment to match the one created for Ocean?

Initially getting involved with this project, my intention was to upload data generated as part of TRACMIP to Pangeo's cloud bucket and play around with it in a hub deployed specifically for atmospheric sciences. However, it seems like we may be able to get by using the hub for oceanography.

Is there interest within the community to continue maintaining a hub specifically for atmospheric sciences?

[rabernat]

Thanks for raising this issue Charles.

We collectively should try to make some decisions about how we want to organize these hubs. After pangeo.pydata.org, we had this idea to split out into many different hubs. Now we are heading in the opposite direction! I'm interested in hearing from others involved in this work what would be a good strategy.

[rabernat]

Perhaps we want to forget about ocean as well and just all use hub.pangeo.io.

[robfatland]

Hope this is relevant:

For me it is simpler therefore preferable to use an existing hub; and maybe these implicit issues don't apply but I perceive them as: What is the time frame we "promise" to keep the lights on for? Who is paying for the cycles? Is there an admin supporting my group's participation? If so who is that? How do I customize my team's environments on that hub ("I need to conda install NetworkX")? What happens when my team decides to host a Hackweek with 70 users (or other burst or expansion scenarios)?

As pangeo has a lot of AWS credits at the moment this maybe isn't too pressing but perhaps it will be good to be ready for when those expire.

[rabernat]

What is the time frame we "promise" to keep the lights on for?

At least until the end of our NSF award (September 2020)

Who is paying for the cycles?

NSF / Google

Is there an admin supporting my group's participation? If so who is that?

No

How do I customize my team's environments on that hub ("I need to conda install NetworkX")?

Currently the process would be to open an issue or make a pull request here. We would like to move to a situation where users can bring their own images, but we don't know how to do that right now.

What happens when my team decides to host a Hackweek with 70 users (or other burst or expansion scenarios)?

This should not be a problem.

[jhamman]

From my perspective, there is still some value in maintaining multiple hubs. I'll make a few arguments for keeping these separate hubs for now.

• Marketing. I think it useful to have a atmos.pangeo.io deployment as a way of saying Pangeo works for Atmospheric Science here.
• Administrative challenges. We're using hubploy to build notebook images and deploy the juypyterhub helm chart. Currently, we're confined to a single image per deployment (Support multiple images  berkeley-dsep-infra/hubploy#10). This means everyone on ocean.pangeo.io will be confined to the same image. Our experience from pangeo.pydata.org tells us that as the community using a single hub grows, it becomes difficult to balance competing demands on that single image.

Longer term, I'd like to see the following happen -

• Better support for multiple images
• Change the delineation from science domain to cloud/region. Maybe we would have aws-us-east-1.pangeo.io instead of ocean.pangeo.io.

[rabernat]

@jhamman - I agree with what you wrote. I really like the idea of cloud-region specific clusters.

For now, let's move the atmos config here so we have everything in one place. And let's plug it into the same GKE cluster as ocean and hub. @charlesbluca - can you make a pull request to transfer the atmos configuration to this repo? You can basically just start from scratch by copying the ocean config and changing a few lines. We can iterate with you and help you get the setup right.

[rabernat]

Hi @charlesbluca--how can we help you with the transfer of the atmos hub?