Create a Jupyter account for GCP and transfer the gke.mybinder.org project there

[choldgraf]

Background

Google is supplying us credits to power mybinder.org, these are currently run off of the Google Research Team's project, which is powering gke.mybinder.org. Recently Karan reached out and said it would be better if this project were under @fperez's account rather than theirs, so that we have more control over it.

I'm meeting with Karan on Friday to make this transfer. Will post the meeting time below in case anybody else would like to join (if so, let me know here or in a DM!)

Implementation details

I think the best path forward would be to create a Project Jupyter Google Cloud org, and then move this project underneath that organization. We could give ownership access to any member of the JupyterHub team that wanted it.

Tasks

• Meet with Karan on Friday July 2nd at 11am Pacific to try and make a transition
• Does anybody object to the plan here: Create a Jupyter account for GCP and transfer the gke.mybinder.org project there #427 (comment) ?

[choldgraf]

Update from meeting with Karan:

The main thing we need to do is more the binderhub project outside of Karan's org (which is a Google org account) and into some other org. I don't think it matters what org this is, because the billing account (attached to Fernando) should stay the same regardless.

A few options:

• We move to the berkeley.edu Google Cloud org
  • But we should check whether this would come with extra restrictions on access (e.g., "you need an @berkeley.edu address for this")
• We move to the jupyter.org Google Cloud org
  • But I have no idea if this org exists (doesn't anybody know if this org exists?)
• We move to the 2i2c.org Google Cloud org
  • We have the most "control" over this org account and can give everybody access this way

Also just another note, we have about $60,000 in credits remaining, and we have until December 22nd to spend them.

[yuvipanda]

My vote is to create a jupyter.org Google Cloud Org, and move it there. However, maybe you need a google workspace account to 'create' an org?

Also the current binderhub project is not under any organization:

So I'm not sure what Karan means?

[fperez]

On Sun, Aug 8, 2021 at 11:51 PM Yuvi Panda ***@***.***> wrote: My vote is to create a jupyter.org Google Cloud Org, and move it there. Orgs are free, so we should 'just do it'.

Yup, that sounds like a good plan! Even if for now it's mostly a single-task account for binder, it could come in handy in the future to manage other Jupyter services hosted on GCP. We're gradually organizing working groups around various areas better in Jupyter itself (see e.g. the security stuff recently led by Rollin, Rick and Tiffany). This goes in that same direction, so +1 from me.

[choldgraf]

for creating a jupyter.org GCP org, what is the pathway to doing this? Does it need to be done by a specific person (e.g. who has access to the Jupyter Google Workspaces account or something)? @fperez perhaps this is something to discuss with the governance team?

[choldgraf]

Could somebody please point me in a direction here? We are getting close to the end-date of the current credits, and it's not going to look good if we still haven't satisfied Karan's request by then.

I need guidance on: how can I create a Project Jupyter Google cloud org? I would like to create this organization, provide "owner" status to members of the JupyterHub team, and then migrate the binder GCP project to this org.

I've also updated the top comment with (I think) the next steps that need to happen

[consideRatio]

@choldgraf to create a GCP organization, this is a starting point I think.

1. https://cloud.google.com/resource-manager/docs/creating-managing-organization
2. Within that site, you click to sign up for a cloud identity
3. From that site, you choose between a free and premium cloud identity. Choose free!
   The free and premium differs as described here, but my conclusion is that we should absolutely choose the free one as the premium is very enterprise focused and not relevant to us.
4. The link to sign up for a free cloud identity is here: https://workspace.google.com/signup/gcpidentity/welcome#0
5. In this process, you will need control of the jupyter.org domain I believe, but the process should otherwise be well guided I think.

---

PS: GCP organization names can have a display name and an id that must be unique. I like the idea of using the id jupyter-org, the id must be lower case and can't contain . or spaces if I recall correctly. This won't be seen by end users so it's not really important, not even the display name will be seen by end users I think. I think the id will be unchangable, but that the display name will be possible to change.

[consideRatio]

I'd love to learn about the project migration process so I'd be very happy to take part of this in any way.

[choldgraf]

Proposal (any objections?)

I'd like to propose the following actions, I welcome guidance from others to let me know if we should instead open up a JEP about this:

• @consideRatio and I work together to create a jupyter-org organization on Google Cloud
• We move the gke.mybinder.org project underneath this organization
• All JupyterHub Red/Blue team members may request admin access to this organization
• We open up a JEP about "officially adopting" this organization for the Jupyter project more generally
• If that JEP is accepted, we figure out how to implement it (e.g. either move the project again to an existing organization, or just give access to more people like the Steering Council)

[betatim]

Do we need more 👍 on #427 (comment)? To me it looks like enough people have said yes and we have people who want to implement it. That means the next step is to work on it. Is that how everyone else sees it as well or is there something else we need to take care of first?

[consideRatio]

Thanks for nudging this onwards @betatim!

@choldgraf should we try get some work done with regards to this today? We may require control of the jupyter.org domain to do this, but I'm not sure. What domain registration service is used to manage jupyter.org currently? I think we may need access to it in order to set a TXT entry or similar to proove ownership to GCP as part of this.

[yuvipanda]

@consideRatio DNS recoreds for jupyter.org are managed via cloudflare. Although I don't use lastpass, it was what was used to share the password with me. I've added your email to lastpass so you should have access to this as well.

[consideRatio]

Thanks @yuvipanda, I've verified access and I'm all set!

[choldgraf]

@betatim i do have the same impression. @consideRatio I'd love to have a quick session today to understand whether it will be complicated or not.

[choldgraf]

OK @consideRatio and I just spent several hours trying to figure this out (thank you so much @consideRatio) and we made some progress but hit some roadblocks. Here's an update:

What we did

• Create a Google Cloud Organization for the domain jupyter.org.

• Added a TXT record to jupyter.org with the value google-site-verification=<SOMECODE>. This is what Google Workspaces uses to confirm that we control jupyter.org

• Create a Google Cloud Identity that was the first user for the org above. It has these credentials:

  • U: [email protected]
  • P: stored in bitwarden, ask @choldgraf or @consideRatio to send you the password

• Granted Project Owner and Organizational Administrator roles to the people that were on the email with Karan (listed below). We should add others once we confirm that these roles are the correct thing to do w/ Karan:

  • @fperez
  • @choldgraf
  • @consideRatio
  • @minrk

• We also granted Project Owner role to Karan and to [email protected] with the binderhub GCP project

• Tried to migrate the binderhub GCP project, but ran into a "permission denied" response.

• Tried a million permutations of giving as much permissions as possible to ourselves on both projects, but none of these worked :-(

We think that the problem is that the Billing Account powering the GCP project is linked to the berkeley.edu GCP organization, and this has much more restrictive policies that disallow migrating projects associated with their accounts. I really doubt that we will get Berkeley to change these policies.

So I've sent an email to Karan giving him an update about the same steps above, and asking if he can help us figure out the next step.

I think out optimal outcome here would be:

• We use the jupyter.org GCP organization for all GCP-related Jupyter cloud infrastructure
• We create a new billing account that exists under this organization, instead of under Berkeley.edu
• We link the binderhub project to use this billing account instead of the Berkeley.edu one (perhaps in December when the credits are about to expire).

That way, we can have a GCP organizational account that is tied to "The Jupyter Project's DNS record", instead of to UC Berkeley.

Tasks to follow-up

• Figure out where Jupyter credentials are stored centrally (e.g. lastpass) and add the U/P for this account there
• Send an email to Karan to understand how to get past the roadblock listed here
• Take follow-up actions as needed

[yuvipanda]

We think that the problem is that the Billing Account powering the GCP project is linked to the berkeley.edu GCP organization, and this has much more restrictive policies that disallow migrating projects associated with their accounts. I really doubt that we will get Berkeley to change these policies.

Do you have more information on this? I don't see any link between the project and berkeley with the access I have. Berkeley also doesn't have any requirements around projects being funded by them required to be in the berkeley.edu org (the data8x hub is funded with a chart string and isn't in their org, and that isn't a requirement).

[choldgraf]

Update: conversation with Karan and the GCP team

@consideRatio and I spent another hour talking to some GCP folks this morning, and we confirmed a few things and also have some more next steps.

Where the binderhub GCP project currently lives: In a Google Cloud organization called gcp.solutions. This is the org that we need to move away from.

Where is the billing account for the binderhub project?: In a GCP Billing account linked to UC Berkeley under Fernando's name.

Why can't we just move the binderhub project to the jupyter.org GCP organization we created?: Because the billing account is under Berkeley's access policy and this prohibits people from transferring projects without explicit approval, even if the project is not under the UC Berkeley organization (in this case, only the billing account is).

A few options

Here are a few options that we could follow:

1. Disconnect the Binder billing account from UC Berkeley and move it to the jupyter.org account. Doing this will require getting permission from a UC Berkeley administrator, adding a Binder team member as a billing account administrator, and then moving the account over.
2. Wait for this billing account to run out of funding, and shift focus towards the next round of funding. In this case we remain in steady-state until mid-December when our credits run out, and instead spend our energy on Identify credits for the next year of gke.mybinder.org #463 . If we are able to get new credits, we should ensure they are not gifted to UC Berkeley's cloud organization, so we have more flexibility over what is done with them.

There is an extra consideration they brought up about where to move the project. Apparently running our own Google Cloud organization will make us lose some benefits that the academic organizations get - this would primarily by (1) lower egress fees, (2) a 20% discount, and (3) more dedicated support. However, speaking with @consideRatio, it seems like none of these would be deal-breakers, and the benefits of using a jupyter-specific organization probably outweigh the costs.

Tasks to follow up

I think that these are the next steps

• Flora from GCP is going to reach out to her contact at UC Berkeley (she is the Berkeley rep for GCP) and see if we can quickly get binderhub approved for moving the billing account outside of the university.
• Shift our energy towards Identify credits for the next year of gke.mybinder.org #463 instead.

The rationale here is that it'd be nice if we can get Berkeley to let us move accounts, but it will be catastrophic if Binder runs out of credits on GCP. So we should focus on that, and only put time into transferring our pre-existing credits if we know it will be worth it.

I'll update #463 with more information about next steps there.

🙏 THANK YOU @consideRatio for joining these conversations!

[choldgraf]

This is now completed! Close by #470

[minrk]

Heroic! Thanks for persevering.

[betatim]

Nice!

The one thing that I've lost track of is: has the project moved or not? The "closed by #470" made me think "aha that issue will explain it to me" but from a quick look the issue is about updating contact info but I couldn't work out what that meant in terms of "did we move or not?"

[consideRatio]

The GCP org is created, the binderhub GCP project moved, and the associated billing account is moved! We are entirely based on in the GCP organization called jupyter.org now!

[betatim]

Whoop!

Can you add [email protected] to the jupyter.org org?

[consideRatio]

Done! See gitter for a link of relevance!

[choldgraf]

Naive question: what is the difference between being a member of a GCP org vs a project? I believe the permissions of the binderhub project haven't changed at all, but we should make sure that all team members have access to the things they had before! Are there other folks we need to add to the org?

[sgibson91]

Confirming that my old link to the project still works and I can still see the clusters and I don't believe I have been added to the new org

[minrk]

I don't think much happens at the org level other than enforcing policies and quotas that we probably won't use. I think the main thing is that a project must have an owner that's either a person or an org, just like a GitHub repo.

I think the main thing we'll do at the org level is probably create new projects, and potentially billing accounts, depending on funding situations. ~everything else happens at the project level.