On merging jupyter-lsp org

[Carreau]

The Jupyter Security team was recently informed of a vulnerability that impacts the jupyter-lsp package (which is installed by default with JupyterLab). However, this package's source is hosted under the jupyter-lsp/ GitHub org, rather than an official Jupyter GitHub org like jupyter/ or jupyterlab/. This makes it challenging for us to open private discussion about potential security issues, or review any best practices.

Can the executive council consider merging the jupyter-lsp/ GitHub org into jupyterlab/ or jupyter/?

cc @jtpio @krassowski and @martinRenou

[krassowski]

CC @bollwyvl

I think this was never formally confirmed, but I could agree that jupyter-lsp org is under the Jupyter EC governance (although without representation); indeed, at least one member of the EC was added to the team of owners after the LSP JEP (jupyter/enhancement-proposals#72) was merged. In that case the question is why not make it an official Jupyter GitHub org?

Previously there was a proposal to move the server part to jupyter-server (jupyter-server/team-compass#30) but it was abandoned because there was no bandwidth to sync across repos across orgs. Moving all the repositories to another org rather than splitting out a bit would be easier, but still it would make it harder to manage.

One non-disruptive solution would be adding representative(s) of the security team to the jupyter-lsp org as a security manager(s) (jupyter/security#68), an approach that we discussed on at least two security meetings.

[Carreau]

In that case the question is why not make it an official Jupyter GitHub org?

See #12 among other discussion.

but still it would make it harder to manage.

The question here is for whom ? Having to manage 15+ org and search repositories, or check settings, or remove write permissions from someone because they got their machine compromised in 15 org is definitely not fun.

[krassowski]

Another simple solution is to remove the jupyter-lsp dependency from jupyterlab. It is not really used right now as jupyterlab does not ship with any LSP server.

[krassowski]

Just to spell it out, it looks like we are stuck in a bad place here (either inconveniencing maintainers or security team) because we are using a free produce whereas the platform also offers a paid version which does not have the limitation that the security team is facing:

One of the main differences between GitHub Enterprise Cloud and other plans for GitHub.com is access to an enterprise account. Enterprise accounts provide administrators with a single point of visibility and management across multiple organizations. For more information, see "About enterprise accounts."

Link: https://docs.github.com/en/enterprise-cloud@latest/admin/overview/about-github-enterprise-cloud

[Carreau]

Let's keep the discussion on reducing the number of orgs into a issue to reduce the number of orgs.

Another simple solution is to remove the jupyter-lsp dependency from jupyterlab. It is not really used right now as jupyterlab does not ship with any LSP server.

It's not sufficient for me, if we remove it and jupyter-lsp org is not official then it also need to clearly update the description and the log and actually be unaffiliated.

It also does not solve how we open a security discussion, because we have a report and it affects existing deployment.

[krassowski]

I have now created security managers team in juypter-lsp and invited security team members there.

[jasongrout]

@krassowski and others - what do you think about proposing jupyter-lsp be adopted by the jupyterlab subproject and github org?

[krassowski]

No strong opinion, also depends on what exactly you mean; again we had this discussion before:

• Integrating jupyterlab-lsp into jupyterlab jupyterlab/frontends-team-compass#148 our original intent
• Integrate jupyterlab-lsp into jupyterlab jupyterlab/jupyterlab#12534 - this part happened
• Move jupyterlab-lsp server extension into jupyter-server organization jupyter-server/team-compass#30 - this one did not (equally because there were pros and cons on maintenance side and because no one pushed to finalise the transition, which requires committing a number of hours to split up the monorepo and this is not only extracting out the server part but then also cleaning up the other bit, which no one offered to do).

Keeping it as a monrepo and moving it to jupyterlab org:

• pro: easier maintenance-wise
• pro: better on issue migration (most issues will be filled against jupyterlab anyways, or transferred therein from notebook)
• con: creates a question what is the status of jupyterlab-lsp which so far was treated as "unofficial" extension

Splitting up jupyter-lsp (server) from jupyterlab-lsp (front) and moving only jupyter-lsp into jupyterlab org:

• pro: clarity on what is official and what is unofficial
• con: substantial maintenance and development overhead
• con: then it make more sense to move it to jupyter-server org, right?
• con: users may be confused on where to report issues

Here are the current repos that could be migrated:

• https://github.com/jupyter-lsp/jupyterlab-lsp - contains both jupyterlab-agnostic jupyter-lsp and "unofficial" jupyterlab-lsp extension which is easily confused with a subset of the code that was merged into jupyterlab v4
• https://github.com/jupyter-lsp/demo-r
• https://github.com/jupyter-lsp/demo-julia
• https://github.com/jupyter-lsp/demo-python
• https://github.com/jupyter-lsp/yaml-lsp
• https://github.com/jupyter-lsp/json-lsp

Of course we could only migrate the first repo and leave the gutted jupyter-lsp org as an unofficial org for jupyter-lsp-adjacent stuff (where you would find e.g. ipython-optimized language server extensions for python-lsp-server).

At this point it is probably better for me to distance myself from the decision and let EC decide.

[krassowski]

I guess this issue would be friendlier if Jupyter Security team approached jupyter-lsp directly first rather than starting with EC, especially given that a presence of a security vulnerability is now being discussed in public although it was not reported to maintainers, and I still don't know if I should prepare for a sleepless night because of a potential a critical issue which as stated affects existing deployments ;)

[krassowski]

I opened jupyter/security#73 to consider a different solution to the underlying problem.

[Carreau]

I guess this issue would be friendlier if Jupyter Security team approached jupyter-lsp directly first rather than starting with EC, especially given that a presence of a security vulnerability is now being discussed in public although it was not reported to maintainers, and I still don't know if I should prepare for a sleepless night because of a potential a critical issue which as stated affects existing deployments ;)

During the EC meeting someone was present and pointed out that some discussion of having lsp into jupyterLab were already discussed and took a long time, likely hence the reason for the separate org.

Hence our decision to directly open an issue at a higher level.

I don't think the discussion that there is a security vulnerability report is a problem, It does not give attacker any information, nor say if the report is valid. Which I don't know as I have not read it.

[krassowski]

Thanks for the extra context and thank you for taking care of everything security!

[ellisonbg]

Thanks for looking at this @krassowski and @Carreau

There are two routes for incorporating jupyter lsp:

• We we want to pull it into an existing subproject, that subproject Council can vote to do that (for example, the new jupyter-frontend subproject).
• If we want to pull it into Jupyter as a new official subproject, with its own Council, that would go through the Software Steering Council and Executive Council. The process for that is a bit out of date and the Software Steering Council is working on updating it.

[jtpio]

We we want to pull it into an existing subproject, that subproject Council can vote to do that (for example, the new jupyter-frontend subproject).

That would make sense. Maybe https://github.com/jupyter-lsp/jupyterlab-lsp could be transferred to https://github.com/jupyterlab as is, under the Jupyter Frontends subproject?

[krassowski]

Yeah, I am happy for it to happen if someone is going to champion it (and help with the associated reconfigurations).

[jtpio]

Maybe we could have an issue on the Jupyter Frontends team compass repo to discuss this?

[krassowski]

Yes, I reopened jupyterlab/frontends-team-compass#67.

@ellisonbg regarding a conversation ( jupyterlab/frontends-team-compass#247 (comment)) about AWS team being constrained to contribute to repositories in jupyterlab oganization, did having jupyter-lsp as a separate organization prevent the team from contributing to jupyter-lsp or was this never considered? I am aware that some AWS products make use of jupyterlab-lsp (1, 2).

There are two routes for incorporating jupyter lsp:

I presume this implies that the EC's stance is then that jupyter-lsp is not currently an official part of Project Jupyter governance?

• We we want to pull it into an existing subproject, that subproject Council can vote to do that (for example, the new jupyter-frontend subproject).
• If we want to pull it into Jupyter as a new official subproject, with its own Council, that would go through the Software Steering Council and Executive Council. The process for that is a bit out of date and the Software Steering Council is working on updating it.

If you don't change the process too much I already have a checklist filled for this option, ready to go since 2020, here: jupyterlab/frontends-team-compass#67 (comment). But seriously, think I would lean that it makes more sense to make it a part of Jupyter Frontends subproject than create a new thing. The overhead of holding an election for representative does not seem worth it, and it would be odd for this rather a minor feature to have the same voting power as both Jupyter Notebook and JupyterLab combined. And the debugger does not have its own subproject either ;).

[jtpio]

think I would lean that it makes more sense to make it a part of Jupyter Frontends subproject than create a new thing. The overhead of holding an election for representative does not seem worth it, and it would be odd for this rather a minor feature to have the same voting power as both Jupyter Notebook and JupyterLab combined. And the debugger does not have its own subproject either

Thanks for reopening jupyterlab/frontends-team-compass#67 👍

Yes it would make sense to make it part of the Jupyter Frontends, and eventually retire the jupyter-lsp organization.

[krassowski]

So apparently Jupyter is now using Enterprise as per jupyter/enhancement-proposals#122 (comment). I don't think it changes much, except reducing the number of arguments against having a separate jupyter-lsp org.

[jasongrout]

So apparently Jupyter is now using Enterprise as per jupyter/enhancement-proposals#122 (comment). I don't think it changes much, except reducing the number of arguments against having a separate jupyter-lsp org.

See jupyter/governance#219 for more info about the enterprise org.

[krassowski]

I accepted the invite to the Jupyter Enterprise Org for jupyter-lsp based on the invite as per jupyter/governance#221 (comment)

[jasongrout]

Thanks @krassowski. Based on the messages in the past few weeks about jupyter-lsp being adopted under the Project Jupyter umbrella, can we have a jupyter frontend subproject council vote about adopting the jupyter-lsp org under the frontend subproject before the final approval from our end to join the jupyter enterprise org?

[jasongrout]

Ping @krassowski and @jtpio (the frontends SSC rep) for their thoughts. What do you think about having a frontends subproject council vote about adopting the jupyter-lsp github org and repos?

[jtpio]

Yes that makes sense.

We can probably comment on jupyterlab/frontends-team-compass#67 as mentioned above, and open a new issue on the team compass for the vote.

[Carreau]

Bumping this; -lsp is still not an official Jupyter org but have the Jupyter-logo and name; it looks like it is officially par of jupyter but not under the /enterprise ?

Also the PyPI package is not own by Jupyter.

[krassowski]

Bumping this; -lsp is still not an official Jupyter org but have the Jupyter-logo and name; it looks like it is officially par of jupyter but not under the /enterprise ?

I now removed the Jupyter logo and removed [email protected] from the account contact details. There is many packages which have jupyter- prefix on PyPI, so I think renaming is not required.

I invited a co-developer of the jupyter-lsp to the Jupyter frontends council because I wanted them to have a say. I waited with opening a vote before for this reason. My understanding (based on my private conversation with them) is that they politely declined the invitation as they do not want to be involved in governance.

It was important to me given how the governance structure is set up in Jupyter, but maybe it is not that important because in practice members of councils do not have a vote on governance decisions.

I am happy for anyone to open a voting PR for the frontends council to vote on adopting the LSP org for governance purposes. Once my workload reduces I can do that myself, unless better ideas are raised in the meantime.

---

The entire discussion about "we need to have this under Jupyter frontends in order to invite you to enterprise" seems a bit moot to me, frankly there was a JEP which said:

We would like to propose its incorporation as an official sub-project of Project Jupyter

• Pre-proposal: incorporation of jupyter(lab)-lsp jupyter/enhancement-proposals#67
• Language server protocol (LSP) jupyter/enhancement-proposals#72

And it was approved. I know this was a previous governance and since then a different structure of subprojects emerged, but at the time I invited EC members to join the org and tried to coordinate the package rights and stuff but it seems this was not followed through (some invites were ignored, but one past EC who since left Jupyter did accept and was an admin).

I previously also said that LSP does not really warrant a separate governance representation, but I also feel like dumping everything into "Frontends" umbrella is creating an opposite problem - individual contributors to projects under "Frontends" have dis-proportionally small influence on governance decisions due to dilution of the one vote that this subproject has.

[Carreau]

I'm personally happy with whatever as long as it's clear; I was diving into other stuff to make sur the PyPI and GitHub structure where correctly represented and writing tools to work across orgs; m

I'm also happy to invite Jupyter-LSP to enterprise if it helps. Even if part of enterprise, I think we can make this fact private; and at least it gives visibility on all the org – but I can see the confusion it can brings.

For PYPI, I think you are the only maintainer listed of lsp, so you may want to have backups.

[jasongrout]

@krassowski - thanks for reminding us about jupyter/enhancement-proposals#72.

I am happy for anyone to open a voting PR for the frontends council to vote on adopting the LSP org for governance purposes. Once my workload reduces I can do that myself, unless better ideas are raised in the meantime.

+1 - I think it makes a lot of sense these days to have jupyter-lsp in the jupyter frontends subproject.