Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Tool Shed 2.0 #15639

Merged
merged 73 commits into from
Oct 3, 2023
Merged

Tool Shed 2.0 #15639

merged 73 commits into from
Oct 3, 2023

Conversation

jmchilton
Copy link
Member

@jmchilton jmchilton commented Feb 23, 2023

Goals

The goal here is to introduce a new tool shed that breaks nothing but cuts all sorts of ties to old technologies and old concepts that we’re no longer interested in supporting.

  • Not a single line of mako, jQuery, or untyped JavaScript is used in the modern tool shed interface. The frontend is built on Typescript, Vue 3.2, Vite, Pinia, and Quasar.
  • Not a single old controller (web or API) is needed - the entire API is FastAPI based.
  • No Twill is needed to run the remaining and new test cases - everything can (and in some cases must) be run in Playwright.
  • A GraphQL engine and client is used to power a lot of the frontend and backend.
  • Planemo and Ephemeris are heavily featured in UX for the new frontend - the best practice is now supported more strongly and other paths that are not longer used are no longer supported.
  • The new tool shed implements the GA4GH TRS API (to some degree)

Technology Choices

At its core, these technologies mirror the directions Galaxy has moved over the last decade but in some cases there is small divergence. The long term goal of stripping the tool shed out of the Galaxy code base remains firmly intact. The tool shed now has its own client build process and such and doesn’t depend on any shared mako or Javascript glue with Galaxy client - and I hope this allows for that freedom to make some small different choices at the top of the client stack. In particular, the tool shed is very small (especially now) compared Galaxy so the freedom and flexibility and configurability of say Bootstrap and WebPack that are important for large projects with significant legacy features are probably not as important as the rapid prototyping and more on the rails experience (both visually and in terms of developer workflow) of say Quasar, Material, and Vite.

GraphQL remains an interesting experiment to my mind. I’ve done the hard work here of figuring out how to integrate it with a Galaxy-like app (both in terms of the backend and frontend). I don’t think it should be thrown out and I think it should remain in the tool shed - I’m somewhat more ambivalent about whether I’d use it in Galaxy or if I would keep in a second cleanroom rewrite of the Tool Shed. What is here is a good and successful experiment though I think and remains valuable data.

A Non-exhaustive List of Things I’ve Removed from the Frontend

  • Workflows
  • Datatypes
  • Repository Dependencies
  • Tool Dependencies
  • Visual Clues about repository “type” (there is only one best practice type)
  • Upload to the repository (everyone should be using Planemo)
  • Navigating repository contents (these are firmly packages - there is zero indication anymore that these are development repositories - we’re not competing against Github)
  • Pagination (everything is infinite scroll)

Developing the Tool Shed

The TypeScript GraphQL development tools require a GraphQL server to query for typing details and such. So the backend of the tool shed must be running before the frontend development tools can be launched (at least the GraphQL ones).

Backend Runtime

The backend (along with a GraphQL console) can be started with the following command:

TOOL_SHED_CONFIG_OVERRIDE_BOOTSTRAP_ADMIN_API_KEY=tsadminkey TOOL_SHED_CONFIG_CONFIG_HG_FOR_DEV=1 TOOL_SHED_VITE_PORT=4040 TOOL_SHED_API_VERSION=v2 ./run_tool_shed.sh

If that API key is set as that, the following script will bootstrap the new tool shed against the main tool shed to have some things to play with.

. .venv/bin/activate; python scripts/bootstrap_test_shed.py

Frontend Runtime:

make dev (from the frontend directory) or yarn dev-all from that same directory will run a Vite development server for the frontend (kind of the equivalent of webpack watch) and it will monitor the frontend code for GraphQL queries and generated typed TypeScript artifacts against the the backend server.

The one thing that isn’t covered by that - is TypeScript fetcher stuff for the API. I’ve piggy-backed on the Galaxy Makefile task for this - so make update-client-api-schema from Galaxy’s Makefile will generate these.

Production:

Run make client from the frontend script to build artifacts needed to for deploying the frontend.

Frontend Linting:

yarn and Makefile targets for linting and running prettier are included. make lint will check both of those as well as type checking. "make format” will format the code.

How to test the changes?

(Select all options that apply)

  • I've included appropriate automated tests.
  • Instructions for manual testing are as follows:
    1. See above under developing the tool shed.

License

  • I agree to license these and all my past contributions to the core galaxy codebase under the MIT license.

@jmchilton jmchilton changed the title Tool shed 2 Tool Shed 2.0 Feb 23, 2023
jmchilton added a commit to jmchilton/galaxy that referenced this pull request May 12, 2023
This is a piece that is needed to create a schema package - which I would really like to have around for galaxyproject#15639 and to in general write higher-level API tests.
@jmchilton jmchilton force-pushed the tool_shed_2 branch 3 times, most recently from 235f27e to 82086d0 Compare August 31, 2023 14:59
Copy link
Member

@ElectronicBlueberry ElectronicBlueberry left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Following the instructions outlined in the PR description, I ran into two errors which prevented me from running the toolshed locally:

The frontend threw an error which I detailed in the review comment for [...]/frontend/package.json

The backend threw the following error, causing a worker to be stuck in a restart loop:

Traceback (most recent call last):
  File "/home/lailalos/Projects/galaxy/lib/tool_shed/webapp/buildapp.py", line 91, in app_pair
    app = tool_shed.webapp.app.UniverseApplication(global_conf=global_conf, **kwargs)
  File "/home/lailalos/Projects/galaxy/lib/tool_shed/webapp/app.py", line 56, in __init__
    self.config: Any = config.Configuration(**kwd)
  File "/home/lailalos/Projects/galaxy/lib/tool_shed/webapp/config.py", line 41, in __init__
    self._process_config(kwargs)
  File "/home/lailalos/Projects/galaxy/lib/tool_shed/webapp/config.py", line 136, in _process_config
    self.password_expiration_period = timedelta(days=int(self.password_expiration_period))
AttributeError: 'ToolShedAppConfiguration' object has no attribute 'password_expiration_period'

I tried creating a tool_shed.yml from the provided sample and setting password_expiration_period to 0, which did not fix the issue.

"scripts": {
"dev": "vite --port 4040 --strict-port",
"build": "vue-tsc --noEmit && vite build",
"graphql": "graphql-codegen --watch",
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Running this either directly, or through dev all, throws the following error:

Failed to load schema from http://localhost:9009/graphql/:
connect ECONNREFUSED ::1:9009
Error: connect ECONNREFUSED ::1:9009
at TCPConnectWrap.afterConnect [as oncomplete] (node:net:1494:16)
GraphQL Code Generator supports:
- ES Modules and CommonJS exports (export as default or named export "schema")
- Introspection JSON File
- URL of GraphQL endpoint
- Multiple files with type definitions (glob expression)
- String in config file
Try to use one of above options and run codegen again.

Steps from clean slate:

  • dependencies installed with yarn
  • ran make dev

@jdavcs jdavcs mentioned this pull request Sep 26, 2023
18 tasks
@jmchilton jmchilton force-pushed the tool_shed_2 branch 2 times, most recently from 6f991ad to 276b5d0 Compare September 26, 2023 17:31
@jmchilton
Copy link
Member Author

The toolshed backend error was caused by a messed up rebase and I think I've fixed it now. The backend serves as the GraphQL server the frontend needs to generate code... so that is why dev code doesn't work... it needs a GraphQL server to monitor for model changes. Maybe it is worth exploring if a vite server without the model watching for GraphQL is viable... it wouldn't be as full reactive to dev changes but it would be sufficient for all sorts of development.

self._repo_install(changeset="1")
version1 = tool_shed_install.ToolVersion()
version1.tool_id = "github.com/galaxyproject/example/test_tool/0.1"
self.app.install_model.context.add(version1)
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I think the context for backward compatibility? Given that context here is the session, do you want to switch to calling it session (or sa_session, etc.) for consistency? (i.e., self.app.install_model.session)

Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I think that comment meant context.current was the backward compatible thing. I'm fine with install_model.session but it hasn't been my go to for tens of thousands of lines of code since that comment 😅. We should document a best practice here maybe.

Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I like session (or sa_sesssion). I think session is a slightly more accurate term for what the object is; even with the occasional presence of galaxy_session, I think context is more generic and may be harder to immediately recognize as the identifier for SQLAlchemy's session. But I'm +100 on documenting best practices (naming key objects like Session, as well as how to handle typical db access tasks, etc.)

if deleted and not trans.user_is_admin:
raise exceptions.AdminRequiredException("Only administrators can query deleted categories.")
for category in (
trans.sa_session.query(Category).filter(Category.table.c.deleted == deleted).order_by(Category.table.c.name)
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Suggested change
trans.sa_session.query(Category).filter(Category.table.c.deleted == deleted).order_by(Category.table.c.name)
trans.sa_session.scalars(select(Category).where(Category.deleted == deleted).order_by(Category.name))

Goals:

- Replace direct database access with pydantic typed API requests.
- Replace Twill with playwright.
- Eliminate access to the Galaxy UI in the tests - the tests should assume API access from the installing party.

pytest-playwright requires small bits of galaxyproject#13909.

This is a worthy project that could let us remove a bunch of deprecated stuff from the Galaxy admin controllers and a bunch of mako stuff that is unused and could be used to test PRs like galaxyproject#14609 (which prompted me to do this) but I'm anxious about growing emotionally attached to code I want to remove and I'm worried about losing track of which helpers are required for Planemo/Emphemeris/Galaxy and which helpers are just being used to test the tool shed.
app.model.Repository.table.c.user_id == app.model.User.table.c.id,
)
]
repository = app.model.context.query(app.model.Repository).filter(*clause_list).first()
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Suggested change
repository = app.model.context.query(app.model.Repository).filter(*clause_list).first()
repository = app.model.context.scalars(select(app.model.Repository).where(*clause_list).limit(1)).first()

Comment on lines +207 to +210
app.model.Repository.table.c.deprecated == false(),
app.model.Repository.table.c.deleted == false(),
app.model.Repository.table.c.name == name,
app.model.User.table.c.username == owner,
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Suggested change
app.model.Repository.table.c.deprecated == false(),
app.model.Repository.table.c.deleted == false(),
app.model.Repository.table.c.name == name,
app.model.User.table.c.username == owner,
app.model.Repository.deprecated == false(),
app.model.Repository.deleted == false(),
app.model.Repository.name == name,
app.model.User.username == owner,

app.model.Repository.table.c.deleted == false(),
app.model.Repository.table.c.name == name,
app.model.User.table.c.username == owner,
app.model.Repository.table.c.user_id == app.model.User.table.c.id,
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Suggested change
app.model.Repository.table.c.user_id == app.model.User.table.c.id,
app.model.Repository.user_id == app.model.User.id,

@jdavcs
Copy link
Member

jdavcs commented Sep 26, 2023

I'll wait to go over the SQLAlchemy syntax until you move it out of review.

@jmchilton
Copy link
Member Author

I'll wait to go over the SQLAlchemy syntax until you move it out of review.

Such a chicken and egg thing here... they will be further divergent from the other code that you also wanted to update that brought you here. I'll collect your suggestion into a commit though! I appreciate the feedback. A fun winter break project becoming the whole project's chore is just such a John thing 😆 - looking at you planemo and pulsar.

@jdavcs
Copy link
Member

jdavcs commented Sep 26, 2023

I'll wait to go over the SQLAlchemy syntax until you move it out of review.

Such a chicken and egg thing here... they will be further divergent from the other code that you also wanted to update that brought you here. I'll collect your suggestion into a commit though! I appreciate the feedback. A fun winter break project becoming the whole project's chore is just such a John thing 😆 - looking at you planemo and pulsar.

LOL :) Only reason: I thought you were force-pushing stuff - I was afraid my suggestions would be immediately outdated. I'll go over it today.

@jdavcs
Copy link
Member

jdavcs commented Sep 26, 2023

It'll be easier to fix all syntax issues in a follow-up PR: too much of the same cut and paste, and my browser is literally stalling (I suspect it's the 23K-lines editable change set it's trying to display). It's all under lib/tool_shed and test/ - I won't touch those until this is merged (I suppose we'll be merging this soon).

self._repo_install(changeset="1")
version1 = tool_shed_install.ToolVersion()
version1.tool_id = "github.com/galaxyproject/example/test_tool/0.1"
self.app.install_model.context.add(version1)
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I like session (or sa_sesssion). I think session is a slightly more accurate term for what the object is; even with the occasional presence of galaxy_session, I think context is more generic and may be harder to immediately recognize as the identifier for SQLAlchemy's session. But I'm +100 on documenting best practices (naming key objects like Session, as well as how to handle typical db access tasks, etc.)

Comment on lines +227 to +231
app.model.Repository.table.c.deprecated == false(),
app.model.Repository.table.c.deleted == false(),
app.model.Repository.table.c.name == name,
app.model.User.table.c.username == owner,
app.model.Repository.table.c.user_id == app.model.User.table.c.id,
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Same as above: app.model.Foo.table.c.bar >> app.model.Foo.bar

app.model.Repository.table.c.user_id == app.model.User.table.c.id,
)
]
repository = app.model.context.current.sa_session.query(app.model.Repository).filter(*clause_list).first()
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Suggested change
repository = app.model.context.current.sa_session.query(app.model.Repository).filter(*clause_list).first()
repository = app.model.context.current.sa_session.scalars(app.model.Repository).where(*clause_list).limit(1)).first()

Comment on lines +278 to +280
app.model.Repository.table.c.deprecated == false(),
app.model.Repository.table.c.deleted == deleted,
)
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

same as above: drop table.c.

Comment on lines +285 to +287
app.model.User.table.c.username == owner,
app.model.Repository.table.c.user_id == app.model.User.table.c.id,
)
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

same: table.c.

Comment on lines +167 to +176
repository = (
self.sa_session.query(self.app.model.Repository)
.filter(
and_(
self.app.model.Repository.table.c.name == name,
self.app.model.Repository.table.c.user_id == user.id,
)
)
.one()
)
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Suggested change
repository = (
self.sa_session.query(self.app.model.Repository)
.filter(
and_(
self.app.model.Repository.table.c.name == name,
self.app.model.Repository.table.c.user_id == user.id,
)
)
.one()
)
repository = (
self.sa_session.execute(select(self.app.model.Repository)
.where(
and_(
self.app.model.Repository.name == name,
self.app.model.Repository.user_id == user.id,
)
)
).scalar_one()
)

formatting will be off here (my browser is barely handling this inline editing..)

Comment on lines +57 to +67
session.query(galaxy.model.tool_shed_install.ToolShedRepository)
.filter(
and_(
galaxy.model.tool_shed_install.ToolShedRepository.table.c.deleted == false(),
galaxy.model.tool_shed_install.ToolShedRepository.table.c.uninstalled == false(),
galaxy.model.tool_shed_install.ToolShedRepository.table.c.status
== galaxy.model.tool_shed_install.ToolShedRepository.installation_status.INSTALLED,
)
.all()
)
else:
return install_session().query(galaxy.model.tool_shed_install.ToolShedRepository).all()
.all()
)
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

stmt = select(ToolShedRepository).where(and_(ToolShedRepository.deleted == false(), ToolShedRepository.uninstalled == false(), ToolShedRepository.stats == ToolShedRepository.installation_status.INSTALLED)
return session.scalars(stmt).all()

Also, no need to call list(): all() produces a list.

def get_installed_repository_by_name_owner(repository_name, owner, return_multiple=False, session=None):
if session is None:
session = install_session()
query = session.query(galaxy.model.tool_shed_install.ToolShedRepository).filter(
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

convert to select construct? Then, in the return clause, just session.scalars(stmt).all() for multiple and
session.scalars(stmt.limit(1)).first() for one (note the limit clause).

session = install_session()
query = session.query(galaxy.model.tool_shed_install.ToolShedRepository).filter(
and_(
galaxy.model.tool_shed_install.ToolShedRepository.table.c.name == repository_name,
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

table.c. (same as above)

) -> Optional[Dict[str, Any]]:
clause_list = []
if name is not None:
clause_list.append(galaxy_model.ToolShedRepository.table.c.name == name)
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

.table.c

@jmchilton jmchilton marked this pull request as ready for review October 3, 2023 19:36
@github-actions github-actions bot added this to the 23.2 milestone Oct 3, 2023
@mvdbeek mvdbeek merged commit cc6f437 into galaxyproject:dev Oct 3, 2023
37 of 42 checks passed
nsoranzo added a commit to nsoranzo/bioblend that referenced this pull request Oct 18, 2023
nsoranzo added a commit to nsoranzo/bioblend that referenced this pull request Oct 18, 2023
nsoranzo added a commit to nsoranzo/bioblend that referenced this pull request Oct 18, 2023
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
None yet
Development

Successfully merging this pull request may close these issues.

4 participants