plugins/plug-in architecture #1353
Comments
|
The packaging docs for Python on plugin support are an interesting read. So far I like the "naming convention" approach the best. |
|
Using entrypoints for 3rd party IO libraries in xarray: https://twitter.com/xarray_dev/status/1369134763982815234 |
|
Yes to this! I've been thinking about scanpy (scverse/scanpy#271) + sourmash extensions for a bit but haven't gotten around to it yet. Here's an example of one: https://github.com/icbi-lab/scirpy |
|
Also found this for the scanpy ecosystem: https://scanpy.readthedocs.io/en/stable/ecosystem.html |
|
some thoughts on possible plug-in functionality -
|
|
Apparently
So: https://docs.python.org/3/library/importlib.metadata.html ...which, ultimately, after following many small twisty paths, leads to: https://setuptools.pypa.io/en/latest/userguide/entry_point.html |
|
I think it would be easy and maybe useful to add new taxonomy loading via plugins 🤔 . New taxonomies themselves would be trickier (e.g. LINS) and it's not 100% clear we need more than NCBI, GTDB, and LINS. |
|
Other plugin ideas:
|
…ts new signature saving & loading mechanisms (#2428) Implement support for `load_from` and `save_to` plugins via `importlib.metadata` entry points. This supports a few of the plugins suggested in #1353 I am nominating this as an experimental feature that is not under semantic versioning/not public yet. Documentation page [here, in dev_plugins.html](https://sourmash--2428.org.readthedocs.build/en/2428/dev_plugins.html). A template repo for new plugins is at https://github.com/sourmash-bio/sourmash_plugin_template. ## Implementation/this PR This PR refactors the `_load_database` loading and `SaveSignaturesToLocation` saving code to build a prioritized list of functions to try in order, and then adds hooks in via the new `sourmash.plugins` module that insert additional loading/saving functions into that list. This PR also moves the current saving/loading functions out of `sourmash.sourmash_args` into the `sourmash.save_load` submodule, and simplifies the code a bit. ## Example plugins: - read JSON sigs and manifests from URLs: https://github.com/sourmash-bio/sourmash_plugin_load_urls - read and write signatures in Apache Avro: https://github.com/sourmash-bio/sourmash_plugin_avro - use extension `.avrosig` to write. Specific TODOs: - [x] provide a minimal "getting started" template repo - [x] add tests for multiple plugins & priorities - [ ] maybe try writing CSV export/import as a plugin? #1098 For later: - think about other kinds of plugins - new CLI entry points, picklist classes, tax loading, tax structure, ??. - work on getting avro support into rust over in luizirber/2021-02-11-sourmash-binary-format#1
|
I've been wondering about Python plugins that build on the Rust library. Will we have multiple versions of sourmash-rs installed and present? Is that a problem? (probably not a problem, but maybe inefficient) In particular, it might be interesting to try out some of the stuff in https://github.com/ctb/2022-pymagsearch as a CLI plugin. |
|
It isn't strictly an alternative (pluggy also uses entrypoints to discover plugins), I look at it more as a "framework to use when making your application pluggable" or some such. It lets you define hooks, register implementations, etc. |
|
more things to do, coming from #2438:
|
Add command-line plug-in interface, per #1353. This PR builds on #2428 to provide support for adding new sourmash subcommands, under `sourmash scripts <cmd>`. The CLI plugin interface looks like this on the client side - ``` class cmd_somewhat: command = 'somewhat' description = "does a thing" def __init__(self, subparser): super().__init__(p) # add arguments etc here debug_literal('RUNNING cmd_somewhat.__init__') subparser.add_argument('foo', type=int) def main(self, args): super().main(args) # what we actually run. print('RUNNING cmd', self, args) print('XYZ somewhat', args.foo) print(f'2*{args.foo} is {2*args.foo}') ``` Adding this plugin enables: ``` % sourmash scripts -h == This is sourmash version 4.6.1. == == Please cite Brown and Irber (2016), doi:10.21105/joss.00027. == usage: scripts [-h] optional arguments: -h, --help show this help message and exit extension commands: sourmash scripts somewhat --help - do a thing ``` and ``` % sourmash scripts somewhat 5 == This is sourmash version 4.6.1. == == Please cite Brown and Irber (2016), doi:10.21105/joss.00027. == ... 2*5 is 10 ``` which is prety cool 🎉 ## Alternatives to `script` could be `sourmash ext` or `sourmash plugin`?
|
taxonomy plugin for greengenes using code from over in #2538 |
|
save/load plugins were added in #2428
https://github.com/ctb/sourmash-mixers is my attempt to gather "sourmash team" plugins under a common metapackage for single-command install. the future is bright. 😎 moving remaining ideas to:
|
@luizirber says that the answer is yes and I believe him :) |
how might we be thinking about better supporting plugins as a way for people to extend sourmash?
ref https://twitter.com/betatim/status/1355902709237473281
UPDATE July 2023:
save/load plugins were added in #2428
command-line plugins were added in #2438, under
sourmash scripts ...sourmash info -vdisplays available plugins.https://github.com/ctb/sourmash-mixers is my attempt to gather "sourmash team" plugins under a common metapackage for single-command install.
the future is bright. 😎
moving remaining ideas to:
The text was updated successfully, but these errors were encountered: