Dynamic options: add data table filter #12941

bernt-matthias · 2021-11-17T20:22:45Z

In a recent gitter discussion (https://gitter.im/galaxy-iuc/iuc?at=6190aeff8c019f0d0b93414b) it was asked for a way to have a select list containing only elements that are not yet in a data table.. Et voila .. there is a data table filter.

The test tool shows how it might be used:

the options are dynamically create from a file (from_file also had no test so far)
the filter removes all options that are already in the data table

For the first part I needed to apply a little hack: from_file seems to assume that the file is in tool-data in the Galaxy root. It would be much nicer if the file would be loaded from the dir (or a tool-data dir in this dir) containing the tool. Otherwise we would not meet the use case of the gitter discussion.

I did not think about other possible use cases (e.g. intersections of data tables) and possibly needed attributes...

Ping @mvdbeek @pvanheus

The PR also adds more linting to filters.

How to test the changes?

(Select all options that apply)

I've included appropriate automated tests.
This is a refactoring of components with existing test coverage.
Instructions for manual testing are as follows:
1. [add testing steps and prerequisites here if you didn't write automated tests covering all your changes]

License

I agree to license these contributions under Galaxy's current license.
I agree to allow the Galaxy committers to license these and all my past contributions to the core galaxy codebase under the MIT license. If this condition is an issue, uncheck and just let us know why with an e-mail to [email protected].

mvdbeek

This is really cool, but from_file should not be able to reference files outside of the tool dir.

lib/galaxy/tools/parameters/dynamic_options.py

mvdbeek · 2021-11-18T10:58:27Z

test/functional/tools/filter_data_table.xml

+    <inputs>
+        <param name="dynamic_select" type="select">
+            <!-- initialize options from file (contains all entries that could be added to the data table) -->
+            <options from_file="../test/functional/tool-data/test_file.tsv">


That's a security issue and should not work.

Do you have an idea how to specify another dir during tests, eg test/functional/tool-data/?

I guess we can check if os.path.abspath(path) startswith the tool-data dir... could do this in another PR.

It's fine to do this in another PR, but we shouldn't merge this before we sorted that out.

Also there is a safe_contains function in galaxy.util you can use

ae4c401 contains a possible solution for:

Do you have an idea how to specify another dir during tests, eg test/functional/tool-data/?

Any comments?

lib/galaxy/tools/parameters/dynamic_options.py

lib/tool_shed/test/base/twilltestcase.py

mvdbeek · 2022-01-26T11:50:55Z

I'm bumping this to 22.05, we still need to figure out what's up with #12941 (comment)

fix missing import

…tthias/galaxy into topic/data-table-filter

from_file does not work (relative paths are not possible .. and in the tests we do not know the abs path)

lib/galaxy/tools/parameters/dynamic_options.py

…tthias/galaxy into topic/data-table-filter

`Exception: Cannot specify outputs in a test expecting failure.`

lib/galaxy_test/driver/driver_util.py

test/functional/tools/filter_data_table.xml

mvdbeek · 2024-11-15T16:49:35Z

lib/galaxy/tool_util/linters/inputs.py

@@ -490,6 +490,190 @@ def lint(cls, tool_source: "ToolSource", lint_ctx: "LintContext"):
                        )


+FILTER_REQUIRED_ATTRIBUTES = {


@jmchilton is this going to conflict with your modeling work ?

mvdbeek

Could you split the linting from the table filter work ?

bernt-matthias · 2024-11-15T16:55:15Z

Could you split the linting from the table filter work ?

Separate PR?

mvdbeek · 2024-11-15T16:57:12Z

yes

Co-authored-by: Marius van den Beek <[email protected]>

bernt-matthias · 2024-11-21T16:50:16Z

Could you split the linting from the table filter work ?

I have a hard time to find the motivation to invest any more time here. It seems that this will just create another PR that will not be merged for years.

lib/galaxy/tool_util/linters/inputs.py

This reverts commit 77434f2.

github-actions · 2024-11-22T16:54:39Z

This PR was merged without a "kind/" label, please correct.

github-actions bot added area/testing area/tool-framework labels Nov 17, 2021

github-actions bot added this to the 22.01 milestone Nov 17, 2021

bernt-matthias force-pushed the topic/data-table-filter branch from f99c729 to 725f3c2 Compare November 17, 2021 21:19

bernt-matthias changed the title ~~add data table filter~~ dynamic options: add data table filter Nov 17, 2021

bernt-matthias force-pushed the topic/data-table-filter branch from 725f3c2 to d608ba1 Compare November 18, 2021 09:05

mvdbeek reviewed Nov 18, 2021

View reviewed changes

mvdbeek reviewed Dec 21, 2021

View reviewed changes

lib/galaxy/tools/parameters/dynamic_options.py Outdated Show resolved Hide resolved

mvdbeek reviewed Dec 21, 2021

View reviewed changes

lib/tool_shed/test/base/twilltestcase.py Outdated Show resolved Hide resolved

bernt-matthias force-pushed the topic/data-table-filter branch from 8065a08 to 2a47e13 Compare December 21, 2021 10:46

mvdbeek modified the milestones: 22.01, 22.05 Jan 26, 2022

mvdbeek modified the milestones: 22.05, 22.09 May 25, 2022

dannon modified the milestones: 23.0, 23.1 Dec 8, 2022

bernt-matthias added 6 commits June 6, 2023 10:12

add data table filter

e66f4bb

add more linting for filters

26043cd

use test/functional/tool-data/ as tool-data dir in tests

2470e46

forbid from_file to reference files outside of tool-data

d7c99a0

fix linter and syntax errors

64831e0

add link to tool-data/shared in test tool-data

5b7df0f

fix missing import

bernt-matthias force-pushed the topic/data-table-filter branch from 2a47e13 to bce9300 Compare June 6, 2023 08:28

bernt-matthias added 4 commits June 6, 2023 10:29

remove debug messages

bce9300

fix rebase mistake

95bbd88

one more rebase error

8633181

add context to new linter messages

d199a27

mvdbeek removed this from the 23.1 milestone Jun 21, 2023

mvdbeek marked this pull request as draft June 21, 2023 18:30

bernt-matthias added 3 commits November 13, 2024 21:55

revert some changes

001c1dd

Merge branch 'topic/data-table-filter' of https://github.com/bernt-ma…

ba5c2cf

…tthias/galaxy into topic/data-table-filter

try to adapt test

b926ca0

from_file does not work (relative paths are not possible .. and in the tests we do not know the abs path)

bernt-matthias commented Nov 13, 2024

View reviewed changes

lib/galaxy/tools/parameters/dynamic_options.py Outdated Show resolved Hide resolved

bernt-matthias added 6 commits November 13, 2024 22:19

Update lib/galaxy/tools/parameters/dynamic_options.py

e851a45

fix test

174f244

Merge branch 'topic/data-table-filter' of https://github.com/bernt-ma…

f900689

…tthias/galaxy into topic/data-table-filter

remove output assertion for failing test

157d074

`Exception: Cannot specify outputs in a test expecting failure.`

fix test

2676039

remove unused test data

228e62d

bernt-matthias commented Nov 15, 2024

View reviewed changes

lib/galaxy_test/driver/driver_util.py Outdated Show resolved Hide resolved

bernt-matthias added 2 commits November 15, 2024 09:43

Update lib/galaxy_test/driver/driver_util.py

158b29c

readd linebreak

6e17d94

mvdbeek reviewed Nov 15, 2024

View reviewed changes

test/functional/tools/filter_data_table.xml Outdated Show resolved Hide resolved

mvdbeek reviewed Nov 15, 2024

View reviewed changes

test/functional/tools/filter_data_table.xml Outdated Show resolved Hide resolved

mvdbeek reviewed Nov 15, 2024

View reviewed changes

jdavcs modified the milestones: 24.2, 25.0 Nov 20, 2024

bernt-matthias and others added 2 commits November 20, 2024 10:46

access column by name in test

a09ece8

Update test/functional/tools/filter_data_table.xml

f2100cc

Co-authored-by: Marius van den Beek <[email protected]>

bernt-matthias commented Nov 22, 2024

View reviewed changes

lib/galaxy/tool_util/linters/inputs.py Show resolved Hide resolved

lib/galaxy/tool_util/linters/inputs.py Show resolved Hide resolved

bernt-matthias added 2 commits November 22, 2024 10:35

Apply suggestions from code review

77434f2

Revert "Apply suggestions from code review"

db1b370

This reverts commit 77434f2.

mvdbeek merged commit 363ab53 into galaxyproject:dev Nov 22, 2024
52 of 54 checks passed

nsoranzo added the kind/enhancement label Nov 22, 2024

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Dynamic options: add data table filter #12941

Dynamic options: add data table filter #12941

bernt-matthias commented Nov 17, 2021 •

edited

Loading

mvdbeek left a comment

mvdbeek Nov 18, 2021

bernt-matthias Nov 18, 2021

mvdbeek Nov 18, 2021

mvdbeek Nov 18, 2021

bernt-matthias Nov 18, 2021 •

edited

Loading

mvdbeek commented Jan 26, 2022

mvdbeek Nov 15, 2024

mvdbeek left a comment

bernt-matthias commented Nov 15, 2024

mvdbeek commented Nov 15, 2024

bernt-matthias commented Nov 21, 2024

github-actions bot commented Nov 22, 2024

		@@ -490,6 +490,190 @@ def lint(cls, tool_source: "ToolSource", lint_ctx: "LintContext"):
		)


		FILTER_REQUIRED_ATTRIBUTES = {

Dynamic options: add data table filter #12941

Dynamic options: add data table filter #12941

Conversation

bernt-matthias commented Nov 17, 2021 • edited Loading

How to test the changes?

License

mvdbeek left a comment

Choose a reason for hiding this comment

mvdbeek Nov 18, 2021

Choose a reason for hiding this comment

bernt-matthias Nov 18, 2021

Choose a reason for hiding this comment

mvdbeek Nov 18, 2021

Choose a reason for hiding this comment

mvdbeek Nov 18, 2021

Choose a reason for hiding this comment

bernt-matthias Nov 18, 2021 • edited Loading

Choose a reason for hiding this comment

mvdbeek commented Jan 26, 2022

mvdbeek Nov 15, 2024

Choose a reason for hiding this comment

mvdbeek left a comment

Choose a reason for hiding this comment

bernt-matthias commented Nov 15, 2024

mvdbeek commented Nov 15, 2024

bernt-matthias commented Nov 21, 2024

github-actions bot commented Nov 22, 2024

bernt-matthias commented Nov 17, 2021 •

edited

Loading

bernt-matthias Nov 18, 2021 •

edited

Loading