Add rank metric #15 #42

jonasscheid · 2021-10-26T10:12:43Z

The Rank metric is used by multiple prediction tools, including the NetMHC family. This PR aims to provide a solution for issue #15

To implement another score metric fundamental changes to the EpitopePredictionResult class had to be made. For a better overview the new structure is sketched in row 63 in Results.py.
For every external tool supporting the rank metric the output parser parse_external_result has been adjusted.
For every tool not supporting the rank metric the predict method has been adjusted to maintain the overall hierarchy
Overall enums replaced the hardcoded positional accession in prediction output parsers
The (parsed) output of all predictors (epytope/epytope/EpitopePrediction/*) is now condensed into a nested dictionary with the following structure:

{'Allele1': {'Score': {'Pep1': Score1, 'Pep2': Score2,..}, 'Rank': {'Pep1': Rank1, 'Pep2': Rank2,..}, 'Allele2':...}

-> If no rank is supported by the respective tool, the hierarchy is maintained, just no rank information.

This nested dictionary is then packed into an EpitopePredictionResult using the new from_dict method and returns the desired structure.
These changes to the EpitopePredictionResult structure additionally caused to adjust OptiTope.py. OptiTope __init__ got a new boolean parameter rank, in order to optimize differentiate with respect to the given metric. OptiTope aims to minimize the rank or maximize the (prediction) score.

I hope I covered most of the changes.

Looking forward to your feedback!

jonasscheid · 2021-10-26T15:32:23Z

Hey @b-schubert
I encountered an issue in TestSpacerDesign.py caused after adding the new A-2601-9 matrix. I need to fix the underlying problem, I was just wondering why the tests went through when there was no A-2601-9 matrix for syfpeithi available. The test should have reported this problem right?
Thanks in advance!
Jonas

b-schubert · 2021-10-26T17:55:56Z

Hey @b-schubert I encountered an issue in TestSpacerDesign.py caused after adding the new A-2601-9 matrix. I need to fix the underlying problem, I was just wondering why the tests went through when there was no A-2601-9 matrix for syfpeithi available. The test should have reported this problem right? Thanks in advance! Jonas

The reason is the overall problem changed. Now it is considering the A-2601 matrix during optimization, before it was only considering the A02 matrix. The functions work fine, it finds now a new solution that is optimal when considering both A02 and A26.

jonasscheid · 2021-11-03T14:37:27Z

Thanks for the quick response! So because now the allele is there the output differs. I adjusted the test as follows

…problems in github actions can be solved

christopher-mohr

Thanks @jonasscheid ! So far only minor things, please check the syfpeithi matrices again, should the "wildcard" be there in all cases? I didn't mark it for all of them in case it was missing.

I will continue with the remaining files tomorrow.

.github/workflows/python-test-conda-external.yml

.github/workflows/python-test-conda.yml

epytope/Core/Result.py

epytope/Data/pssms/syfpeithi/mat/B_5802_10.py

epytope/EpitopeAssembly/EpitopeAssembly.py

epytope/EpitopePrediction/ANN.py

epytope/EpitopePrediction/External.py

christopher-mohr

Some more suggestions and things to change/discuss but overall it looks good to me. Thanks a lot @jonasscheid !

epytope/EpitopePrediction/PSSM.py

epytope/test/TestEpitopeAssembly.py

epytope/test/TestSpacerDesign.py

epytope/tutorials/PolymorphicEpitopePrediction.ipynb

epytope/tutorials/data/allele_probabilities_europe.csv

christopher-mohr · 2021-11-22T07:39:37Z

Hi @jonasscheid, thanks for the update! 👍 There are still a couple of unresolved conversations, could you please marke them as resolved if applicable or create issues for them and mark them afterwards as resolved .

christopher-mohr

Some minor things and the filtering functionality that might have to be adapted as we discussed. 👍

epytope/EpitopeSelection/OptiTope.py

epytope/test/TestSpacerDesign.py

epytope/tutorials/PolymorphicEpitopePrediction.ipynb

epytope/Core/Result.py

…ltiple lengths

christopher-mohr

Some comments on some of the recently changed files.

epytope/tutorials/CleavageAndTAPPrediction.ipynb

epytope/tutorials/HLATyping.ipynb

christopher-mohr

More comments on remaining changed files.

epytope/Core/Result.py

christopher-mohr · 2021-11-28T12:18:12Z

epytope/Core/Result.py

        """
        Filters a result data frame based on a specified expression consisting of a list of triple with
-        (method_name, comparator, threshold). The expression is applied to each row. If any of the columns fulfill
-        the criteria the row remains.
+        (method_name/scoretype_name, comparator, threshold) and a boolean if the scoretype is specified.


I was just wondering if it would be cleaner to have the first part of the triple only for method_name and the additional parameter scoretype with a default value Score.

I changed this method according to your suggestion. Please have a look again :)

This looks still the same to me, could you please doublecheck?

epytope/Core/Result.py

epytope/EpitopePrediction/External.py

christopher-mohr

Looks like a typo has been introduced in one of the HLA class II alleles.

epytope/Data/pssms/syfpeithi/mat/B_4001_9.py

epytope/Data/pssms/syfpeithi/mat/DRB1_1101_9.py

epytope/Data/pssms/syfpeithi/mat/DRB1_1501_9.py

epytope/Data/pssms/syfpeithi/mat/DRB1_0701_9.py

epytope/Data/pssms/syfpeithi/mat/DRB1_0101_9.py

epytope/Data/pssms/syfpeithi/mat/DRB1_0301_9.py

christopher-mohr

Looks good to me 👍

* Version bump 3.0.0rc2 * Fix master / main branch naming in GH action * Fix typos in README file * Add pypi GH action * Fix PyPI linting errors * Reduce version to 3.0.0rc1 * Add a changelog * Install changelog with package * Change PyPI CD trigger to published release * Add rank metric #15 (#42) * Push all changes made on fork * Set Setuptools version also for external yml * Fixed erroneous variable names in matrix files * deleted A_2601_9 matrix for now. Caused troubles * Add A2601_9 syf matrix for debugging * Fixed bug in test caused by addition of A*26:01 matrix * Change solver from cbc to glpk to investigate if macOS dependant env problems in github actions can be solved * Corrected after review * Adjust tutorials to new structure * Change filter_result as discussed * Adjusted filter method and tutorials according to #12 * Fixed a bug occuring for netMHCfamily tools when peptide input has multiple lengths * remove logging * Alter filter_result method as discussed * Fixed issues #38, #44 and #45 (#46) * Fixed issues #44 and #45 * Fix #48, include review suggestions * Improve/update documentation (#50) * Update CHANGELOG * Extend README * Change framework name in code comment * Remove logging warning * Change file ending in tutorial * Add docstrings, minor formatting * Update CHANGELOG version and setup.py * Update date Co-authored-by: Leon Kuchenbecker <[email protected]> Co-authored-by: Jonas Scheid <[email protected]>

* Version bump 3.0.0rc2 * Fix master / main branch naming in GH action * Fix typos in README file * Add pypi GH action * Fix PyPI linting errors * Reduce version to 3.0.0rc1 * Add a changelog * Install changelog with package * Change PyPI CD trigger to published release * Add rank metric #15 (#42) * Push all changes made on fork * Set Setuptools version also for external yml * Fixed erroneous variable names in matrix files * deleted A_2601_9 matrix for now. Caused troubles * Add A2601_9 syf matrix for debugging * Fixed bug in test caused by addition of A*26:01 matrix * Change solver from cbc to glpk to investigate if macOS dependant env problems in github actions can be solved * Corrected after review * Adjust tutorials to new structure * Change filter_result as discussed * Adjusted filter method and tutorials according to #12 * Fixed a bug occuring for netMHCfamily tools when peptide input has multiple lengths * remove logging * Alter filter_result method as discussed * Fixed issues #38, #44 and #45 (#46) * Fixed issues #44 and #45 * Fix #48, include review suggestions * Improve/update documentation (#50) * Update CHANGELOG * Extend README * Change framework name in code comment * Remove logging warning * Change file ending in tutorial * Add docstrings, minor formatting * Update CHANGELOG version and setup.py * Update date * Fix #52 (#53) * add check if transcript sequence available from BioMart, cleanup (#58) * Add interface for netMHCpan 4.1 (#59) * add interface for netmhcpan 4.1 * remove duplicate alleles from list * Update supportedAlleles of syfpeithi (#62) Co-authored-by: Christopher Mohr <[email protected]> * Fix protobuf version for tests, prepare docs for 3.1.0 release (#64) * Prepare docs for new release * minor changes/additions docs * check if fixing protobuf version resolves testing errors * check if changing github actions workflow resolves testing issue * allow lower versions of protobuf * Update epytope/doc/conf.py Co-authored-by: Gisela Gabernet <[email protected]> Co-authored-by: Christopher Mohr <[email protected]> Co-authored-by: Jonas Scheid <[email protected]> Co-authored-by: Gisela Gabernet <[email protected]> * Add netMHCIIpan 4.1 interface (#66) * add netmhciipan 4.1 interface * remove logging * remove __name method * update changelog * Update CHANGELOG.md Co-authored-by: Christopher Mohr <[email protected]> Co-authored-by: Christopher Mohr <[email protected]> Co-authored-by: Christopher Mohr <[email protected]> * minor doc improvements, cleanup setup.py * bump version Co-authored-by: Leon Kuchenbecker <[email protected]> Co-authored-by: Jonas Scheid <[email protected]> Co-authored-by: Jonas Scheid <[email protected]> Co-authored-by: Gisela Gabernet <[email protected]>

* Version bump 3.0.0rc2 * Fix master / main branch naming in GH action * Fix typos in README file * Add pypi GH action * Fix PyPI linting errors * Reduce version to 3.0.0rc1 * Add a changelog * Install changelog with package * Change PyPI CD trigger to published release * Add rank metric #15 (#42) * Push all changes made on fork * Set Setuptools version also for external yml * Fixed erroneous variable names in matrix files * deleted A_2601_9 matrix for now. Caused troubles * Add A2601_9 syf matrix for debugging * Fixed bug in test caused by addition of A*26:01 matrix * Change solver from cbc to glpk to investigate if macOS dependant env problems in github actions can be solved * Corrected after review * Adjust tutorials to new structure * Change filter_result as discussed * Adjusted filter method and tutorials according to #12 * Fixed a bug occuring for netMHCfamily tools when peptide input has multiple lengths * remove logging * Alter filter_result method as discussed * Fixed issues #38, #44 and #45 (#46) * Fixed issues #44 and #45 * Fix #48, include review suggestions * Improve/update documentation (#50) * Update CHANGELOG * Extend README * Change framework name in code comment * Remove logging warning * Change file ending in tutorial * Add docstrings, minor formatting * Update CHANGELOG version and setup.py * Update date * Fix #52 (#53) * add check if transcript sequence available from BioMart, cleanup (#58) * Add interface for netMHCpan 4.1 (#59) * add interface for netmhcpan 4.1 * remove duplicate alleles from list * Update supportedAlleles of syfpeithi (#62) Co-authored-by: Christopher Mohr <[email protected]> * Fix protobuf version for tests, prepare docs for 3.1.0 release (#64) * Prepare docs for new release * minor changes/additions docs * check if fixing protobuf version resolves testing errors * check if changing github actions workflow resolves testing issue * allow lower versions of protobuf * Update epytope/doc/conf.py Co-authored-by: Gisela Gabernet <[email protected]> Co-authored-by: Christopher Mohr <[email protected]> Co-authored-by: Jonas Scheid <[email protected]> Co-authored-by: Gisela Gabernet <[email protected]> * Add netMHCIIpan 4.1 interface (#66) * add netmhciipan 4.1 interface * remove logging * remove __name method * update changelog * Update CHANGELOG.md Co-authored-by: Christopher Mohr <[email protected]> Co-authored-by: Christopher Mohr <[email protected]> Co-authored-by: Christopher Mohr <[email protected]> * minor doc improvements, cleanup setup.py * Update MartsAdapter (#69) * Rewrite, extend, cleanup MartsAdapter, adapt tests * add requests and beautifulsoup4 dependency * prevent too long requests, avoid server request for each attribute * add gene to test object * fix enum ref * adapt MartsAdapter in other test * add function for getting gene names, add tests * change method name, add test * add lxml as dependency * workaround for pandas read_xml, remove dependency * add missing all() * fix test * add retry strategy for GET requests * Update epytope/IO/MartsAdapter.py Co-authored-by: Gisela Gabernet <[email protected]> * add default biomart url Co-authored-by: Gisela Gabernet <[email protected]> * Outsource supported alleles (#63) * Draft for outsourcing supported alleles * Further outsourcing of netmhc alleles * Finish outsourcing external alleles * Outsource alleles from pssm and ann predictors * Correct minor erroneous hla nomenclatures of smmpmbec * Change allele imports by importing frozensets * Add __allele_import_name to classes to increase readability * Refactor: convert_alleles is now classmethod in pssm * Incorporate feedback * Update __init__.py * Update uniprot adapter (#71) * remove HLAtyping and distance2self tests, update CHANGELOG * fix reading sequences in uniprot adapter * add test for uniprot adapter * remove HLAtyping and distance2self tests, update CHANGELOG (#70) * Fix netmhcii4.0 parser (#73) * fix netmhciipan4.0 issue * update changelog * Add function for peptides to check if created by variant (#74) * remove HLAtyping and distance2self tests, update CHANGELOG * add Peptide functon to determine if peptide originates from a variant * fix peptide call, update CHANGELOG * Improve function to check peptide origin (#75) * remove HLAtyping and distance2self tests, update CHANGELOG * add Peptide functon to determine if peptide originates from a variant * fix peptide call, update CHANGELOG * improve method for variant-peptide check * minor CHANGELOG change * change peptide to self * update setup.py and CHANGELOG * Fix errorneous supported alleles (#78) * Draft for outsourcing supported alleles * Further outsourcing of netmhc alleles * Finish outsourcing external alleles * Outsource alleles from pssm and ann predictors * Correct minor erroneous hla nomenclatures of smmpmbec * Change allele imports by importing frozensets * Add __allele_import_name to classes to increase readability * Refactor: convert_alleles is now classmethod in pssm * Incorporate feedback * Fix parsing error and sort allele list * Adjust variable naming Co-authored-by: Leon Kuchenbecker <[email protected]> Co-authored-by: Jonas Scheid <[email protected]> Co-authored-by: Jonas Scheid <[email protected]> Co-authored-by: Gisela Gabernet <[email protected]>

jonasscheid and others added 5 commits October 26, 2021 11:26

Push all changes made on fork

e902432

Set Setuptools version also for external yml

6d23a7f

Fixed erroneous variable names in matrix files

a52eb9e

deleted A_2601_9 matrix for now. Caused troubles

eea16e0

Add A2601_9 syf matrix for debugging

12b121d

jonasscheid added 2 commits November 3, 2021 15:46

Fixed bug in test caused by addition of A*26:01 matrix

bbe2d06

Change solver from cbc to glpk to investigate if macOS dependant env …

cee9128

…problems in github actions can be solved

christopher-mohr self-requested a review November 17, 2021 15:31

christopher-mohr requested changes Nov 17, 2021

View reviewed changes

christopher-mohr requested changes Nov 18, 2021

View reviewed changes

Corrected after review

76d865e

Adjust tutorials to new structure

0be7047

christopher-mohr requested review from lkuchenb and b-schubert November 23, 2021 10:13

christopher-mohr requested changes Nov 23, 2021

View reviewed changes

Change filter_result as discussed

fa81e6d

jonasscheid mentioned this pull request Nov 24, 2021

Notebooks + doc require porting to Python 3 #12

Closed

jonasscheid added 3 commits November 24, 2021 14:14

Adjusted filter method and tutorials according to #12

6534b0e

Fixed a bug occuring for netMHCfamily tools when peptide input has mu…

d716eee

…ltiple lengths

remove logging

338face

christopher-mohr requested changes Nov 28, 2021

View reviewed changes

christopher-mohr requested changes Dec 5, 2021

View reviewed changes

Alter filter_result method as discussed

0cb386c

christopher-mohr approved these changes Dec 5, 2021

View reviewed changes

christopher-mohr merged commit 65fd57a into develop Dec 13, 2021

jonasscheid deleted the feature-15-add-rank-metric branch December 13, 2021 21:57

christopher-mohr mentioned this pull request Dec 22, 2021

Support multiple epitope prediction scores per tool #15

Closed

jonasscheid mentioned this pull request May 24, 2022

Missing alleles in supportedAlleles set of syfpeithi #60

Closed

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Add rank metric #15 #42

Add rank metric #15 #42

jonasscheid commented Oct 26, 2021 •

edited

Loading

jonasscheid commented Oct 26, 2021

b-schubert commented Oct 26, 2021 •

edited

Loading

jonasscheid commented Nov 3, 2021

christopher-mohr left a comment

christopher-mohr left a comment

christopher-mohr commented Nov 22, 2021

christopher-mohr left a comment

christopher-mohr left a comment

christopher-mohr left a comment •

edited

Loading

christopher-mohr Nov 28, 2021

jonasscheid Nov 28, 2021

christopher-mohr Dec 5, 2021

christopher-mohr left a comment

christopher-mohr left a comment

Add rank metric #15 #42

Add rank metric #15 #42

Conversation

jonasscheid commented Oct 26, 2021 • edited Loading

jonasscheid commented Oct 26, 2021

b-schubert commented Oct 26, 2021 • edited Loading

jonasscheid commented Nov 3, 2021

christopher-mohr left a comment

Choose a reason for hiding this comment

christopher-mohr left a comment

Choose a reason for hiding this comment

christopher-mohr commented Nov 22, 2021

christopher-mohr left a comment

Choose a reason for hiding this comment

christopher-mohr left a comment

Choose a reason for hiding this comment

christopher-mohr left a comment • edited Loading

Choose a reason for hiding this comment

christopher-mohr Nov 28, 2021

Choose a reason for hiding this comment

jonasscheid Nov 28, 2021

Choose a reason for hiding this comment

christopher-mohr Dec 5, 2021

Choose a reason for hiding this comment

christopher-mohr left a comment

Choose a reason for hiding this comment

christopher-mohr left a comment

Choose a reason for hiding this comment

jonasscheid commented Oct 26, 2021 •

edited

Loading

b-schubert commented Oct 26, 2021 •

edited

Loading

christopher-mohr left a comment •

edited

Loading