Add COMEbin as another optional binner #234
Open
+88
−6
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
|
Weird test failure: STDERR was: b'11/28/2024 02:00:39 AM INFO: Time - 02:00:39 28-11-2024
11/28/2024 02:00:39 AM INFO: Command - /usr/share/miniconda/envs/test/bin/aviary assemble -1 /home/runner/work/aviary/aviary/test/data/wgsim.1.fq.gz -2 /home/runner/work/aviary/aviary/test/data/wgsim.2.fq.gz --output /tmp/tmpnd6jzx19/test --conda-prefix /home/runner/work/aviary/aviary/test/data/.conda --dryrun --tmpdir /tmp/tmpnd6jzx19 --snakemake-cmds --quiet
11/28/2024 02:00:39 AM INFO: Version - 0.10.0
11/28/2024 02:00:39 AM INFO: Configuration file written to /tmp/tmpnd6jzx19/test/config.yaml
11/28/2024 02:00:39 AM INFO: Executing: snakemake --snakefile /home/runner/work/aviary/aviary/aviary/modules/Snakefile --directory /tmp/tmpnd6jzx19/test --cores 16 --rerun-incomplete --keep-going --quiet --rerun-triggers mtime --configfile /tmp/tmpnd6jzx19/test/config.yaml --nolock --retries 0 --conda-frontend mamba --use-conda --conda-prefix /home/runner/work/aviary/aviary/test/data/.conda --dryrun complete_assembly_with_qc
Traceback (most recent call last):
File "/usr/share/miniconda/envs/test/lib/python3.10/site-packages/snakemake/__init__.py", line 671, in snakemake
workflow.include(
File "/usr/share/miniconda/envs/test/lib/python3.10/site-packages/snakemake/workflow.py", line 1362, in include
snakefile = infer_source_file(snakefile, basedir)
File "/usr/share/miniconda/envs/test/lib/python3.10/site-packages/snakemake/sourcecache.py", line 323, in infer_source_file
if is_local_file(path_or_uri):
File "/usr/share/miniconda/envs/test/lib/python3.10/site-packages/snakemake/common/__init__.py", line 92, in is_local_file
return parse_uri(path_or_uri).scheme == "file"
File "/usr/share/miniconda/envs/test/lib/python3.10/site-packages/snakemake/common/__init__.py", line 96, in parse_uri
from smart_open import parse_uri
File "/usr/share/miniconda/envs/test/lib/python3.10/site-packages/smart_open/__init__.py", line 34, in <module>
from .smart_open_lib import open, parse_uri, smart_open, register_compressor # noqa: E402
File "/usr/share/miniconda/envs/test/lib/python3.10/site-packages/smart_open/smart_open_lib.py", line 35, in <module>
from smart_open import doctools
File "/usr/share/miniconda/envs/test/lib/python3.10/site-packages/smart_open/doctools.py", line 21, in <module>
from . import transport
File "/usr/share/miniconda/envs/test/lib/python3.10/site-packages/smart_open/transport.py", line 104, in <module>
register_transport("smart_open.s3")
File "/usr/share/miniconda/envs/test/lib/python3.10/site-packages/smart_open/transport.py", line 49, in register_transport
submodule = importlib.import_module(submodule)
File "/usr/share/miniconda/envs/test/lib/python3.10/importlib/__init__.py", line 126, in import_module
return _bootstrap._gcd_import(name[level:], package, level)
File "/usr/share/miniconda/envs/test/lib/python3.10/site-packages/smart_open/s3.py", line 25, in <module>
import boto3
File "/usr/share/miniconda/envs/test/lib/python3.10/site-packages/boto3/__init__.py", line 17, in <module>
from boto3.session import Session
File "/usr/share/miniconda/envs/test/lib/python3.10/site-packages/boto3/session.py", line 17, in <module>
import botocore.session
File "/usr/share/miniconda/envs/test/lib/python3.10/site-packages/botocore/session.py", line 26, in <module>
import botocore.client
File "/usr/share/miniconda/envs/test/lib/python3.10/site-packages/botocore/client.py", line 15, in <module>
from botocore import waiter, xform_name
File "/usr/share/miniconda/envs/test/lib/python3.10/site-packages/botocore/waiter.py", line 18, in <module>
from botocore.docs.docstring import WaiterDocstring
File "/usr/share/miniconda/envs/test/lib/python3.10/site-packages/botocore/docs/__init__.py", line 15, in <module>
from botocore.docs.service import ServiceDocumenter
File "/usr/share/miniconda/envs/test/lib/python3.10/site-packages/botocore/docs/service.py", line 14, in <module>
from botocore.docs.client import (
File "/usr/share/miniconda/envs/test/lib/python3.10/site-packages/botocore/docs/client.py", line 18, in <module>
from botocore.docs.example import ResponseExampleDocumenter
File "/usr/share/miniconda/envs/test/lib/python3.10/site-packages/botocore/docs/example.py", line 13, in <module>
from botocore.docs.shape import ShapeDocumenter
File "/usr/share/miniconda/envs/test/lib/python3.10/site-packages/botocore/docs/shape.py", line 19, in <module>
from botocore.utils import is_json_value_header
File "/usr/share/miniconda/envs/test/lib/python3.10/site-packages/botocore/utils.py", line 39, in <module>
import botocore.httpsession
File "/usr/share/miniconda/envs/test/lib/python3.10/site-packages/botocore/httpsession.py", line 45, in <module>
from urllib3.contrib.pyopenssl import (
File "/usr/share/miniconda/envs/test/lib/python3.10/site-packages/urllib3/contrib/pyopenssl.py", line 43, in <module>
import OpenSSL.SSL # type: ignore[import-untyped]
File "/usr/share/miniconda/envs/test/lib/python3.10/site-packages/OpenSSL/__init__.py", line 8, in <module>
from OpenSSL import crypto, SSL
File "/usr/share/miniconda/envs/test/lib/python3.10/site-packages/OpenSSL/crypto.py", line 1579, in <module>
class X509StoreFlags(object):
File "/usr/share/miniconda/envs/test/lib/python3.10/site-packages/OpenSSL/crypto.py", line 1598, in X509StoreFlags
NOTIFY_POLICY = _lib.X509_V_FLAG_NOTIFY_POLICY
AttributeError: module \'lib\' has no attribute \'X509_V_FLAG_NOTIFY_POLICY\'
11/28/2024 02:00:39 AM CRITICAL: Command \'[\'snakemake\', \'--snakefile\', \'/home/runner/work/aviary/aviary/aviary/modules/Snakefile\', \'--directory\', \'/tmp/tmpnd6jzx19/test\', \'--cores\', \'16\', \'--rerun-incomplete\', \'--keep-going\', \'--quiet\', \'--rerun-triggers\', \'mtime\', \'--configfile\', \'/tmp/tmpnd6jzx19/test/config.yaml\', \'--nolock\', \'--retries\', \'0\', \'--conda-frontend\', \'mamba\', \'--use-conda\', \'--conda-prefix\', \'/home/runner/work/aviary/aviary/test/data/.conda\', \'--dryrun\', \'complete_assembly_with_qc\']\' returned non-zero exit status 1.
'STDOUT was: b'' |
|
Fyi I tried this on the comebin branch I think. Gave up as it was funky in tests. Better to concentrate on vamb tax?
-\-------------
Ben Woodcroft
Group leader, Centre for Microbiome Research, QUT
…________________________________
From: Samuel Aroney ***@***.***>
Sent: Thursday, November 28, 2024 12:02:08 PM
To: rhysnewell/aviary ***@***.***>
Cc: Subscribed ***@***.***>
Subject: Re: [rhysnewell/aviary] Add COMEbin as another default binner (PR #234)
Weird test failure:
STDERR was: b'11/28/2024 02:00:39 AM INFO: Time - 02:00:39 28-11-2024
11/28/2024 02:00:39 AM INFO: Command - /usr/share/miniconda/envs/test/bin/aviary assemble -1 /home/runner/work/aviary/aviary/test/data/wgsim.1.fq.gz -2 /home/runner/work/aviary/aviary/test/data/wgsim.2.fq.gz --output /tmp/tmpnd6jzx19/test --conda-prefix /home/runner/work/aviary/aviary/test/data/.conda --dryrun --tmpdir /tmp/tmpnd6jzx19 --snakemake-cmds --quiet
11/28/2024 02:00:39 AM INFO: Version - 0.10.0
11/28/2024 02:00:39 AM INFO: Configuration file written to /tmp/tmpnd6jzx19/test/config.yaml
11/28/2024 02:00:39 AM INFO: Executing: snakemake --snakefile /home/runner/work/aviary/aviary/aviary/modules/Snakefile --directory /tmp/tmpnd6jzx19/test --cores 16 --rerun-incomplete --keep-going --quiet --rerun-triggers mtime --configfile /tmp/tmpnd6jzx19/test/config.yaml --nolock --retries 0 --conda-frontend mamba --use-conda --conda-prefix /home/runner/work/aviary/aviary/test/data/.conda --dryrun complete_assembly_with_qc
Traceback (most recent call last):
File "/usr/share/miniconda/envs/test/lib/python3.10/site-packages/snakemake/__init__.py", line 671, in snakemake
workflow.include(
File "/usr/share/miniconda/envs/test/lib/python3.10/site-packages/snakemake/workflow.py", line 1362, in include
snakefile = infer_source_file(snakefile, basedir)
File "/usr/share/miniconda/envs/test/lib/python3.10/site-packages/snakemake/sourcecache.py", line 323, in infer_source_file
if is_local_file(path_or_uri):
File "/usr/share/miniconda/envs/test/lib/python3.10/site-packages/snakemake/common/__init__.py", line 92, in is_local_file
return parse_uri(path_or_uri).scheme == "file"
File "/usr/share/miniconda/envs/test/lib/python3.10/site-packages/snakemake/common/__init__.py", line 96, in parse_uri
from smart_open import parse_uri
File "/usr/share/miniconda/envs/test/lib/python3.10/site-packages/smart_open/__init__.py", line 34, in <module>
from .smart_open_lib import open, parse_uri, smart_open, register_compressor # noqa: E402
File "/usr/share/miniconda/envs/test/lib/python3.10/site-packages/smart_open/smart_open_lib.py", line 35, in <module>
from smart_open import doctools
File "/usr/share/miniconda/envs/test/lib/python3.10/site-packages/smart_open/doctools.py", line 21, in <module>
from . import transport
File "/usr/share/miniconda/envs/test/lib/python3.10/site-packages/smart_open/transport.py", line 104, in <module>
register_transport("smart_open.s3")
File "/usr/share/miniconda/envs/test/lib/python3.10/site-packages/smart_open/transport.py", line 49, in register_transport
submodule = importlib.import_module(submodule)
File "/usr/share/miniconda/envs/test/lib/python3.10/importlib/__init__.py", line 126, in import_module
return _bootstrap._gcd_import(name[level:], package, level)
File "/usr/share/miniconda/envs/test/lib/python3.10/site-packages/smart_open/s3.py", line 25, in <module>
import boto3
File "/usr/share/miniconda/envs/test/lib/python3.10/site-packages/boto3/__init__.py", line 17, in <module>
from boto3.session import Session
File "/usr/share/miniconda/envs/test/lib/python3.10/site-packages/boto3/session.py", line 17, in <module>
import botocore.session
File "/usr/share/miniconda/envs/test/lib/python3.10/site-packages/botocore/session.py", line 26, in <module>
import botocore.client
File "/usr/share/miniconda/envs/test/lib/python3.10/site-packages/botocore/client.py", line 15, in <module>
from botocore import waiter, xform_name
File "/usr/share/miniconda/envs/test/lib/python3.10/site-packages/botocore/waiter.py", line 18, in <module>
from botocore.docs.docstring import WaiterDocstring
File "/usr/share/miniconda/envs/test/lib/python3.10/site-packages/botocore/docs/__init__.py", line 15, in <module>
from botocore.docs.service import ServiceDocumenter
File "/usr/share/miniconda/envs/test/lib/python3.10/site-packages/botocore/docs/service.py", line 14, in <module>
from botocore.docs.client import (
File "/usr/share/miniconda/envs/test/lib/python3.10/site-packages/botocore/docs/client.py", line 18, in <module>
from botocore.docs.example import ResponseExampleDocumenter
File "/usr/share/miniconda/envs/test/lib/python3.10/site-packages/botocore/docs/example.py", line 13, in <module>
from botocore.docs.shape import ShapeDocumenter
File "/usr/share/miniconda/envs/test/lib/python3.10/site-packages/botocore/docs/shape.py", line 19, in <module>
from botocore.utils import is_json_value_header
File "/usr/share/miniconda/envs/test/lib/python3.10/site-packages/botocore/utils.py", line 39, in <module>
import botocore.httpsession
File "/usr/share/miniconda/envs/test/lib/python3.10/site-packages/botocore/httpsession.py", line 45, in <module>
from urllib3.contrib.pyopenssl import (
File "/usr/share/miniconda/envs/test/lib/python3.10/site-packages/urllib3/contrib/pyopenssl.py", line 43, in <module>
import OpenSSL.SSL # type: ignore[import-untyped]
File "/usr/share/miniconda/envs/test/lib/python3.10/site-packages/OpenSSL/__init__.py", line 8, in <module>
from OpenSSL import crypto, SSL
File "/usr/share/miniconda/envs/test/lib/python3.10/site-packages/OpenSSL/crypto.py", line 1579, in <module>
class X509StoreFlags(object):
File "/usr/share/miniconda/envs/test/lib/python3.10/site-packages/OpenSSL/crypto.py", line 1598, in X509StoreFlags
NOTIFY_POLICY = _lib.X509_V_FLAG_NOTIFY_POLICY
AttributeError: module \'lib\' has no attribute \'X509_V_FLAG_NOTIFY_POLICY\'
11/28/2024 02:00:39 AM CRITICAL: Command \'[\'snakemake\', \'--snakefile\', \'/home/runner/work/aviary/aviary/aviary/modules/Snakefile\', \'--directory\', \'/tmp/tmpnd6jzx19/test\', \'--cores\', \'16\', \'--rerun-incomplete\', \'--keep-going\', \'--quiet\', \'--rerun-triggers\', \'mtime\', \'--configfile\', \'/tmp/tmpnd6jzx19/test/config.yaml\', \'--nolock\', \'--retries\', \'0\', \'--conda-frontend\', \'mamba\', \'--use-conda\', \'--conda-prefix\', \'/home/runner/work/aviary/aviary/test/data/.conda\', \'--dryrun\', \'complete_assembly_with_qc\']\' returned non-zero exit status 1.
'STDOUT was: b''
―
Reply to this email directly, view it on GitHub<#234 (comment)>, or unsubscribe<https://github.com/notifications/unsubscribe-auth/AAADX5F55MHIYJAGQV77QNL2CZ2SBAVCNFSM6AAAAABSUCANVWVHI2DSMVQWIX3LMV43OSLTON2WKQ3PNVWWK3TUHMZDKMBVGEYTSMJRHA>.
You are receiving this because you are subscribed to this thread.Message ID: ***@***.***>
|
|
Ok. Can't find that branch. It runs fine here in the integration tests. I think the failure here is unrelated. |
|
Happy for this to be added as an optional binner, but that weird test failure should probs get fixed up. I can't really make sense of what that error might be due to, perhaps a python versioning issue? |
AroneyS
changed the title
|
@AroneyS let me know when you'd like me to review |
|
Ready. New integration test passes. |
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Test passes, but it adds a bunch of tmp files at the position of the input assembly:
ls example/test_short_read_recovery_comebin/assembly.fasta* example/test_short_read_recovery_comebin/assembly.fasta example/test_short_read_recovery_comebin/assembly.fasta.frag.faa example/test_short_read_recovery_comebin/assembly.fasta.bacar_marker.2quarter_lencutoff_1001.seed example/test_short_read_recovery_comebin/assembly.fasta.frag.ffn example/test_short_read_recovery_comebin/assembly.fasta.bacar_marker.hmmout example/test_short_read_recovery_comebin/assembly.fasta.frag.gff example/test_short_read_recovery_comebin/assembly.fasta.bacar_marker.hmmout.err example/test_short_read_recovery_comebin/assembly.fasta.frag.out example/test_short_read_recovery_comebin/assembly.fasta.bacar_marker.hmmout.out example/test_short_read_recovery_comebin/assembly.fasta_lengths.txt example/test_short_read_recovery_comebin/assembly.fasta.frag.errI guess we could symlink the assembly into the comebin output folder to contain the mess.