errors downloading dbs

[microbemarsh]

When trying to download the databases using the aviary configure --download method, I keep running into an error with the conda envs. Running v0.10.0 and installed using the conda / mamba method. Any help would be greatly appreciated!

(aviary) [tmhagm8@hmrihpcp03 aviary_dbs]$ aviary configure --download checkm2 eggnog singlem
12/01/2024 08:20:41 PM INFO: Time - 20:20:41 01-12-2024
12/01/2024 08:20:41 PM INFO: Command - /home/tmhagm8/scratch/.conda/envs/aviary/bin/aviary configure --download checkm2 eggnog singlem
12/01/2024 08:20:41 PM INFO: Version - 0.10.0
12/01/2024 08:20:41 PM INFO: The current aviary environment variables are:
12/01/2024 08:20:41 PM INFO: CONDA_ENV_PATH: /home/tmhagm8/scratch/.conda
12/01/2024 08:20:41 PM INFO: TMPDIR: /home/tmhagm8/scratch/abx_depl/meta_workdir/aviary_tmp
12/01/2024 08:20:41 PM INFO: GTDBTK_DATA_PATH: /home/tmhagm8/scratch/gtdbtk_db
12/01/2024 08:20:41 PM INFO: EGGNOG_DATA_DIR: /home/tmhagm8/scratch/aviary_dbs/eggnog_db/
12/01/2024 08:20:41 PM INFO: CHECKM2DB: /home/tmhagm8/scratch/aviary_dbs/checkm2/
12/01/2024 08:20:41 PM INFO: SINGLEM_METAPACKAGE_PATH: /home/tmhagm8/scratch/aviary_dbs/singlem_db/
12/01/2024 08:20:41 PM INFO: Configuration file written to /condo/neurobiome/tmhagm8/aviary_dbs/config.yaml
12/01/2024 08:20:41 PM INFO: Executing: snakemake --snakefile /home/tmhagm8/scratch/.conda/envs/aviary/lib/python3.11/site-packages/aviary/modules/Snakefile --directory /condo/neurobiome/tmhagm8/aviary_dbs --cores 16 --rerun-incomplete --keep-going  --rerun-triggers mtime --configfile /condo/neurobiome/tmhagm8/aviary_dbs/config.yaml --nolock  --retries 0 --conda-frontend mamba --resources mem_mb=256000   --use-conda --conda-prefix /home/tmhagm8/scratch/.conda   download_databases
Building DAG of jobs...
Creating conda environment /home/tmhagm8/scratch/.conda/envs/aviary/lib/python3.11/site-packages/aviary/modules/annotation/../../envs/checkm2.yaml...
Downloading and installing remote packages.
CreateCondaEnvironmentException:
Could not create conda environment from /home/tmhagm8/scratch/.conda/envs/aviary/lib/python3.11/site-packages/aviary/modules/annotation/../../envs/checkm2.yaml:
Command:
mamba env create --quiet --file "/home/tmhagm8/scratch/.conda/ad2fc3b4b673c1aa0247f9bb03dc8f55_.yaml" --prefix "/home/tmhagm8/scratch/.conda/ad2fc3b4b673c1aa0247f9bb03dc8f55_"
Output:
error    libmamba Non-conda folder exists at prefix
critical libmamba Aborting.

12/01/2024 08:20:45 PM CRITICAL: Command '['snakemake', '--snakefile', '/home/tmhagm8/scratch/.conda/envs/aviary/lib/python3.11/site-packages/aviary/modules/Snakefile', '--directory', '/condo/neurobiome/tmhagm8/aviary_dbs', '--cores', '16', '--rerun-incomplete', '--keep-going', '--rerun-triggers', 'mtime', '--configfile', '/condo/neurobiome/tmhagm8/aviary_dbs/config.yaml', '--nolock', '--retries', '0', '--conda-frontend', 'mamba', '--resources', 'mem_mb=256000', '--use-conda', '--conda-prefix', '/home/tmhagm8/scratch/.conda', 'download_databases']' returned non-zero exit status 1.

I looked through this github issue which had a similar issue and when I deleted the --prefix file and reran, it appeared to create a prefix file.

[rhysnewell]

Hello,
Would you mind retrying but set the --conda-prefix parameter to an empty folder? I think for some reason it is complaining because the envs/ folder exists inside your .conda/ folder

Cheers,
Rhys

[microbemarsh]

It's still giving the same issue, I've always used the .conda folder as where to add the conda envs that aviary creates (no idea if this is good practice or not but its worked so far!)

(aviary) [tmhagm8@hmrihpcp03 aviary_dbs]$ aviary configure --conda-prefix /home/tmhagm8/scratch/aviary_dbs/blank/ --download checkm2 eggnog singlem
12/01/2024 08:38:32 PM INFO: Time - 20:38:32 01-12-2024
12/01/2024 08:38:32 PM INFO: Command - /home/tmhagm8/scratch/.conda/envs/aviary/bin/aviary configure --conda-prefix /home/tmhagm8/scratch/aviary_dbs/blank/ --download checkm2 eggnog singlem
12/01/2024 08:38:32 PM INFO: Version - 0.10.0
12/01/2024 08:38:32 PM INFO: The current aviary environment variables are:
12/01/2024 08:38:32 PM INFO: CONDA_ENV_PATH: /home/tmhagm8/scratch/aviary_dbs/blank/
12/01/2024 08:38:32 PM INFO: TMPDIR: /home/tmhagm8/scratch/abx_depl/meta_workdir/aviary_tmp
12/01/2024 08:38:32 PM INFO: GTDBTK_DATA_PATH: /home/tmhagm8/scratch/gtdbtk_db
12/01/2024 08:38:32 PM INFO: EGGNOG_DATA_DIR: /home/tmhagm8/scratch/aviary_dbs/eggnog_db/
12/01/2024 08:38:32 PM INFO: CHECKM2DB: /home/tmhagm8/scratch/aviary_dbs/checkm2/
12/01/2024 08:38:32 PM INFO: SINGLEM_METAPACKAGE_PATH: /home/tmhagm8/scratch/aviary_dbs/singlem_db/
12/01/2024 08:38:32 PM INFO: Configuration file written to /condo/neurobiome/tmhagm8/aviary_dbs/config.yaml
12/01/2024 08:38:32 PM INFO: Executing: snakemake --snakefile /home/tmhagm8/scratch/.conda/envs/aviary/lib/python3.11/site-packages/aviary/modules/Snakefile --directory /condo/neurobiome/tmhagm8/aviary_dbs --cores 16 --rerun-incomplete --keep-going  --rerun-triggers mtime --configfile /condo/neurobiome/tmhagm8/aviary_dbs/config.yaml --nolock  --retries 0 --conda-frontend mamba --resources mem_mb=256000   --use-conda --conda-prefix /home/tmhagm8/scratch/aviary_dbs/blank/   download_databases
Building DAG of jobs...
Creating conda environment /home/tmhagm8/scratch/.conda/envs/aviary/lib/python3.11/site-packages/aviary/modules/annotation/../../envs/singlem.yaml...
Downloading and installing remote packages.
CreateCondaEnvironmentException:
Could not create conda environment from /home/tmhagm8/scratch/.conda/envs/aviary/lib/python3.11/site-packages/aviary/modules/annotation/../../envs/singlem.yaml:
Command:
mamba env create --quiet --file "/home/tmhagm8/scratch/aviary_dbs/blank/21257d5355ea9b7e9afc92be3897f294_.yaml" --prefix "/home/tmhagm8/scratch/aviary_dbs/blank/21257d5355ea9b7e9afc92be3897f294_"
Output:
error    libmamba Non-conda folder exists at prefix
critical libmamba Aborting.

12/01/2024 08:38:36 PM CRITICAL: Command '['snakemake', '--snakefile', '/home/tmhagm8/scratch/.conda/envs/aviary/lib/python3.11/site-packages/aviary/modules/Snakefile', '--directory', '/condo/neurobiome/tmhagm8/aviary_dbs', '--cores', '16', '--rerun-incomplete', '--keep-going', '--rerun-triggers', 'mtime', '--configfile', '/condo/neurobiome/tmhagm8/aviary_dbs/config.yaml', '--nolock', '--retries', '0', '--conda-frontend', 'mamba', '--resources', 'mem_mb=256000', '--use-conda', '--conda-prefix', '/home/tmhagm8/scratch/aviary_dbs/blank/', 'download_databases']' returned non-zero exit status 1.

[wwood]

As per that GitHub issue you mentioned @microbemarsh are you running mamba v2 or v1?

[wwood]

I wonder if it makes sense to (1) pin mamba to 1.x or (2) just go back to using the default conda, since now it has the faster solver integrated. WDYT @rhysnewell ?

[microbemarsh]

Hi @wwood,

I'm running mamba 2 in the environment, at first I was using whatever ships with aviary and had the problem above. I then ran mamba install conda-forge::mamba==2.04 to get the newest version with hopes of that fixing the issue and it persisted.