Run bgcflow efficiently on a lot of small snipplets

[FriederikeBiermann]

Hi,
I am trying to use bgcflow on a lot of nucleotide snippets (ca 19.000, all longer than 5 kb, species, genes etc all already annotated by NCBI) to get an overview of their biosynthetic potential and I have access to an HPC where I can request a lot of resources. Is bgcflow made to be parallelized that heavy? Or would it make more sense to split my 19.000 files into smaller projects (say 100 seqs per project) and then run BGCflow over them in an embarrassingly parallel fashion? I guess I would need to do my bigscape analysis again afterwards to really compare all of them. Is there a nicer way of doing that? Thank you so much for your help! Best, Frida

[matinnuhamunada]

Hi Frida, thanks for the inquiry, this is actually something that I wanted to try (running large scale in SLURM or Azure Batch) but haven't got the chance to do so.

Basically, BGCFlow is a Snakemake workflow with several snakefiles for different purposes. The CLI we build for bgcflow is just a commonly used snakemake commands. The command bgcflow run is a wrapper to:

snakemake --snakefile workflow/Snakefile --use-conda --keep-going --rerun-incomplete --rerun-triggers mtime -c 2 --dryrun --wms-monitor http://127.0.0.1:5000

Snakemake should be able to distribute large jobs, or even do batches of smaller jobs (e.g. 500 at a time). More about the command line here: https://snakemake.readthedocs.io/en/stable/executing/cli.html

What I think can help right now with BGCFlow:

• Use custom genbank file as input
• Use custom taxonomic assignment file
• Customize resource parameter (number of threads, memory, etc.) with snakemake profile to machine

What I can't help right now but is planned in the future:

• Making sure BGCFlow works with SLURM or other cloud services (Azure Batch, AWS, kubernetes etc)

My suggestion:

• I'm guessing that you already have annotated genbanks as input (This is also on my to do list since a while ago :)?
  I'm working on it in this development branch right now: https://github.com/NBChub/bgcflow/tree/dev-0.8.0-1. If you can share me 3-5 genbanks for a dummy test, I can try to create the config for you.
• Are the species information in the genbank files? For BGCFlow, we follow GTDB-tk taxonomy output format. If you don't have it, we can write simple scripts to fetch them from the genbank files.
• We can build a customized snakemake profile for your machine. Can you describe a little bit about the system you have?

[FriederikeBiermann]

Thank you so much! For now, I just manually ran antismash on them via SLURM on the HPC, because I guess that's one of the computationally heavier tasks to do (especially with a lot of clusterblast options). I am working on either a SLURM scheduled CPU HPC (running CentOS Linux 8 (Core)) or on my personal computer (Ubuntu 22.04, 64 GB RAM, Intel Core vPRO i9). I guess since I already have kind of nicely annotated genbank files, some of the analysis of your workflows are maybe not that necessary. I attached some files for you for the genbank files. There is an organism in there, but I know for a fact that this is not correctly annotated (these are all prokaryotic contigs, but the annotation states eukaryotes). I also have a large table with all the lineage information of the true organism including all taxids. I uploaded the files again with manually changed organisms, that might not be the prettiest, but I can also write a script to exchange the species. Note that I changed the file extensions of the files I uploaded to .txt, because github won't let me upload .gb files. Thanks again so much for the super helpful and fast answer!
NW_003318762.1_0_43680.txt
NW_019110264.1_0_31676.txt
NW_003385442.1_0_46725.txt
NZ_KE386655.1_0_25886.txt
NW_003318762.1_0_43680_changed_species.txt
NW_019110264.1_0_31676_changed_species.txt
NW_003385442.1_0_46725_changed_species.txt
NZ_KE386655.1_0_25886_changed_species.txt

[matinnuhamunada]

Tried to implement it here: https://github.com/matinnuhamunada/custom_gbk_project_bgcflow, do please give it a try.

FYI, antiSMASH 7.0.0 doesn't detect any regions in the input file. I look into the genbanks, and some of the sequence has multiple nnnn. Is that expected?

[matinnuhamunada]

Oh, I just tried running bgcflow on an LSF HPC cluster and it worked. There are some failed rules because our cluster does not allow FTP, but for those I can run it locally (download from NCBI, download antismash db, etc).

You can find more about using the profiles here: https://github.com/Snakemake-Profiles/doc

[FriederikeBiermann]

Thank you so much! I know, most of the files are not to expected to have anything in there :)

[FriederikeBiermann]

Hi,
I tried out the implementation and it seems to work as long as I don't use the custom profile. If I utilize the custom profile, it does not seem to do anything (the output stops after
GTDB API | Grabbing metadata using GTDB release version: r214
GTDB API | Testing connection to: https://gtdb-api.ecogenomic.org/status/db
GTDB API | Database is online: True
GTDB API | Searching in offline mode: False
without showing any error). I utilised this SLURM Script:

#!/bin/bash
#SBATCH --job-name=bgcflow
#SBATCH --ntasks=1
#SBATCH --nodes=1
#SBATCH -o output_filepath
#SBATCH --time=24:00:00
#SBATCH -p batch
#SBATCH --cpus-per-task=64
#SBATCH -e error_filepath

source ~/.bashrc
conda activate bgcflow
snakemake \
  --snakefile workflow/Snakefile \
  --use-conda \
  --keep-going \
  --rerun-incomplete \
  --rerun-triggers mtime \
  -c 64 \
#  --profile profile \
  --unlock

Thank you so much for taking the time to create an extra workflow for me!!
Best
Frida

[matinnuhamunada]

Thanks for giving it a try, getting a feedback like this help us to improve the workflow :)

Unfortunately, I don't have access to a SLURM facility for testing.
Have you tried using the cookie cutter template for SLURM? It is available here: https://github.com/Snakemake-Profiles/slurm/tree/master

If you are using the template, you can edit the config file generated in ${HOME}/.config/snakemake/config.yaml:

cluster-sidecar: "slurm-sidecar.py"
cluster-cancel: "scancel"
restart-times: "0"
jobscript: "slurm-jobscript.sh"
cluster: "slurm-submit.py"
cluster-status: "slurm-status.py"
max-jobs-per-second: "10"
max-status-checks-per-second: "10"
local-cores: 1
latency-wait: "5"
use-conda: "True"
use-singularity: "False"
jobs: "500"
printshellcmds: "True"

# Example resource configuration
# default-resources:
#   - runtime=100
#   - mem_mb=6000
#   - disk_mb=1000000
# # set-threads: map rule names to threads
# set-threads:
#   - single_core_rule=1
#   - multi_core_rule=10
# # set-resources: map rule names to resources in general
# set-resources:
#   - high_memory_rule:mem_mb=12000
#   - long_running_rule:runtime=1200

You can then customize the set-threads variable accordingly (e.g. change number of cores for bigscape etc.).