Failed to pull singularity image

[ilenia6]

Description of the bug

I was trying to run the sanger-tol/curationpretext pipeline to generate the contact map for my genome, but it gives me a "Failed to pull singularity image" which seems to come from "access to the requested resource is not authorized" (I am running it in a job in HTCondor). Is there something I am missing in the configuration?

Command used and terminal output

nextflow run sanger-tol/curationpretext \
  --input assembly.fasta \
  --cram cram_files/ \
  --longread in/ \
  --longread_type ont \
  --sample cb3 \
  --teloseq AATGG \
  --outdir output_dir \
  -profile singularity


Failed to pull singularity image
    command: singularity pull  --name quay.io-sanger-tol-cramfilter_bwamem2_minimap2_samtools_perl-0.001-c1.img.pulling.1731944247517 docker://quay.io/sanger-tol/cramfilter_bwamem2_minimap2_samtools_perl:0.001-c1 > /dev/null
    status : 255
    hint   : Try and increase singularity.pullTimeout in the config (current is "20m")
    message:
      INFO:    /etc/singularity/ exists; cleanup by system administrator is not complete (see https://apptainer.org/docs/admin/latest/singularity_migration.html)
      FATAL:   While making image from oci registry: error fetching image to cache: failed to get checksum for docker://quay.io/sanger-tol/cramfilter_bwamem2_minimap2_samtools_perl:0.001-c1: reading manifest 0.001-c1 in quay.io/sanger-tol/cramfilter_bwamem2_minimap2_samtools_perl: unauthorized: access to the requested resource is not authorized

Relevant files

No response

System information

No response

[yumisims]

Hello, we are in the process of updating the modules in the pipeline due to a policy change on Anaconda, which has led to the image being deleted. I will get back to you once the update is complete. Thank you.

[Surbhigrewal]

Hi is there any update on this please?

[DLBPointon]

Hi all,

Apologies for not seeing this sooner!

I will prioritise this for Monday 2nd December. This shouldn't take too long, I've upgraded the other pipelines I work on, in around a day or so.

The particular issue here is that this container is simply no longer available, ours are cached which is why we won't have been affected.

Apologies again!

[DLBPointon]

Hi all,
Updates are pretty much finished, I just need to update the Changelog. However you should be able to download the branch: https://github.com/sanger-tol/curationpretext/tree/dp24_spelling if you need it right now.

[Surbhigrewal]

Hi Damon,

It didnt't work.

ERROR ~ Execution aborted due to an unexpected error

-- Check script 'curationpretext-dp24_spelling/./workflows/../subworkflows/local/./../../modules/nf-core/windowmasker/mkcounts/main.nf' at line: 6 or see '.nextflow.log' file for more details.

[DLBPointon]

Hi @Surbhigrewal

Is that the whole error? Just that mkcounts is wrong?

Could you post the .log please

[Surbhigrewal]

Hi @DLBPointon,

I just ran my files through the branch (attached local.log) and through sanger-tol (attached sanger-tol.log). Both ended with errors.
nextflow_local.log
nextflow_sanger-tol.log

[ilenia6]

Hi @DLBPointon, I still have the same error related to singularity image.

[DLBPointon]

Morning Both,

I can't reproduce the same errors. I've run the pipeline myself in a handfull of different ways and can't get close to the issues you are experiencing at this point. The original singularity error is fixed and that image is no longer in use. Any fursther singularity issues

However, I have:

• finished all container migrations so try deleting your $NXF_SINGULARITY_CACHEDIR, it may help and then making sure you are on the dp24_spelling branch. It may even be useful to create a new $NXF_SINGULARITY_CACHEDIR simply for testing.
• I think both of you are running using the local.executor rather than an infrastructure specific one, e.g., SLURM, LSF and I did find a bug in which the pipeline will fail with a 'requested cores = 2, avail = 1'. This has been mitigated in tests, however will require you to create a small config containing:

executor.cpus = 50
executor.memory = 50.GB
executor.time = 1.h

With values replaced with those appropriate for you.