Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

I am encountering an ANE evaluation error when running Dorado on macOS. #1087

Open
GanchimegNamuunbayar opened this issue Oct 17, 2024 · 12 comments

Comments

@GanchimegNamuunbayar
Copy link

GanchimegNamuunbayar commented Oct 17, 2024

Issue Report

Please describe the issue:

Please provide a clear and concise description of the issue you are seeing and the result you expect.

Steps to reproduce the issue:

Please list any steps to reproduce the issue.

Run environment:

  • Dorado version: 0.8.0
  • Dorado command: dorado basecaller --batchsize 16 sup,4mC_5mC,6mA ~/desktop/AA/pod5/ -r > ~/desktop/AA/bam/calls.bam
  • Operating system: MacOS Sonoma 14.0
  • Hardware (CPUs, Memory, GPUs): M2 Max, Memory 64GB
  • Source data type (e.g., pod5 or fast5 - please note we always recommend converting to pod5 for optimal basecalling performance): multiple pod 5 files
  • Source data location (on device or networked drive - NFS, etc.): on device
  • Details about data (flow cell, kit, read lengths, number of reads, total dataset size in MB/GB/TB): totaling 14GB (each 800MB to 4.5GB)
  • Dataset to reproduce, if applicable (small subset of data to share as a pod5 to reproduce the issue):

Logs

  • Please provide output trace of dorado (run dorado with -v, or -vv on a small subset)
    I was able to run successfully with a single pod5 file of 850MB.
    command: dorado basecaller --batchsize 32 sup, 4mC_5mC, 6mA ~/desktop/pod5/ > ~/desktop/calls.bam
@HalfPhoton
Copy link
Collaborator

Hi @GanchimegNamuunbayar,
Can you add logs please?

@GanchimegNamuunbayar
Copy link
Author

GanchimegNamuunbayar commented Oct 21, 2024

@HalfPhoton Thank you for your response. I have added logs to the issue report. Could you please reference the updated version.

@HalfPhoton
Copy link
Collaborator

You've only added the version information here - The -vv flags must be placed after the sub-command: dorado basecaller -vv to generate detailed logs.

Kind regards,
Rich

@GanchimegNamuunbayar
Copy link
Author

GanchimegNamuunbayar commented Oct 23, 2024

@HalfPhoton Thank you for the details.
I have tried it as you said
command: dorado basecaller -vv --batchsize 16 sup,4mC_5mC,6mA ~/desktop/pod5/ -r > ~/desktop/bam/calls.bam

These are the last lines before the computer shut down:
[2024-10-22 17:42:51.715] [trace] Checking adapter/primer PCR_PSK_rev1
[2024-10-22 17:42:51.715] [trace] Checking adapter/primer PCR_PSK_rev2
[2024-10-22 17:42:51.715] [trace] Checking adapter/primer cDNA_VNP
[2024-10-22 17:42:51.715] [trace] Checking adapter/primer cDNA_SSP
[2024-10-22 17:42:51.715] [trace] Checking adapter/primer PCS110_forward
[2024-10-22 17:42:51.715] [trace] Checking adapter/primer PCS110_reverse
[2024-10-22 17:42:51.715] [trace] Checking adapter/primer RAD

@HalfPhoton
Copy link
Collaborator

Could you please write the logs to a file and share them? These last few lines aren't enough for us to see the issue.

dorado basecaller -vv --batchsize 16 sup,4mC_5mC,6mA ~/desktop/pod5/ -r > ~/desktop/bam/calls.bam 2> dorado.trace

Kind regards,
Rich

@HalfPhoton
Copy link
Collaborator

Does you issue persist with dorado 0.8.2? It has a number of bug fixes which might be causing your crash.

@GanchimegNamuunbayar
Copy link
Author

GanchimegNamuunbayar commented Oct 29, 2024

@HalfPhoton Thank you for your previous advice. I attempted to install a newer version, but the issue remains unresolved. The command for the logs executes successfully on smaller or fewer files; however, when processing larger datasets (~13GB), the system crashes and restarts due to a panic. I've included the trace file in a Google Drive link for your review, as I couldn't attach it here directly.
Logs

@HalfPhoton
Copy link
Collaborator

Hi @GanchimegNamuunbayar,

For some reason I can't view these logs, can you create a ticket on the ONT customer support page please?
We can then get these large log files and help you from there.

Please reference this GitHub issue and the great people at customer support will be able to get this information to me.

Kind regards,
Rich

@malton-ont
Copy link
Collaborator

@HalfPhoton I've fixed the link for @GanchimegNamuunbayar, logs should be accessible now.

@GanchimegNamuunbayar
Copy link
Author

Hi @HalfPhoton ,
I hope you're doing well! It's been a while since I shared the logs, and I just wanted to check in and see if there have been any updates or if you’ve had a chance to review them. Looking forward to your reply.
Thank you again for your support!

Also thanks to @malton-ont for kindly fixing the link earlier—really appreciate the help!!

Best,
Ganchimeg

@malton-ont
Copy link
Collaborator

Hi @GanchimegNamuunbayar,

dorado-0.8.3 was released yesterday and contains a couple of fixes for SUP basecalling on Apple silicon - could you give this a try?

@GanchimegNamuunbayar
Copy link
Author

@malton-ont
I tried it, but the error persists. Are there others facing the same problem? Could it be related to something in my environment or command?

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

No branches or pull requests

3 participants