-
Notifications
You must be signed in to change notification settings - Fork 4
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Error when use fastmultigather against rocksdb (Error: No such file or directory (os error 2) - Tested with multiple cases) #381
Comments
Thanks @tnmquann for this very detailed issue! Looking into it now. First question, which is unrelated to the problem you're experiencing, I think, but I wanted to ask - you shouldn't need to unzip
directly, without the unzip and the use of |
OK, I can replicate the problem with In brief,
|
Hi @ctb
So I have another workaround for this problem: I decompress the resulting file from the manysketch module and "try" to reconstruct the *.sig.zip files individually (it seems this tool uses the module multisearch to run for each sample separately). The results show that this workaround is quite good :D. The only problem I'm facing is that this combination takes a lot of time (that's why I want to use the fastmultigather module with rocksdb to decrease data processing time). I'd be happy to discuss further if you have any questions. |
thanks! no, that all makes sense. And we should talk to the YACHT authors (with whom we are quite friendly ;)) about updating their code! |
This was rapidly turning into a heisenbug for me, so I brute-forced it and wrote a script to explore - tl;dr RocksDB indexes built from .zip files FAIL when referenced from other directories, while RocksDB indexes built from lists of files work fine! (@bluegenes may owe me a drink because it was so hard to nail down this problem!)
|
A more succinct version
|
OK, it looks like by default the rocksdb does not store the sketches internally, and what is happening is that the path to the zip file containing sketches is being interpreted problematically. 🤿 time. |
hi @tnmquann this led in some interesting directions 😅 The problem is described in detail here, #415, but not resolved. See PRs #390 and #416 for better default behavior and improved documentation. If it's OK, I'm going to close this issue (since we now know what's going on) in favor of #415, which describes it without (yet) fixing it. |
Hi @ctb ,
Currently I'm using these commands:
Prepare data
Solution 1 & 2: Work perfectly
Solution 3: fastmultigather with rocksdb
Solution 3.1 : Use the path to the indexed database
Solution 3.2: Copy indexed database into the processing folder and then run the commands
Solution 3.3: Base on @ctb ‘s suggestion
Solution 3.4: Base on @bluegenes 's suggestion
I think there's a problem with the RocksDB folder configuration when running the index command.
The text was updated successfully, but these errors were encountered: