Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Merging multiple databases not working #63

Open
Mariewelt opened this issue Oct 3, 2020 · 1 comment
Open

Merging multiple databases not working #63

Mariewelt opened this issue Oct 3, 2020 · 1 comment

Comments

@Mariewelt
Copy link

Hey @lorton

I tried to launch a search over a huge database and used gpusim_mergedb.py to process the database files in parallel. However, merging didn't work. It created empty files. After a little bit of digging into the problem, I found the cause.

The gpusim_createdb.py writes 4 values to the top of each database

    qds.writeInt(DATABASE_VERSION)
    qds.writeString(args.dbkey.encode())
    qds.writeInt(gpusim_utils.BITCOUNT)
    qds.writeInt(count)

However, gpusim_mergedb.py read (and then writes to a merged fsim file) only 3: everything except for dbkey. First reading the dbkey for each database and then writing it to a merged file solves the problem.

@Mariewelt Mariewelt changed the title Merging multiple databases into 1 not working Merging multiple databases not working Oct 3, 2020
@lorton
Copy link
Member

lorton commented Oct 4, 2020

Thank you very much @Mariewelt for doing this investigation! We don't have sufficient testing set up of merge_database.py obviously, so this was missed. Do you want to do a pull request of your changes?

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

No branches or pull requests

2 participants