New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

[PQ] Random starting id for dataset limit #90

Merged

var77 merged 5 commits into main from varik/pq-random-subset

Mar 1, 2024

Collaborator

var77 commented Feb 29, 2024

Select random starting offset id, if limit is specified.
Pass dataset size and offset id via args to not fetch on each task, this will save time
Add timing logs to external indexing

var77 added 5 commits

February 29, 2024 17:01


          Select random subset from dataset if limit is passed

6c6b3bb


          Add time logs for external indexing

d697c5f


          Fix pq tests

62352d8


          Pass dataset limit to gcp jobs

8de7d88


          Add allocation policy to gcp batch jobs

ac66fd9

var77 merged commit b77db72 into main

15 checks passed

var77 deleted the varik/pq-random-subset branch

March 1, 2024 13:55

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet