You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
Backfill with >1 concurrency will seemingly just miss all the links (or some other data type) for a fid(s) but get all casts, reactions, etc from that user.
The user(s) and data type(s) it skips are inconsistent and there are no errors in the logs. It's as if only part of the getFullProfileFromHub() function is running, or Postgres is somehow missing requests.
Verified by running a backfill with WORKER_CONCURRENCY=5, then running it again with WORKER_CONCURRENCY=1 and seeing that some tables had rows with newer updated_at values even though the message timestamp was old.
Thanks to @stephancill for finding the bug by comparing the number of rows in different tables after backfills with different concurrency values.
The text was updated successfully, but these errors were encountered:
Backfill with >1 concurrency will seemingly just miss all the links (or some other data type) for a fid(s) but get all casts, reactions, etc from that user.
The user(s) and data type(s) it skips are inconsistent and there are no errors in the logs. It's as if only part of the
getFullProfileFromHub()
function is running, or Postgres is somehow missing requests.Verified by running a backfill with
WORKER_CONCURRENCY=5
, then running it again withWORKER_CONCURRENCY=1
and seeing that some tables had rows with newerupdated_at
values even though the message timestamp was old.Thanks to @stephancill for finding the bug by comparing the number of rows in different tables after backfills with different concurrency values.
The text was updated successfully, but these errors were encountered: