Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Restore pre-fetching state during restore of UrlDBFunction state #121

Open
kkrugler opened this issue Mar 23, 2018 · 0 comments
Open

Restore pre-fetching state during restore of UrlDBFunction state #121

kkrugler opened this issue Mar 23, 2018 · 0 comments
Assignees
@kkrugler
Labels
bug

Comments

@kkrugler
Copy link
Member

kkrugler commented Mar 23, 2018 •
edited
Loading

Since anything inflight is assumed lost, we need to restore state for any URLs with state == FETCHING to what it was before.

We already keep the previous state around in the CrawlStateUrl, but we have to ensure that when we change the state from QUEUED to FETCHING, we don't lose the previous (pre-queued) state.
@kkrugler kkrugler self-assigned this Mar 23, 2018
@kkrugler kkrugler added the bug label Jul 18, 2018
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees

@kkrugler kkrugler

Labels
bug
Projects
None yet
Milestone
No milestone
Development

No branches or pull requests
1 participant
@kkrugler