Feat: cheerio experimental queue #188
Conversation
|
Let's keep this non-merged for some time. |
| } | ||
|
|
||
| await Promise.all(orderedKeys.map(({ key }) => { | ||
| return storeClient.deleteRecord(key); |
There was a problem hiding this comment.
This could be dangerous if you crash right after this delete, you loose everything. I would prefer to figure out how to overwrite properly.
| await this.persistQueueState(); | ||
| this.log.info('State persisted!'); | ||
| }); | ||
| } |
There was a problem hiding this comment.
I would probably make an option store on interval too. This would not persist anything in case of abrupt abort.
| @@ -0,0 +1,27 @@ | |||
| export default class Queue<T> { | |||
There was a problem hiding this comment.
Add a comment explaining where is this used, wasn't obvious at first
|
|
||
| for (let i = 0; i < arr.length; i += chunkSize) { | ||
| const chunk = arr.slice(i, i + chunkSize); | ||
| // Faster with got |
| method: 'PUT', | ||
| json: chunk, | ||
| http2: false, | ||
| }).catch((e: any) => this.log.exception(e.message, e.status))); |
There was a problem hiding this comment.
This looks like if we get rate limited or fail on 502 we just give up. I know this means reimplementing our client...perhaps there is a way to use the client in the end...
| this.log.info('State Loaded!', { requestCount: this.requests.size() }); | ||
| } | ||
|
|
||
| _stripRequest(requestLike: Request | RequestOptions) { |
There was a problem hiding this comment.
some comment here would be nice
| } | ||
|
|
||
| addRequest(requestLike: any) { | ||
| const uid = requestLike.uniqueKey ? requestLike.uniqueKey : requestLike.url; |
There was a problem hiding this comment.
We should use the request.id hashing function from crawlee. e.g. url is converted to uniqueKey (if not present) and then hashed to id (we can throw that away later). This way we replicate Crawlee and hashes are smaller to keep.
But not sure if it wouldn't have bad perf for when you need to refill after migration. If yes, please add comment that we decided to skip this
This is a POC version with minimal API and minimal request footprint.
Possible improvements:
large-setpackage to bypass nodeJs internal limit for Set