Method to perform finer-grained selection of ARCs and WARCs

[lintool]

We currently only have one method to load ARCs or WARCs:

RecordLoader.loadArchives("/path/to/many/warcs/*.gz", sc)

It'd be nice to have some fine grained control, e.g., I want all (W)ARCs starting with a prefix, except for A, B, and C - this would help us debug large collections that maybe errors/corruption/etc.

[borislin]

@lintool @ianmilligan1

Based on our discussion on Slack, we would like to limit the size of individual records instead of archive files. A lot of Spark jobs fail because of records that are too big.

[ianmilligan1]

Should this still be an open issue? I don't think we've been running into any ingestion issues lately, including on some very large collections.

[ruebot]

I think the work done here is worth revisiting in the future, since I believe it got at the spirit of @lintool original post:

It'd be nice to have some fine grained control, e.g., I want all (W)ARCs starting with a prefix, except for A, B, and C - this would help us debug large collections that maybe errors/corruption/etc.

But, I'll leave it up to him to close or keep it open.

[ianmilligan1]

Sounds good, thanks @lintool @ruebot !

[lintool]

Please leave open.

[ruebot]

The spirit of this issue being identifying problematic W/ARCs will be solved with #533, so I'm going to close this once it is merged.

The additions to loadArchives here still might be worth bringing in if there is a demand for it in the future.