Remove Java w/arc processing, and replace it with Sparkling.

[ruebot]

GitHub issue(s):

• Add domain column to webpages() #534
• Discard date RDD filter only takes a single string, not a list of strings. #532
• Replace Java ARC/WARC record processing library #494
• Extract gzip data from transfer-encoded WARC #493
• ARC reader string vs int error on record length #492
• Improve ArchiveRecord.scala test coverage #260
• Method to perform finer-grained selection of ARCs and WARCs #247
• Unit testing for RecordLoader #182
• Improve ArchiveRecordWritable.java test coverage #76
• Improve WarcRecordUtils.java test coverage #74
• Improve ArcRecordUtils.java test coverage #73
• Remove org.apache.commons.httpclient #23
• Unnecessary buffer copying #18

What does this Pull Request do?

Removes legacy Java w/arc processing and replaces it with Sparkling. There is no longer any Java code in the project.

In addition, a couple other issues have been resolved:

• fix discardDate issue Discard date RDD filter only takes a single string, not a list of strings. #532
• update tests for Replace Java ARC/WARC record processing library #494 (resolved with Sparkling integration)
• add test for Extract gzip data from transfer-encoded WARC #493
• add test for Discard date RDD filter only takes a single string, not a list of strings. #532
• move issue specific tests to their own directory
• updates GitHub actions

How should this be tested?

• GitHub actions
• I'm also going to be doing some exhaustive testing on tuna/orca with some of the legacy issues, especially Heap space!! java.lang.OutOfMemoryError: Java heap space at java.util.Arrays.copyOf(Arrays.java:3332) #317. I'll report back here when I'm done.

Additional Notes:

This work is co-authored by @helgeho. I'm very grateful for his help on this one, and also very grateful for Sparkling being open sourced now.

Also of note, Sparkling is being pull in by the latest commit now using Jitpack. We might want to pin it to a specific commit hash so things are stable there.

[codecov]

Codecov Report

Merging #533 (e75d1b1) into main (8104a65) will increase coverage by 3.90%.
The diff coverage is 86.81%.

@@             Coverage Diff              @@
##               main     #533      +/-   ##
============================================
+ Coverage     88.83%   92.74%   +3.90%     
+ Complexity       57       42      -15     
============================================
  Files            43       39       -4     
  Lines          1012      813     -199     
  Branches         85       52      -33     
============================================
- Hits            899      754     -145     
+ Misses           74       35      -39     
+ Partials         39       24      -15     

[ruebot]

I'm fairly certain this is an apples to apples comparison, and if it is, if looks like this PR speeds things up slightly:

Other thing to note here, is that in the previous benchmark test (0.50.1), we were giving Spark 500G of RAM. In this test, I only gave Spark 8G of RAM. So, that's a SIGNIFICANT improvement 😉

[ruebot]

I'm going to do a couple more tests, and if they're good to go, I'm going to squash and merge this.

[ruebot]

PySpark testing is good: archivesunleashed/notebooks@969fbea

BAnQ problem collection is the last one left!

[ruebot]

Good news, bad news.

Bad news: The BAnQ collection surfaced the #317 errors still. At the end of the day, that's because we're still reading records into memory with getBinaryBytes. If it's a big record, and malformed record, we're going to overflow RAM, and we can't really get around that.

Good news: We solved this in ARCH with streaming. It shouldn't be that difficult for me to further modify what we have here with Sparkling to pivot to streaming. In addition, we'll need to update the matchbox utilities to compensate for streaming as well. We did this in ARCH, so again, shouldn't be too difficult.

So, I'm going to propose that we squash and merge what we have here since it's getting a little out of control with the number of issues we're fixing. I've taken #317 out of the resolved column, and will leave that open. Let me know if you're good with that @ianmilligan1

[ruebot]

Forgot to add something; we could keep the getBinaryBytes method, but we'd need to limit or discard records over a given size. I don't like this. It make the archivist/scientist in me uncomfortable given what aut is supposed to do.

[ianmilligan1]

So, I'm going to propose that we squash and merge what we have here since it's getting a little out of control with the number of issues we're fixing. I've taken #317 out of the resolved column, and will leave that open. Let me know if you're good with that @ianmilligan1

Makes a lot of sense to me, @ruebot - happy to test + merge this PR once you're good to go with it.

Forgot to add something; we could keep the getBinaryBytes method, but we'd need to limit or discard records over a given size. I don't like this. It make the archivist/scientist in me uncomfortable given what aut is supposed to do.

Agreed - esp. given the jobs we have around binary files - would be misleading to say get a dump of video records that excludes the biggest ones or something.

[ruebot]

Yeah, if you want to give it some tests, please do! If you want to merge it as well, let me know and we can coordinate the commit message like we've done in the past.

[ianmilligan1]

Noted in Slack, but builds nicely and more importantly it works. I noticed how clean the text extractions are compared to earlier ones - way less noise. Kudos and congratulations! 🎉🎉🎉