v0.8.26

Update warc.go

v0.8.25

Don't write "ERROR" for payload digest in case of error (#29)

Return an error if there's an issue generating the payload digest

I'm genuinely unsure how we've reached this in the past, but this should prevent it from every showing up again.

Co-authored-by: Jake LaFountain <[email protected]>

v0.8.24

Add counter for the total amount of data crawled since startup (#28)

Add: counter for the total amount of data crawled since startup

v0.8.23

Fix proxy implementation (#27)

* WIP proxy test

* Fix TestHTTPClientWithProxy

* Fix proxy implementation

* Fix: remove WARC-IP-Address when being used with proxies due to us connecting to the "proxy" ip address, rather than the actual host.

Fix goleak.IgnoreTopFunction

Co-authored-by: Jake LaFountain <[email protected]>

v0.8.22

Update uTLS to the latest version with certificate compression (#26)

Update uTLS to the latest version with certificate compression.

Co-authored-by: Jake LaFountain <[email protected]>

v0.8.21

What's Changed

• Changing our TLS fingerprint with uTLS by @NGTmeaty in #25

Full Changelog: v0.8.20...v0.8.21

v0.8.20

What's Changed

• Add benchmark and calculate Block-Digest on dialer.go by @NGTmeaty in #22
• Add optimized, parallel gzip for payloads over 1MB and parallel WARC writing to multiple different WARC files by @CorentinB in #23

Full Changelog: v0.8.19...v0.8.20

v0.8.19

Move to klauspost/compress/gzip (#21)

v0.8.18

Add: custom struct to add count to WaitGroup

v0.8.17

Properly check error when removing temp file