Fix detection of encoding again

[benoit74]

Fix #312

Changes

• first try to find charset in document first 1024 bytes with the ascii or UTF-16 or UTF-32 encoding and regex ; if found, decode with this encoding in 'replace' mode (always going to work, do not care if encoding specified was wrong, not our fault)
  • NOTA: or UTF-16 or UTF-32 is a slight change compared to issue proposal because with UTF-16 and UTF-32 first bytes cannot be decoded to ascii properly (they always use at least 16 bytes or 32 bytes even for ascii characters)
• if no charset in found in document first 1024 bytes, search for charset in HTTP Content-Type header ; if found, decode with this encoding in 'replace' mode (always going to work, do not care if encoding specified was wrong, not our fault)
• if no charset found in document or HTTP header, we will have to guess a little bit (situation is quite common for JS and CSS); try UTF-8 in 'strict' mode ; if it fails, try ISO-8859-1 in 'strict' mode ; if it fails, stop the scraper (we cannot guess encoding when not specified in document or HTTP header) ; this list of charset to try should even probably be exposed as a scraper parameter --guessing-charsets so that it could be possible when needed to tweak this list when needed

[codecov]

Codecov Report

Attention: Patch coverage is 93.75000% with 1 line in your changes missing coverage. Please review.

Project coverage is 84.49%. Comparing base (4c12681) to head (9933304).

❗ Current head 9933304 differs from pull request most recent head b1c8a35

Please upload reports for the commit b1c8a35 to get more accurate results.

Files	Patch %	Lines
src/warc2zim/utils.py	91.66%	1 Missing ⚠️

Additional details and impacted files

@@            Coverage Diff             @@
##             main     #314      +/-   ##
==========================================
+ Coverage   84.06%   84.49%   +0.42%     
==========================================
  Files          14       14              
  Lines        1268     1238      -30     
  Branches      249      245       -4     
==========================================
- Hits         1066     1046      -20     
+ Misses        155      149       -6     
+ Partials       47       43       -4     

☔ View full report in Codecov by Sentry.
📢 Have feedback on the report? Share it here.

[benoit74]

CodeFactor is complaining about files which should be ignored since they are outside our scope, these are sample files from online websites.

[rgaudin]

Can you add exceptions then? Either inline, via a config file or via Codefactor UI

[rgaudin]

Can we rename guessed_charsets with charsets_to_try ?

Good luck with Codefactor 😀

[benoit74]

Can we rename guessed_charsets with charsets_to_try ?

Sure

Good luck with Codefactor 😀

I will kill it ^^ It worked once, I supposed my last modification was ok so I cleaned up everything I've left over, it's not working anymore, I rolled-back to what was working, it is not working anymore. But I will nail it ^^

[benoit74]

Codefactor issues its configuration from main branch, so you have to merge first to main before you can review the PR ...

[rgaudin]

Go ahead

[kelson42]

@benoit74 Bravo... and bon courage!

[benoit74]

Go ahead

I already pushed only the configuration to main branch. My comment was more for "the posterity". I still have to rewrite this branch to change arg name + simplify commits.

Bravo... and bon courage!

We will finish by nailing this down ^^