Upgrade browsertrix crawler and remove redirect handling

[benoit74]

Fix #256
Fix #284
Fix #166

This PR adopts browsertrix crawler 1.0.0-beta5 1.0.0-beta.6.

Among other things, this release now handles nicely redirect (webrecorder/browsertrix-crawler#476).

We hence have to remove the handling we've previously done on our side and caused issues (#256). We just keep the cleaning of the URL (remove default ports 443 and 80).

As a side-effect, this will also solve #166 since browsertrix crawler is already permissive in terms of SSL certificates issues. The only SSL issues which will continue to be blocked are the ones where the browser cannot establish at all the connection, like https://panzer-war.com/ were the browser has no cipher in common with the server

Redirect handling has been tested with https://metafilter.com:

docker run -v $PWD/output:/output:rw --name zimit2_test--rm local-zimit:zimit2 zimit --limit 10 --adminEmail="contact+zimfarm@kiwix.org" --description="Test" --lang="en" --name="metafilter.com_en_all" --output="/output" --publisher="openZIM" --scopeType="prefix" --statsFilename="/output/task_progress.json" --title="Metafilter" --url="https://metafilter.com:443" --verbose

Handling of insecure connection withhttps://www.moneyinstructor.com (which still fails without the simplification of check_url):

docker run -v $PWD/output:/output:rw --name zimit2_test --rm local-zimit:zimit2 zimit --limit 10 --adminEmail="contact+zimfarm@kiwix.org" --description="Test" --lang="en" --name="www.moneyinstructor.com_en_all" --output="/output" --publisher="openZIM" --scopeType="prefix" --statsFilename="/output/task_progress.json" --title="MoneyInstructor" --url="https://www.moneyinstructor.com/" --verbose

This PR should not be merged before openzim/warc2zim#196

[codecov]

Codecov Report

Attention: Patch coverage is 16.66667% with 5 lines in your changes are missing coverage. Please review.

Project coverage is 14.91%. Comparing base (857ae56) to head (5c71674).
Report is 1 commits behind head on zimit2.

Files	Patch %	Lines
src/zimit/zimit.py	16.66%	5 Missing ⚠️

Additional details and impacted files

@@            Coverage Diff             @@
##           zimit2     #285      +/-   ##
==========================================
+ Coverage   14.88%   14.91%   +0.03%     
==========================================
  Files           1        1              
  Lines         262      248      -14     
  Branches       38       35       -3     
==========================================
- Hits           39       37       -2     
+ Misses        223      211      -12     

☔ View full report in Codecov by Sentry.
📢 Have feedback on the report? Share it here.

[benoit74]

"Luckily", tests are failing due to openzim/warc2zim#198 (but even once this is merged, we still need to wait for openzim/warc2zim#196)

[benoit74]

@mgautierfr I did not asked you for a formal review of this since as far as I've understood you are less experienced with zimit, but do not hesitate to have a look and comment as well

[benoit74]

I had to fix the tests by updating the number of expected WARC records from 8 to 7, because we do not have anymore the "weird / unexpected" https://dict.brave.com/edgedl/chrome/dict/en-us-10-1.bdic in the WARC anymore (item #3 below)

Before:

WARC item	Comment
response https://isago.rskg.org/	1
request https://isago.rskg.org/	not included (request)
response https://isago.rskg.org/static/favicon256.png	2
request https://isago.rskg.org/static/favicon256.png	not included (request)
response https://dict.brave.com/edgedl/chrome/dict/en-us-10-1.bdic	3
request https://dict.brave.com/edgedl/chrome/dict/en-us-10-1.bdic	not included (request)
response https://maxcdn.bootstrapcdn.com/bootstrap/4.0.0/css/bootstrap.min.css	4
request https://maxcdn.bootstrapcdn.com/bootstrap/4.0.0/css/bootstrap.min.css	not included (request)
response https://isago.rskg.org/static/tarifs-isago.pdf	5
request https://isago.rskg.org/static/tarifs-isago.pdf	not included (request)
response https://isago.rskg.org/conseils	6
request https://isago.rskg.org/conseils	not included (request)
response https://isago.rskg.org/a-propos	7
request https://isago.rskg.org/a-propos	not included (request)
response https://isago.rskg.org/faq	8
request https://isago.rskg.org/faq	not included (request)

After:

WARC item	Comment
response http://isago.rskg.org/	not included (redirect)
request http://isago.rskg.org/	not included (request)
response https://isago.rskg.org/	1
request https://isago.rskg.org/	not included (request)
response https://maxcdn.bootstrapcdn.com/bootstrap/4.0.0/css/bootstrap.min.css	4
request https://maxcdn.bootstrapcdn.com/bootstrap/4.0.0/css/bootstrap.min.css	not included (request)
response https://isago.rskg.org/static/favicon256.png	2
request https://isago.rskg.org/static/favicon256.png	not included (request)
resource urn:pageinfo:http://isago.rskg.org/	not included (resource)
response https://isago.rskg.org/conseils	6
request https://isago.rskg.org/conseils	not included (request)
resource urn:pageinfo:https://isago.rskg.org/conseils	not included (resource)
response https://isago.rskg.org/faq	8
request https://isago.rskg.org/faq	not included (request)
resource urn:pageinfo:https://isago.rskg.org/faq	not included (resource)
response https://isago.rskg.org/a-propos	7
request https://isago.rskg.org/a-propos	not included (request)
resource urn:pageinfo:https://isago.rskg.org/a-propos	not included (resource)
response https://isago.rskg.org/static/tarifs-isago.pdf	5
request https://isago.rskg.org/static/tarifs-isago.pdf	not included (request)
resource urn:pageinfo:https://isago.rskg.org/static/tarifs-isago.pdf	not included (resource)

[benoit74]

Review welcomed again, changing a test "to make it works" probably needs to be confirmed to be OK 🤣

[rgaudin]

LGTM ; but the commit (296b104) must include the relevant information (for future blame's sake): why we had and expected 8 before and why we have and expect 7 now.

[benoit74]

Done, commit updated.

[rgaudin]

👍