Add arguments to make_zim_file and relax dependencies constraints

[benoit74]

Changes

• Add support for disable_metadata_checks, ignore_duplicates and compression arguments in make_zim_file function ("zimwritefs-mode")
  • We forgot about this disable_metadata_checks when adding this functionality ... while it is the first (and maybe even main) use case ...
    • This is a minor fix, but mandatory for migration of TED scraper to Zimscraperlib 3.x
  • We should support as well ignore_duplicates in make_zim_file function ("zimwritefs-mode")
    • It will default to True, since it most cases this is what we want (we usually do not run a deduplication function before-hand on files stored in the filesystem
    • But we could (should ?) do it add replace these duplicates by aliases / redirections
• Relax dependencies versions
  • Tests are running fine, this is especially important for lxml + beautifulsoup4 where some scraper might need features from a more recent version and usage in zimscraperlib is so limited that it probably doesn't hurt to relax constraints (at least all tests are green)
• Upgrade dev/qa dependencies

I intend to release this as 3.3.2 as soon as it is merged since we want to proceed with TED upgrade to benefit from new encoder presets.

[codecov]

Codecov Report

All modified and coverable lines are covered by tests ✅

Project coverage is 100.00%. Comparing base (b479038) to head (d18275c).

Additional details and impacted files

@@            Coverage Diff            @@
##              main      #147   +/-   ##
=========================================
  Coverage   100.00%   100.00%           
=========================================
  Files           32        32           
  Lines         1399      1399           
  Branches       240       240           
=========================================
  Hits          1399      1399           

☔ View full report in Codecov by Sentry.
📢 Have feedback on the report? Share it here.

[benoit74]

Nota: this has already been tested locally to work fine with ted scraper

[rgaudin]

Let me know if there's a reason behind those additions or if you were just filling a gap

[benoit74]

I was just filling a gap, since it was intentional I will remove this right now.

[benoit74]

Change done and first comment updated to reflect the new PR content.

[rgaudin]

Why did you remove the ignore_duplicates? We do want this one. Especially on old scrapers still using this to bundle assets

[benoit74]

Arf, I added two args to fill the gap and you made two remarks, so I misread them as being linked to these two args. I did not realized your first remark was about the workaround_nocancel which was already there before this PR. Let's discuss it live to be sure we are aligned

[benoit74]

After discussion:

• we keep workaround_nocancel as-is
• we add ignore_duplicates which is always True (not exposed)

After some thought, I think that we have to expose ignore_duplicates, someone very cautious in his scraper could run a deduplication on the files before calling make_zim_file, add needed redirects / aliases after that, and expect the libzim to stop if it detects a duplicate.

[rgaudin]

👍 ; as long as it defaults to True :)

[rgaudin]

But someone cautious would not be using the zimwriterfs mode 😅

[benoit74]

Does your 👍 means you've reviewed this PR and I can merge or that you've approved my proposition?

[rgaudin]

Yes