Display in DEBUG all the important ZIM metadata

[kelson42]

If I look to a log I see something like this:

[DEBUG] Confirming output is writable using /output/tmpda8c84ht
[DEBUG] Title: Personal Security Checklist
[DEBUG] Language: en
[DEBUG] Favicon: https://github.githubassets.com/favicons/favicon.png
[DEBUG] Found WARC record for favicon: https://github.githubassets.com/favicons/favicon.png

I would like to list here all the important ZIM metadata, in particular the Description and the LongDescription which might just after make the whole process die (if bigger than the maximal size). See https://wiki.openzim.org/wiki/Metadata for the whole list of ZIM metadata.

[richterdavid]

Should this issue be moved to the repo for warc2zim?

"Confirming output is writable using" is at

warc2zim/src/warc2zim/main.py

Line 267 in 367844d

logger.debug(f"Confirming output is writable using {fh.name}")

"Found WARC record for favicon" is at

warc2zim/src/warc2zim/main.py

Line 563 in 367844d

logger.debug(f"Found WARC record for favicon: {self.favicon_url}")

[kelson42]

@richterdavid Yes, it is something to implement in warc2zim.

[richterdavid]

I'd like to work on this.

[benoit74]

I don't think this issue is relevant anymore, at least not as it is phrased / oriented today.

What has already been decided for all scrapers is that we must use the validate_metadata methods from zimscraperlib to fail the scraper early when metadata used are incorrect: this displays the offending metadata value.

This still has to be implemented in warc2zim, I've just opened #123

Do we consider it would still help to display all metadata once validated, e.g. for the case where the validation checks are improperly implemented? If yes, this would anyway be better to do this in python-scraperlib so that all scrapers benefit from this enhancement.

[benoit74]

Oups, I'm wrong, the validation does not display the offending value. Maybe we can consider to also display the offending value.

[richterdavid]

This still has to be implemented in warc2zim, I've just opened #123

That link goes to this issue; which issue did you mean @benoit74 ?

[kelson42]

Oups, I'm wrong, the validation does not display the offending value. Maybe we can consider to also display the offending value.

Yes, this is so important!

[kelson42]

This still has to be implemented in warc2zim, I've just opened #123

I guess this is the wrong issue number

Do we consider it would still help to display all metadata once validated, e.g. for the case where the validation checks are improperly implemented? If yes, this would anyway be better to do this in python-scraperlib so that all scrapers benefit from this enhancement.

Yes and yes

[benoit74]

This still has to be implemented in warc2zim, I've just opened #123

Yes, proper issue is #235

Yes and yes

Closing this issue then, we will implement the necessary in python scraperlib and next upgrade of the dependency in warc2zim will deploy it "automatically", I don't think we need to track this here.

[benoit74]

I've just opened it: openzim/python-scraperlib#155

[kelson42]

@benoit74 Few points regarding the process

• If Validate metadata as early as possible #235 is the new home for this issue, then this ticket is a "duplicate" and IMHO should not be closed as "won't be implemented" (but this last point is "difficult")
• Validate metadata as early as possible #235 is not the same request as what has been requested here, my guess is that 50% chances it will get forgotten (in particular if you don't do it yourself)
• if Validate metadata as early as possible #235 is actually for scraperlib, to apply to all scrapers, the i would recommend to move it to the proper repository.

[kelson42]

Last comment mostly invalidated by newly created ticket at scraperlib. @benoit74 thx

[benoit74]

Last comment mostly invalidated by newly created ticket at scraperlib. @benoit74 thx

Next time I will open the ticket before closing the other one ^^

#235 is indeed a different thing. I will most probably implement it, but it is not necessary. It is not urgent because we anyway validate metadata now with recent scraperlib, the goal of 235 is to do it as early as possible, i.e. enhance current behavior where validation is done a bit late since it is not done in the "early check" done by zimit but after the crawl and after some processing of warc2zim.