LookupError: unknown encoding: unicode

[rgaudin]

Not sure if still valid (we merged related stuff last week I think) but this zimit.kiwix.org run failed in rewrite

Traceback (most recent call last):
  File "/usr/bin/zimit", line 8, in <module>
    sys.exit(zimit.zimit())
             ^^^^^^^^^^^^^
  File "/app/zimit/lib/python3.12/site-packages/zimit/zimit.py", line 585, in zimit
    run(sys.argv[1:])
  File "/app/zimit/lib/python3.12/site-packages/zimit/zimit.py", line 507, in run
    return warc2zim(warc2zim_args)
           ^^^^^^^^^^^^^^^^^^^^^^^
  File "/app/zimit/lib/python3.12/site-packages/warc2zim/main.py", line 146, in main
    return converter.run()
           ^^^^^^^^^^^^^^^
  File "/app/zimit/lib/python3.12/site-packages/warc2zim/converter.py", line 330, in run
    self.add_items_for_warc_record(record)
  File "/app/zimit/lib/python3.12/site-packages/warc2zim/converter.py", line 748, in add_items_for_warc_record
    payload_item = WARCPayloadItem(
                   ^^^^^^^^^^^^^^^^
  File "/app/zimit/lib/python3.12/site-packages/warc2zim/items.py", line 56, in __init__
    ).rewrite(pre_head_template, post_head_template)
      ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/app/zimit/lib/python3.12/site-packages/warc2zim/content_rewriting/generic.py", line 108, in rewrite
    return self.rewrite_html(pre_head_template, post_head_template)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/app/zimit/lib/python3.12/site-packages/warc2zim/content_rewriting/generic.py", line 247, in rewrite_html
    ).rewrite(self.content_str)
              ^^^^^^^^^^^^^^^^
  File "/app/zimit/lib/python3.12/site-packages/warc2zim/content_rewriting/generic.py", line 93, in content_str
    return to_string(
           ^^^^^^^^^^
  File "/app/zimit/lib/python3.12/site-packages/warc2zim/utils.py", line 175, in to_string
    return input_.decode(head_encoding, errors="replace")
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
LookupError: unknown encoding: unicode

[rgaudin]

And https://farm.zimit.kiwix.org/pipeline/2c1770f9-1281-48f0-ab8a-4bb2a727552b/debug

[benoit74]

Thank you!

[benoit74]

Same problem with iso-utf-8 on https://farm.openzim.org/pipeline/d1fa0c7a-29c2-4229-80cf-686aba6ac0f5/debug

Note that on https://farm.zimit.kiwix.org/pipeline/2c1770f9-1281-48f0-ab8a-4bb2a727552b/debug the problem was with iso-8559-1 (this is a typo, encoding is most probably iso-8859-1)

[benoit74]

So problematic pages are https://www.qsl.net/emporiaars/newsletter.html, https://marxists.incn.su/history/etol/writers/goldman/1936/09/campaign.html and https://www.qsl.net/vk2jem/swlogs.htm

[benoit74]

Same kind of problem on page http://api.map.baidu.com/mapCard/js/finish/scriptURL.js where the charset is badly found with regex in JS code.

[benoit74]

Again on https://openai.com/_next/static/chunks/8492.86a146b3037613c7.js

[benoit74]

Again with https://educacionenquimica.com.ar/index.php/edenlaq/article/download/76/135?inline=1 (unicode)

[benoit74]

Again with https://cdn.cookielaw.org/consent/796524c5-24c2-4cdf-907c-573192ba6a9d/otSDKStub.js (this), bad regex in JS code

[benoit74]

And

• https://mi_shell.gitlab.io/models_online/ose_webgl/js/three.min.js (JS problem also)
• https://wiki.opensourceecology.org/wiki/Main_Page (JS problem also)
• https://vercel.com/_next/static/chunks/83266-67bf9ae568a2b318.js?dpl=dpl_2hmfJkPd9Zdg3JRu5ruKqVazZ59j (JS problem also)
• https://tertullian.org/french/adversus_omnes_haereses.htm (windows-utf-8 in HTML document)
• https://www.answering-islam.org/Thai/index.html (windows-874 in HTML document)

[benoit74]

It looks like there is two problems in one error log:

• the code trying to guess the charset is too permissive and grab a significant number of false positives in JS documents ; is there really a reason to run this regex on non-HTML documents? AFAIK, only HTML has a spec to declare charset in its header ; I suggest to detect charset in document header with regex only on HTML documents
• there is a variety of websites which provide wrong encodings, or at least encodings unknown in Python (unicode, windows-utf-8, windows-874, iso-8559-1, iso-utf-8) ; add automated support for these encodings is going to be cumbersome, because we never really know what is correct to use. I suggest that we simply add yet another parameter to configure additional (Python already has many) encoding aliases (unicode=utf-8,iso-8559-1=iso-8859-1,windows-874=cp874)