Encode headers as ASCII

[vojtechsimetka]

Problem

According to the RFC 7230, the headers should be encoded in ASCII (previously in iso-8859-1). Trying to send header in any other encoding may fail. Most browsers and nodejs allow UTF8 encoding as well but IMHO I would not rely on this. Most headers we use should be fine, but some are user defined e.g. index-document and error-document.

Reproduce

(please node the "—"in the index-document is a character from iso-8859-2)
swarm-cli upload --name test --stamp <STAMP> --index-document "test/02 — Mixed.png" test

ERROR Invalid character in header content ["swarm-index-document"]

or upload single file containing "—" in filename with bee-dashboard or gateway

Quick fix

Encode as URI (which is ASCII) and unescape the characters like space. unescape(encodeURIComponent(<HEADER>))

Proper fix

Encode headers as UTF8 and then convert them to ASCII (UTF8 is backwards compatible with ASCII). Please note that many solutions rely on buffer which is not something we want.

References

• fix in gateway fix: handle non iso-8859-1 in the index document name gateway-ui#200
• strings manipulator library which is maintained by ethersjs https://www.npmjs.com/package/@ethersproject/strings