UTF-8, MIME Type Declaration, BOMs, and other encoding standards

[Martii]

This is going to be a very general issue ticket in order to discuss and address a few inconsistencies with node projects, including ours. And I'd like to hear some experiences encountered with our node project and any others.

DONE - UTF-8: All node projects should be using uppercase UTF-8 and not utf-8 according to the spec. When this is not consistent this can lead to unpredictable behavior between deployments whether on dev or production servers. Some book page websites may say it can be lowercase but some software components may not be smart enough to distinguish the difference.

DONE - Byte Order Mark (BOM) : BOMs are said to be not used everywhere... currently we have one in at least one file. This should be remedied on a system that can control that. This from my experience usually happens when there is a Unicode character inserted into a file and saved. Our current STYLEGUIDE hints at this with "Avoid use of international characters
because they may not read well or be understood everywhere.". Unfortunately I don't see an easier way to detect if a pr or commit is generating these or not.

In general encodings may need to be explicitly defined in contradiction to our current STYLEGUIDE saying the server handles it.

_EDIT_:
MIME types: These should always be included rather than having the server/client guessing off of file extensions.

See also:

• http://tools.ietf.org/html/rfc3629#section-4
• http://stackoverflow.com/questions/4990095/json-specification-and-usage-of-bom-charset-encoding
• Clean up #173 (comment) a BOM noticed here.

Applies to and isolated from #19. Most of this will go in either STYLEGUIDE and/or CONTRIBUTING

[sizzlemctwizzle]

Moved the UTF-8 bug hunting work to #200.

[Zren]

I just imported /ZrenTest/TestUserScript/folder/folder/%E3%83%86%E3%82%B9%E3%83%88.user.js. On dev, it redirects okay.

On Production it redirects to: https://openuserjs.org/scripts/zrentest/%C6%B9%C8 %C6%B9%C8
Instead of https://openuserjs.org/scripts/zrentest/%E3%83%86%E3%82%B9%E3%83%88 %E3%83%86%E3%82%B9%E3%83%88

The script imports fine.

[sizzlemctwizzle]

@Zren

That discussion has been moved to #200. And yes, I ran into the same problem myself (as has @Martii).

[Martii]

One thing I've noticed is there usually isn't a clear path of:

1. Input encoding and/or encoded
2. Working encoding and/or encoded
3. Output encoding and/or encoded

With a specific example... in our search routines Input is currently converted to a Working ISO 8859-1 "sort of" regular expression compatible syntax via deprecated escape and originally a plain ASCII regular expression match with some UTF-8 to match against the assumed UTF-8 data. e.g. Not much consistency here.

Ideally Working encoding should always be UTF-8 but we know regular expressions have some issues (as do some other support routines expecting something else)... which is why they aren't always reliable to use with variant Input. Another reason why I opted out of using a re with sanitizing certain linked @keys in another pr.

For the docs we should clarify for prs that all transformations like this should be commented briefly in the source as to what the I/W/O is and perhaps a variable name identifier prefix ... unless it's in HTML/XML where the tags could of course denote this.