encoding issue

[Adaptivity]

I'm trying to update russian translation to the latest PZ version but I run into a problem which I can't solve (I remember same issue was here year ago). So what issue is.
Currently all russian .txt files use windows 1251 encoding which can be read by game (text displayed normally). But for some reason files on github are broken (hieroglyphs all the way). I thought well maybe github can't display windows 1251 files but for some reason SOME files displayed normally and some aren't.

For comparison:

1. https://github.com/TheIndieStone/ProjectZomboidTranslations/blob/master/RU/Challenge_RU.txt this file uses windows 1251 encoding and displayed correctly
2. https://github.com/TheIndieStone/ProjectZomboidTranslations/blob/master/RU/ContextMenu_RU.txt this file uses same windows 1251 encoding but as you can see some crazy characters appear here.

So question is why this happens? How do I fix it? If you download these files they displayed normally but not in repository. Also, UTF-8 works for github but game won't display them although english files encoded in UTF-8 are displayed correctly. There are no problems in displaying russian symbols in minecraft for example in UTF-8, seems like zomboid needs different encoding for each language.

[Adaptivity]

Another interesting thing:
In my fork this file https://github.com/Adaptivity/ProjectZomboidTranslations/blob/updaterussianloc/RU/UI_RU.txt looks correctly but commit itself https://github.com/Adaptivity/ProjectZomboidTranslations/commit/d2c1d2b3d60b596717a16702c263c7d8315b3dd7 have broken symbols... this is so strange

[Connall]

The game does use different encoding for different files, due to variety of fonts that we use I believe. I'm honestly not entirely sure why GitHub sometimes has a problem and sometimes does not. However, since it's generally functioning it's something I have not spent much time looking into. I can maybe spend some time in the future figuring out what's going on. Just bigger issues at the moment, you know?

Edit: Though perhaps unifying the encodings would be a good idea to look into to stop file encoding breakages that can be a real pain.

[Adaptivity]

👍 for unifying encodings. This issue can wait, maybe someone more experienced in github can sort this out

[Connall]

I'll try some experiments in the coming weeks.

[Connall]

Was going to close, but maybe best to leave this open as a reminder.

[narrnika]

Just as a reminder that the issue is still relevant - yes, it would be nice to "combine encodings."
Most likely, this would also solve the issue with incorrect operation of functions print() and file:write() (writing in file question marks instead of national characters, which is especially noticeable when trying to save/load sandbox options or skills/appearance of character)

[Faalagorn]

IMHO it would be the best to use Unicode for everything as long as the engine allows it, and if not find out why. Unicode is de facto standard everywhere nowadays and the issue is only with really old machines and that usually can be solved by a software update. Moden Unicode encodings also support all kind of glyphs, including languages that are not yet supported by PZ probably but might be in future.

[Tiax]

After some experiments: if you add a .gitattributes file with

EN/*.txt text working-tree-encoding=utf-8
DE/*.txt text working-tree-encoding=cp1252
RU/*.txt text working-tree-encoding=cp1251
CN/*.txt text working-tree-encoding=utf-8
...

the Github changesets seem to work fine. You may need to re-encode them to unicode in the git repo but after that it will be fine in any working tree.

[Faalagorn]

After some experiments: if you add a .gitattributes file with

EN/*.txt text working-tree-encoding=utf-8
DE/*.txt text working-tree-encoding=cp1252
RU/*.txt text working-tree-encoding=cp1251
CN/*.txt text working-tree-encoding=utf-8
...

the Github changesets seem to work fine. You may need to re-encode them to unicode in the git repo but after that it will be fine in any working tree.

I didn't test it yet, but if I understand correctly, that would make files appear correctly in repo and in-game? If, so this would solve the problem we have without modification to the game itself, I'll see if that works and if I can make a PR for it. Thanks!

[Tiax]

• PR for changes
• Test commit with the changes applied to see on Github