How to fix UTF-8 encoding for the entire ENiGMA½ BBS system

[gatokichi]

I understand that ENiGMA½ BBS prioritizes ANSI display compatibility. However, in my country's language, switching between ANSI(cp437) and UTF-8 can cause characters, especially full-width ones, to be displayed incorrectly, resulting in garbled text. Even if I re-specify the encoding in the terminal, the display may not be corrected. I believe this issue also affects Chinese characters, Japanese Kana, Korean Hangul, and other non-alphabetic scripts.
Is there a way to disable the switching between ANSI and UTF-8 and fix the encoding to UTF-8 for the entire ENiGMA½ BBS system (for all users)?

{"name":"ENiGMA½ BBS","hostname":"host","pid":21238,"level":20,"encoding":"cp437","msg":"Terminal type changed to xterm-256color; Adjusting output encoding","time":"2025-01-26T03:41:53.625Z","v":0}
{"name":"ENiGMA½ BBS","hostname":"host","pid":21238,"level":30,"encoding":"utf8","currentEncoding":"cp437","msg":"Output encoding changed to utf8","time":"2025-01-26T03:41:53.626Z","v":0}
{"name":"ENiGMA½ BBS","hostname":"host","pid":21238,"nodeId":1,"sessionId":"IdIdIdIdId","level":20,"termType":"XTERM-256COLOR","msg":"Set terminal type","time":"2025-01-26T03:41:53.626Z","v":0}


{"name":"ENiGMA½ BBS","hostname":"host","pid":20955,"level":20,"encoding":"cp437","msg":"Terminal type changed to xterm-256color; Adjusting output encoding","time":"2025-01-26T02:19:20.760Z","v":0}
{"name":"ENiGMA½ BBS","hostname":"host","pid":20955,"level":30,"encoding":"utf8","currentEncoding":"cp437","msg":"Output encoding changed to utf8","time":"2025-01-26T02:19:20.762Z","v":0}
{"name":"ENiGMA½ BBS","hostname":"host","pid":20955,"nodeId":1,"sessionId":"ppUrKg6QrEK","level":30,"msg":"Terminal identified as UTF-8 but does not appear to be. Overriding to \"ansi\".","time":"2025-01-26T02:19:20.842Z","v":0}
{"name":"ENiGMA½ BBS","hostname":"host","pid":20955,"level":20,"encoding":"utf8","msg":"Terminal type changed to ansi; Adjusting output encoding","time":"2025-01-26T02:19:20.843Z","v":0}
{"name":"ENiGMA½ BBS","hostname":"host","pid":20955,"level":30,"encoding":"cp437","currentEncoding":"utf8","msg":"Output encoding changed to cp437","time":"2025-01-26T02:19:20.843Z","v":0}
{"name":"ENiGMA½ BBS","hostname":"host","pid":20955,"nodeId":1,"sessionId":"IdIdIdIdId","level":20,"termType":"ansi","msg":"Set terminal type","time":"2025-01-26T02:19:20.843Z","v":0}

[cognitivegears]

Hi @gatokichi - I believe that you should be able to do this today, though will take a bit of customization.

First you will need to save your art files in utf-8. Most of the modern ANSI art programs should be able to do this, I'm pretty sure IcyDraw can for example.

Then in the menus config use the "encodedAs" option to tell it that the art file is in utf-8 (unfortunately, I don't see that it is documented though it should be, but if you are comfortable looking at code, you can see more details at:

enigma-bbs/core/art.js

Line 89 in 6766f6e

// Convert from encodedAs -> j

).

After that, change your menu config to force the connection to utf-8 using the setClientEncoding systemMethod - see https://nuskooler.github.io/enigma-bbs/configuration/menu-hjson.html for info on that.

Hopefully, that should force everything into utf-8 and fix the issues you are seeing. Hope this helps!

[gatokichi]

@cognitivegears Thank you.
First, I will start by creating the art files in UTF-8.
I also want to make it possible to enforce UTF-8 in the menu config.
Since I'm not yet familiar with the ENiGMA½ BBS system, I think it will take some time, but I plan to solve the issues one by one.

[cognitivegears]

@cognitivegears Thank you.

First, I will start by creating the art files in UTF-8.

I also want to make it possible to enforce UTF-8 in the menu config.

Since I'm not yet familiar with the ENiGMA½ BBS system, I think it will take some time, but I plan to solve the issues one by one.

Sounds good - if I can suggest, maybe just start with the login screen and maybe one or two post login. Once you get the config working for that you can be sure of what's needed before doing for the rest of the art. Let us know if you run into any troubles!

[NuSkooler]

@gatokichi I love that Enig is being used in a non-Romanized language! As @cognitivegears says, you could force all menus to UTF-8, and if needed, also save all art as UTF-8.

Enigma will read all ANSI file as CP437 and convert to the clients encoding of UTF-8 if needed. Perhaps a small PR could set some global enforcement such as forceOututEncoding: utf8 or whatever.

If @cognitivegears' suggestions work for you, that will give some information as to what we could do here to make things easier.

[cognitivegears]

Exactly, that would be nice. I wonder if instead of a global enforcement, we could either/both of:

1. look for a utf-8 BOM (at least I think that should work, FEFF shouldn't be a valid CP437 sequence I believe,) or
2. use a file extension, in the art.js, SUPPORTED_ART_TYPES, some ideas: .ans8 / '.u8ansi, and .utf8instead of.asc` for plain utf-8 files, for example

Downside to 1) is that I'm not sure whether the modern ansi editors can save a BOM, and anyway BOM has gone out of style generally, so I'm inclined to just use extension version. The reason I'd think that instead of global enforcement is that I could see someone wanting to support both (a multi-lingual BBS for example, or as a second option for languages that have a romanized version and want to support either native encoding or the romanized for, say, old bbs terminal programs.)

EDIT: thinking some more, there is also an option 3) Create the forceOutputEncoding: utf8, but do it at the theme level, that way you could still support multiple - though I still like 1) and 2) better generally I think just for ease of use

[gatokichi]

The method of registering extensions for UTF-8 in the proposed SUPPORTED_ART_TYPES seemed appealing, but we decided not to adopt it this time since it is uncertain which one would be used depending on the user's environment.

Simply adding encodedAs: "utf-8" to /config/menus/*.hjson resolved the issue of garbled text in the menu screen.

exp: {
    ****: ****
    config {
        ****: ****
        encodedAs: "utf-8"
    }
}

However, when using double-width characters such as Chinese, Japanese, or Korean in ART files, the position of the characters following the control characters to the right shifts to the left. This can be prevented by positioning the text further to the right to compensate for the shift, but fixing it can be a bit troublesome.