The text encoding is different when fetching links

[a84r7a3rga76fg]

The text encoding is UCS-2 LE BOM when I fetch links, is there a special command or something I can use in the conf to make it UTF-8?

[mikf]

when I fetch links

You mean writing the results of -g to a file?
There is no option to set the output encoding in gallery-dl, but specifying an encoding using PYTHONIOENCODING might work.

[a84r7a3rga76fg]

Yes, I forgot to clarify it, with the -g command it'll use UCS-2 LE BOM but with any other command it'll use UTF-8.

[mikf]

I did a bit of testing. Even on Windows it uses UTF-8 as output encoding when writing to a terminal, but Python switches to some other encoding when redirecting stdout to a file (gallery-dl -g ... > file).

Setting the PYTHONIOENCODING environment variable lets you force Python's output encoding in all cases, even when doing the redirect. So just do set PYTHONIOENCODING=utf-8 before running gallery-dl or set it globally.

[Hrxn]

Maybe a bit shorter: You can also create an environment variable by using good old CMD:

SetX PYTHONIOENCODING utf-8

Adds it to the current user (HKCU in the Registry)

SetX PYTHONIOENCODING utf-8 /m

To set it machine-wide (HKLM)

You can also use PowerShell, of course.

User:

[Environment]::SetEnvironmentVariable('PYTHONIOENCODING', 'utf-8', 'User')

Machine:

[Environment]::SetEnvironmentVariable('PYTHONIOENCODING', 'utf-8', 'Machine')

Setting it system-wide (HKLM) requires administrative privileges, of course. Setting it for the user works in PowerShell, and should work with SetX as well

[a84r7a3rga76fg]

I tried those but it didn't work

[mikf]

Since commit e480a93 (v1.25.0), it is possible to force a specific encoding with the output.stdout etc options.

---

On Windows, Python seems to use utf-8 when outputting to a terminal and a locale specific encoding when its output is redirected to a file.

On Windows, UTF-8 is used for the console device. Non-character devices such as disk files and pipes use the system locale encoding (i.e. the ANSI codepage). Non-console character devices such as NUL (i.e. where isatty() returns True) use the value of the console input and output codepages at startup, respectively for stdin and stdout/stderr. This defaults to the system locale encoding if the process is not initially attached to a console.

(https://docs.python.org/3/library/sys.html#sys.stdout)