UTF8 Encoding Issue in Windows

[vquilon]

I use Intellij Idea in Windows, with the cp-1252 encoding by default on Windows. I try to execute a python script with the powershell because of problems with subprocess inputs on windows subsystem linux (wsl). The problem is when I try to execute, because powershell pick the encoding by default of windows
The content of the ps1 is:

python -c "print('█')"
python generate_dependencies.py

And the content of the python script is:

print("START", file=sys.stdout)
print("█")

And Im getting this weird results:

Is weird because the execution of the python script returns in cp-1252 and the python -c command inline return in weird encoding, I dont know which encoding is.

[ForNeVeR]

For me, the issue is that the output is different for run from an OS terminal and for run from the plugin. It should be exactly the same.

I was able to fix it using a hacky workaround:

Index: src/main/kotlin/com/intellij/plugin/powershell/ide/run/PowerShellScriptCommandLineState.kt
IDEA additional info:
Subsystem: com.intellij.openapi.diff.impl.patch.CharsetEP
<+>UTF-8
===================================================================
diff --git a/src/main/kotlin/com/intellij/plugin/powershell/ide/run/PowerShellScriptCommandLineState.kt b/src/main/kotlin/com/intellij/plugin/powershell/ide/run/PowerShellScriptCommandLineState.kt
--- a/src/main/kotlin/com/intellij/plugin/powershell/ide/run/PowerShellScriptCommandLineState.kt	(revision Staged)
+++ b/src/main/kotlin/com/intellij/plugin/powershell/ide/run/PowerShellScriptCommandLineState.kt	(date 1695076638030)
@@ -14,6 +14,7 @@
 import com.intellij.openapi.vfs.LocalFileSystem
 import com.intellij.plugin.powershell.lang.lsp.LSPInitMain
 import com.intellij.plugin.powershell.lang.lsp.languagehost.PowerShellNotInstalled
+import com.intellij.terminal.TerminalUiSettingsManager
 import java.io.File
 import java.util.regex.Pattern
 
@@ -29,7 +30,7 @@
         runConfiguration.getCommandOptions(),
         runConfiguration.scriptParameters
       )
-      val commandLine = PtyCommandLine(command)
+      val commandLine = PtyCommandLine(command).apply { withCharset(Charsets.UTF_8) }
       val project = runConfiguration.project
       val module = LocalFileSystem.getInstance().findFileByIoFile(File(runConfiguration.scriptPath))?.let {
         ProjectRootManager.getInstance(project).fileIndex.getModuleForFile(it)

I'll see how to properly fix it (I assume we should read the terminal encoding used by the IDE instead).

[ForNeVeR]

Please note that python -c "print('█')" in a .ps1 file only works if your PowerShell instance is able to read this from the file properly.

For a UTF-8 encoded file, you should either add UTF-8 BOM, or use a modern PowerShell version (6.0+).

We are thinking of switching to PowerShell Core by default if it's available, but that's topic of #176.