Using Windows Terminal with a git-for-windows tab displays odd characters

[briannipper]

• I was not able to find an open or closed issue matching what I'm seeing

Setup

• Which version of Git for Windows are you using? Is it 32-bit or 64-bit?

git version 2.24.1.windows.2
cpu: x86_64
built from commit: 992f0773022527b1b0cb1e0c13aec97dd5248053
sizeof-long: 4
sizeof-size_t: 8 

• Which version of Windows are you running? Vista, 7, 8, 10? Is it 32-bit or 64-bit?

Microsoft Windows [Version 10.0.18363.592]

• What options did you set as part of the installation? Or did you choose the
  defaults?

from C:\Program Files\Git\etc\install-options.txt
Editor Option: VIM
Custom Editor Path: 
Path Option: Cmd
SSH Option: OpenSSH
Tortoise Option: false
CURL Option: OpenSSL
CRLF Option: CRLFAlways
Bash Terminal Option: MinTTY
Performance Tweaks FSCache: Enabled
Use Credential Manager: Enabled
Enable Symlinks: Disabled
Enable Builtin Interactive Add: Disabled

• Any other interesting things about your environment that might be related
  to the issue you're seeing?
  • I've installed arcanist to interact with phabricator and when using the "standard" bash window I have no display issues; however, when I use the Bash window as a tab in Windows Terminal I have an unexpected issue. Here is a link to arcanist in case it's helpful:
    https://secure.phabricator.com/book/phabricator/article/arcanist/

Details

• Which terminal/shell are you running Git from? e.g Bash/CMD/PowerShell/other

  • Windows Terminal

• What commands did you run to trigger this issue? If you can provide a
  Minimal, Complete, and Verifiable example
  this will help us understand the issue.

  • The following commands result in showing the issue which are all Arcanist commands
    • arc diff
    • arc commit
    • arc feature

• What did you expect to occur after running these commands?

  • The command would run without showing any unexpected characters

• What actually happened instead?

  • The commands run; however, there are clearly unexpected characters.

• If the problem was occurring with a specific repository, can you provide the
  URL to that repository to help us with testing?

  • No specific URL for a repo; however, I do have a link to a bug related to Windows Terminal where they explain why this is really an issue with msys/mingw/cygwin runtime
    arcanist with PHP in GitBash tab shows extra characters but not other tabs microsoft/terminal#4076 (comment)

[dscho]

The following commands result in showing the issue which are all Arcanist commands

* arc diff

* arc commit

* arc feature

1. This is not a minimal example
2. This example is about Arcanist, not about Git for Windows. If you can replicate with bare GIt for Windows, please do so and provide the details here.

[briannipper]

@dscho
As I understand the problem based on the comments from the bug I opened (and was later closed) under the Microsoft Terminal the issue lies in...
msys/mingw/cygwin runtime

As such I was offering up the example of the commands I run in arcanist which are attempting to use aspects of color and text formating that result in odd characters.

As i listed in the other bug report with screen shots (and replicated here) I'm looking for help in addressing the way the odd characters appear in the msys/mingw/cygwin when it's used within Windows Terminal... perhaps this isn't an issue with git-for-windows then?

Image of mingw in windows terminal with odd characters:

Image of PS Core in windows terminal running same commands without odd characters:

If this repo is also not the right place to list this bug advise and I'll keep looking for the right place.

Thank you for your time.

[dscho]

I'll need a minimal reproducer to assess (and possibly fix) the issue. Not screenshots. I will not install arcanist and hunt for that reproducer myself.

[rimrul]

Ok, this issue seems to consist of multiple layers.

• you're probably calling the wrong bash
• you're probably using the wrong php
• the way arcanist determines wether they can use ansi colors is broken

Let me explain. You're probably using the recommended profile for Git Bash from the microsoft/terminal repo. That calls bin/bash.exe instead of git-bash.exe among other differences, this sets TERM to cygwin instead of xterm. That's the reason you can't reproduce this in standalone Git Bash.

You're probably also using a PHP for windows, not msys2 or cygwin. That means DIRECTORY_SEPARATOR is probably set to \ and posix_isatty() doesn't exist. It also doesn't handle ANSI color sequences.

arcanist disables ANSI color sequences if DIRECTORY_SEPARATOR is not / and TERM isn't cygwin or posix_isatty() exists and returns false for STANDARDOUT. There are some other cases, but these are the interesting ones for this issue.

That leads arcanist to assume it can use ANSI color sequences, but your php doesn't know what to do with them.

You could test TERM=xterm arc diff and if that works set TERM=xterm from ~/.bashrc if that works.

[dscho]

@briannipper have you tried @rimrul's suggestion? What was the outcome?

[DHowett-MSFT]

You're probably using the recommended profile for Git Bash from the microsoft/terminal repo.

(as the Terminal maintainer, I'd be happy to learn that a bunch of our problems could be resolved by recommending something different.)

[dscho]

@DHowett-MSFT I haven't checked: do you call bin\bash.exe -li or usr\bin\bash.exe -li? The former should work correctly. The latter will miss important environment variable settings.

[briannipper]

My apologizes for the unusually long delay in following up on this.

Thank you so much @rimrul you suggestion to use TERM=xterm FIXED all my formatting issues. Also, thank you for the additional explanation of the various layers involved, it's quite helpful to see the pieces as you outline them.

After creating the entry in ~/.bashrc all my arc problems have disappeared. :)

[DHowett-MSFT]

Our current recommendation is the former, but it looks like we're missing --login and friends. I'll fix that.

Thanks!

[dscho]

Yep, without --login, the environment won't be set up properly.

[trajano]

Perhaps it needs to be recompiled with a newer cygwin library as per microsoft/terminal#4809 (comment)

[dscho]

Perhaps it needs to be recompiled with a newer cygwin library as per microsoft/terminal#4809 (comment)

The latest snapshot includes an MSYS2 runtime based on Cygwin v3.1.5, so it is safely newer than the suggested v3.1.2. Even Git for Windows v2.27.0 comes with an MSYS2 runtime based on a newer Cygwin (namely, v3.1.4): https://github.com/git-for-windows/build-extra/blob/cd17006382667f8bccd23060fa8610591c716709/versions/package-versions-2.27.0.txt#L110

[trajano]

@dscho just verified with https://gist.github.com/ketan/2866a236596636311d64 and I see the full color palette now for git bash.

Though the processes called by git bash is still showing the escape sequences

so it may need more work from the Cygwin end. Is there an issue opened up on their end that discusses this?

Works well on "Git Bash" (no terminal)

[dscho]

It might just be a question of setting TERM appropriately.

[trajano]

It might just be a question of setting TERM appropriately.

I think it may be something that needs to happen in OpenJDK unless there's a per process handling.

[luismbo]

FWIW, here's a minimal example using git 2.29.2.windows.3 and Windows Terminal 1.7.1033.0:

$ echo -e "$(tput setaf 1)$(tput bold)RED$(tput sgr0)" > red
$ cat red
RED
$ /c/Windows/System32/more red
←[31m←[1mRED←(B←[m

Both more and cat work as expected if invoked from cmd.exe, PowerShell or WSL2. Setting TERM to xterm (changed from xterm-256color; not sure who's setting that) doesn't seem to change the result.

[dscho]

@luismbo I just tried to your reproducer, and all the Git parts seem to work all right:

But indeed, more did not work:

$ C:/Windows/system32/more "$(cygpath -aw /tmp/a1)"
←[31m←[1mRED←(B←[m

However, at this stage I really have to ask: since neither Windows Terminal nor more.com are developed by the Git for Windows project, why report that problem here? Would it not make a ton more sense to report it with the Windows Terminal project?

[DHowett]

This one is interesting! more, owing to its age, probably never sets the console mode to include ENABLE_VIRTUAL_TERMINAL_PROCESSING (which is not a default flag, but is enabled by CMD and PowerShell).

If MSYS is using ConPTY and running more in an isolated console session, it is quite likely that it's inheriting the default/new empty console state. This behavior isn't unexpected, honestly -- I would have been surprised to see more opt in to newer console features.

[luismbo]

@DHowett, @dscho I used more for the sake of easy reproducibility, but I actually came across this issue using my own CLI program. I will investigate how to set the console mode and report back. Thanks for the hint, @DHowett. (FWIW, without any sort of knowledge about the constraints involved, I’m inclined to side with @dscho and suggest that perhaps that mode could be set by default by the Windows Terminal. 🤷‍♂️)

[DHowett]

perhaps that mode could be set by default by the Windows Terminal.

Ah! We beat you by about two years; however, any application running inside Terminal is allowed to clear that flag if they want, and any sessions spawned by MSYS are sufficiently detached from the original hosting terminal that we can't know WT was involved to begin with 😄

(EDIT: However, this change is likely to have been released with Windows 11? I can figure out exactly when it came out ... but MSYS will be impacted by that pull request as well. 👍🏻)

[luismbo]

The plot thickens! I should be reunited with my Windows 11 machine some time next week, so I'll try that too.

[uniphiny]

Thanks

[perlun]

Apologies for resurrecting an old thread, but... I seem to be running into something slightly similar to this with Git for Windows 2.35.1.windows.2.

I have a console application running which emits ANSI colour codes. I'm running this inside the Windows Terminal.

When I run it through a regular cmd.exe, it works perfectly:

However, when executed via bash.exe provided by Git for Windows, all the ANSI sequences are suddenly ignored:

The Git Bash profile for Windows Terminal was added using the Git for Windows installer. The exact bash path being used here is C:/Program Files/Git/bin/bash.exe -i -l.

As can be seen above in the shell prompt etc, other ANSI colouring sequences are working perfectly. I also tested with the gist provided at https://gist.github.com/ketan/2866a236596636311d64, and all of this displays flawlessly as far as I can tell.

I really don't know who to blame 😁 or where to start debugging this. Is this a problem with:

• Git Bash
• Windows Terminal
• my own application (built from https://github.com/perlang-org/perlang/tree/f84b8da6953146148055e035a1760651bb3cb00f)
• the .NET Framework (which it runs on)
• cosmic rays? 🙈

Any suggestions are greatly appreciated. 🙏

[knutwannheden]

Did you try with /usr/bin/bash.exe instead of /bin/bash.exe as the executable launched by Terminal? Has fixed issues for me in the past, but even so, I still sometimes have issues with ANSI codes under Git Bash in Terminal ☹️

[dscho]

@perlun @knutwannheden Have any of you gentle people tried the suggestion to set the environment variable TERM=xterm?

[perlun]

@dscho Good point, worth trying. 👍 Here are the results.

TERM=xterm-256color

(This TERM setting seems to be the default in the bash provided with Git for Windows)

TERM=xterm

---

Did you try with /usr/bin/bash.exe instead of /bin/bash.exe as the executable launched by Terminal?

@knutwannheden Interesting point; there is in fact another bash.exe there. 🤔 So if I replace C:/Program Files/Git/bin/bash.exe -i -l with C:/Program Files/Git/usr/bin/bash.exe -i -l... that changes the prompt above from MINGW64 to MSYS, but I can't seem to get ANSI colours propagated through from my program regardless.

I tested with both TERM=xterm-256color and TERM=xterm in this case also.

---

@DHowett - you seem to be a (Microsoft?) hacker in the Windows Terminal subsystem. Could this be related to ENABLE_VIRTUAL_TERMINAL_PROCESSING somehow not being propagated to the child process in this case?

In case anyone wants a MCRE, the latest build available from https://perlang.org/download/ has the ANSI colours enabled:

1. Run the installer in a Git for Windows-provided bash under Windows Terminal
2. Run ~/.perlang/nightly/bin/perlang afterwards. It works fine in CMD.EXE, but fails miserably in Bash for some utterly odd reason.

[dscho]

I have another suggestion: can you run the perlang command via winpty?

[luismbo]

FWIW, I have some more input using my minimal example using more.com (#2483 (comment)):

1. Using Windows 11 (version 10.0.22000 Build 22000) and Windows Terminal 1.11.3471.0 or 1.13.10395.0 doesn't help.
2. winpty more.com red doesn't help either.
3. powershell within Git's bash also fails. Does this give you any hint?

PS Z:\temp> more red
RED

PS Z:\temp> & 'C:\Program Files\Git\bin\bash.exe'

luis@thinkpad MINGW64 /z/temp
$ more.com red
←[31m←[1mRED←(B←[m

luis@thinkpad MINGW64 /z/temp
$ powershell

PS Z:\temp> more red
←[31m←[1mRED←(B←[m

[luismbo]

Just in case using more makes you suspicious, I can replicate the same behavior using this ad-hoc cat compiled with MSYS2's MINGW64 profile:

#include <stdio.h>

int main(int argc, char **argv)
{
        if (argc != 2) { return -1; }
        FILE *f = fopen(argv[1], "r");
        if (f) {
                char buf[512];
                while (fgets(buf, sizeof buf, f))
                        puts(buf);
                fclose(f);
        }
        return 0;
}

[luismbo]

Setting the ENABLE_VIRTUAL_TERMINAL_PROCESSING mode, as hinted by @DHowett in #2483 (comment) works:

// red.c
#include <stdio.h>
#include <windows.h>

int main(void)
{
    HANDLE con = GetStdHandle(STD_OUTPUT_HANDLE);
    DWORD mode;
    GetConsoleMode(con, &mode);
    SetConsoleMode(con, mode | ENABLE_VIRTUAL_TERMINAL_PROCESSING);
    puts("\e[31m\e[1mRED\e(B\e[m\n");
    return 0;
}

This red.exe outputs a red-colored "RED" everywhere: Command Prompt, Windows Terminal (both powershell and Git's bash), mintty.

This seems to suggest that Git's bash turns ENABLE_VIRTUAL_TERMINAL_PROCESSING off at some point, maybe?

[dscho]

This seems to suggest that Git's bash turns ENABLE_VIRTUAL_TERMINAL_PROCESSING off at some point, maybe?

It's not really the Bash, it is the MSYS2 runtime, which is a close fork of the Cygwin runtime.

Could you download the build artifact of https://github.com/git-for-windows/msys2-runtime/actions/runs/1961614385 (which essentially gives a preview of the changes that will come with Cygwin v3.3.5), extract the msys-2.0.dll file and replace the one in your installed Git's usr\bin, then try again?

[luismbo]

@dscho same behaviour. red.exe works, ad-hoc cat or more.com don't.

luis@thinkpad MINGW64 /z/temp
$ cat red
RED

luis@thinkpad MINGW64 /z/temp
$ more.com red
←[31m←[1mRED←(B←[m

luis@thinkpad MINGW64 /z/temp
$ ./my-cat.exe red
←[31m←[1mRED←(B←[m

luis@thinkpad MINGW64 /z/temp
$ winpty ./my-cat.exe red
←[31m←[1mRED←(B←[m

luis@thinkpad MINGW64 /z/temp
$ ./red.exe
RED

luis@thinkpad MINGW64 /z/temp
$ md5sum /usr/bin/msys-2.0.dll
97905bfd688e82d4a43d428229c54457 */usr/bin/msys-2.0.dll

[dscho]

I won't pretend to have time to dig into this deeper. There is a good chance, though, that this is actually an issue in Cygwin itself. If it is, you won't have to rely on me getting back to this issue, you can report it to the Cygwin project and have much better chances of seeing this resolved soon.

[damiangreen]

Is there a workaround for this?

[luismbo]

Is there a workaround for this?

Can you be more specific?

[perlun]

FWIW, my problem described in #2483 (comment) is no loner present. The relevant versions:

• Windows Terminal 1.16.10261.0
• Git Bash 5.2.15(1)-release (from Git for Windows 2.40.0 64-bit)
• Windows 10

I am unsure if this is because of a fix in either one of the above or changes in my own program, but I haven't done any changes that would have an obvious effect on this. (Such changes are being worked on, but they are not yet merged... 😉)