Window line-wrapping can cause colored output to break copied text

[lindhe]

Windows Terminal version

1.17.11461.0

Windows build number

10.0.22621.0

Other Software

bat 0.19.0

Steps to reproduce

I have found this very specific scenario for reproducing the issue reliably:

1. Install bat.

2. Create EXAMPLE.md with the following contents (including the markdown code fence, because we want syntax highlighting to color the output):

   ```shell
   echo '1 22 333 4444 55555 666666 7777777 88888888 999999999 0000000000 1 22 333 4444 55555 666666 7777777 88888888 999999999 0000000000'
   ```
   

3. Resize the terminal to be narrower than that long echo line.

4. Run the following:

   /usr/bin/batcat --color=always --style=header --decorations=always EXAMPLE.md

5. Copy the output.

6. Paste the output in a wider window.

It's worth noting that it works fine (i.e. "as expected") if the window is wide enough to not wrap when the command is run. If I resize the window later, the problem is not there.

Something that really surprises me is that piping the output via cat fixes the issue while looking exactly the same visually:

This bug is very similar to, or actually the same as, #5113.

Expected Behavior

For clarity, I think we need to distinguish between the line breaks introduced by Windows Terminal for the purposes of displaying a long line in a wrapped format and the line breaks that are part of the original string. I will call them wrapping line breaks and string line breaks respectively.

When copying a string from the terminal, I expect to copy the string line breaks but not the wrapping line breaks, so that when I paste it I get the "original" string (not the one with line wrappings).

Actual Behavior

When I copy text under these circumstances, it keeps the wrapping line breaks as well as he string line breaks.

[lindhe]

Alright, I've figured it out. And as always, it's a very very very specific edge case. It's when I use bat (aka. batcat) to output colored text, with certain flags set.

1. Install bat.

2. Create EXAMPLE.md with the following contents (including the markdown code fence, because we want syntax highlighting to color the output):

   ```shell
   echo '1 22 333 4444 55555 666666 7777777 88888888 999999999 0000000000 1 22 333 4444 55555 666666 7777777 88888888 999999999 0000000000'
   ```
   

3. Resize the terminal to be narrower than that long echo line.

4. Run the following:

   /usr/bin/batcat --color=always --style=header --decorations=always EXAMPLE.md

5. Copy the output.

6. Paste the output in a wider window.

It's worth noting that it works fine (i.e. "as expected") if the window is wide enough to not wrap when the command is run. If I resize the window later, the problem is not there.

Something that really surprises me is that piping the output via cat fixes the issue while looking exactly the same visually:

[zadjii-msft]

Can you try this again on Terminal Preview, v1.19/? (manual download link: https://github.com/microsoft/terminal/releases/tag/v1.19.2831.0) We just recently re-wrote how resize/reflow works, and I have a feeling that it may have just fixed this ☺️

[j4james]

I don't think v1.19 will make any difference. If you run batcat inside a script session and look at the log, you can see it's actually splitting the content into multiple lines with a line break between them, so it's expected that a resize won't treat that as a wrapped line.

I also noticed there's an issue in their bug tracker which sounds like it might be related (see sharkdp/bat#1740).

[lindhe]

Can you try this again on Terminal Preview, v1.19/? (manual download link: https://github.com/microsoft/terminal/releases/tag/v1.19.2831.0) We just recently re-wrote how resize/reflow works, and I have a feeling that it may have just fixed this ☺️

Thanks for the suggestion. I just tried with Preview version 1.19.2682.0 and it had the exact same behavior, unfortunately.

I don't think v1.19 will make any difference. If you run batcat inside a script session and look at the log, you can see it's actually splitting the content into multiple lines with a line break between them, so it's expected that a resize won't treat that as a wrapped line.

I also noticed there's an issue in their bug tracker which sounds like it might be related (see sharkdp/bat#1740).

Interesting! That bug does indeed sound very similar to my problems. It is beyond be how there can be a line wrap that affects copying but not resizing. Do you have any clue how that's possible?

[j4james]

It is beyond be how there can be a line wrap that affects copying but not resizing. Do you have any clue how that's possible?

The fact that the resizing "works" for you is actually a bug. When you have a line that's the exact width of the screen, it is mistakenly considered to be wrapping when the screen is resized (see issue #3088). That was supposed to have been fixed recently, but it still doesn't seem to be working correctly in Windows Terminal.

[zadjii-msft]

Can you try repro'ing this with the debug tap to get a trace of all the input and output? Once the bug starts occurring, send us a screenshot/copy-paste of the output and we might be able to figure out what the Terminal thinks it's getting here.

[lindhe]

I'd love to, but:

[j4james]

It looks like there was a typo in Mike's link. It should be:
https://github.com/microsoft/terminal/wiki/Troubleshooting-Tips#enabling-the-debug-tap

[zadjii-msft]

(thanks - forgot that I had fixed that typo but not the links in my quick replies 🤦)

[lindhe]

Here you go! :)

␛[K␍␊
␛[K␛[5;3H␛[?25h␛[13;28;13;0;32;1_␛[200~/usr/bin/batcat␣--color=always␣--style=header␣--decorations=always␣/tmp/tmp.sbiNSfFwSY/EXAMPLE.md␛[201~␛[7m/usr/bin/batcat␣--color=always␣--style=header␣--decorations=always␣/tmp/tmp.sbiNSfFwSY/EXAMPLE.md␛[27m␛[K␛[13;28;13;1;32;1_␛[?25l␛[?1l␛>␛[?2004l␛[5;3H/usr/bin/batcat␣--color=always␣--style=header␣--decorations=always␣/tmp/tmp.sbiNSfFwSY/EXAMPLE.md␍␊
␛[?25h␛[13;28;13;0;32;1_File:␣␛[1m/tmp/tmp.sbiNSfFwSY/EXAMPLE.md␛[38;5;231m␛[22m␍␊
```␛[38;5;141mshell␛[38;5;231m␍␊
echo␣'1␣22␣333␣4444␣55555␣666666␣7777777␣88888888␣999999999␣0000000000␣1␣22␣␍␊
333␣4444␣55555␣666666␣7777777␣88888888␣999999999␣0000000000'␍␊
```␛[m␍␊
␛[K␍␊

[zadjii-msft]

Oh whoops I clearly never hit enter on my reply last week. The debug tap obviously showed a manual line break, that's what we're trying to figure out 🤦

Can you try with script instead/? https://github.com/microsoft/terminal/wiki/Troubleshooting-Tips#script

That'll actually capture what was emitted straight from the application, rather than what conpty re-emitted.

(thanks for your patience)

[lindhe]

(thanks for your patience)

Thanks for the help!

typescript without debug tap

Script started on 2023-11-09 13:45:48+01:00 [TERM="xterm-256color" TTY="/dev/pts/3" COLUMNS="76" LINES="37"]
�[1m�[7m%�[27m�[1m�[0m                                                                           
 
�]9;9;\\wsl.localhost\Ubuntu\tmp\tmp.pxjFyYQaGS�\�]7;file://conoa/tmp/tmp.pxjFyYQaGS�\
�[0m�[27m�[24m�[JÔò¡ÔöÇ�[1m�[32mandreas@conoa�[00m �[1m�[34m/tmp/tmp.pxjFyYQaGS �[00m
Ôò░ÔöÇ�[1m$�[0m �[K�[71C�[1m�[0m�[71D�[?1h�=�[?2004h�[7m/usr/bin/batcat --color=always --style=header --decorations=always EXAMP�[7mL�[7mE.md�[27m�[K�[A��[27m/�[27mu�[27ms�[27mr�[27m/�[27mb�[27mi�[27mn�[27m/�[27mb�[27ma�[27mt�[27mc�[27ma�[27mt�[27m �[27m-�[27m-�[27mc�[27mo�[27ml�[27mo�[27mr�[27m=�[27ma�[27ml�[27mw�[27ma�[27my�[27ms�[27m �[27m-�[27m-�[27ms�[27mt�[27my�[27ml�[27me�[27m=�[27mh�[27me�[27ma�[27md�[27me�[27mr�[27m �[27m-�[27m-�[27md�[27me�[27mc�[27mo�[27mr�[27ma�[27mt�[27mi�[27mo�[27mn�[27ms�[27m=�[27ma�[27ml�[27mw�[27ma�[27my�[27ms�[27m �[27mE�[27mX�[27mA�[27mM�[27mPL�[27mE�[27m.�[27mm�[27md�[?1l�>�[?2004l


File: �[1mEXAMPLE.md�[0m�[m
�[38;5;231m```�[0m�[38;5;141mshell�[0m�[m
�[38;5;231mecho '1 22 333 4444 55555 666666 7777777 88888888 999999999 0000000000 1 22 �[0m�[m
�[38;5;231m333 4444 55555 666666 7777777 88888888 999999999 0000000000'�[0m�[m
�[38;5;231m```�[0m�[m

�[K�[1m�[7m%�[27m�[1m�[0m                                                                           
 
�]9;9;\\wsl.localhost\Ubuntu\tmp\tmp.pxjFyYQaGS�\�]7;file://conoa/tmp/tmp.pxjFyYQaGS�\
�[0m�[27m�[24m�[JÔò¡ÔöÇ�[1m�[32mandreas@conoa�[00m �[1m�[34m/tmp/tmp.pxjFyYQaGS �[00m
Ôò░ÔöÇ�[1m$�[0m �[K�[71C�[1m�[0m�[71D�[?1h�=�[?2004he�exit�[?1l�>�[?2004l


Script done on 2023-11-09 13:45:59+01:00 [COMMAND_EXIT_CODE="0"]

debug tap output, without script

␛[K␍␊
␛[K␛[8;5H␛[?25h␛[O␛[200~/usr/bin/batcat␣--color=always␣--style=header␣--decorations=always␣EXAMPLE.md␛[201~␛[I␛[7m/usr/bin/batcat␣--color=always␣--style=header␣--decorations=always␣EXAMPLE.md␛[27m␛[K␛[13;28;13;1;32;1_␛[13;28;13;0;32;1_␛[?25l␛[?1l␛>␛[?2004l␛[8;5H/usr/bin/batcat␣--color=always␣--style=header␣--decorations=always␣EXAMPLE.md␍␊
␛[?25h␛[?25l␛]9;9;\\wsl.localhost\Ubuntu\tmp\tmp.pxjFyYQaGS␛\␛]7;file://conoa/tmp/tmp.pxjFyYQaGS␛\␛[?1h␛=␛[?2004hFile:␣␛[1mEXAMPLE.md␛[38;5;231m␛[22m␍␊
```␛[38;5;141mshell␛[38;5;231m␍␊
echo␣'1␣22␣333␣4444␣55555␣666666␣7777777␣88888888␣999999999␣00000␍␊
00000␣1␣22␣333␣4444␣55555␣666666␣7777777␣88888888␣999999999␣00000␍␊
00000'␍␊
```␛[m␍␊
╭─␛[32m␛[1mandreas@conoa␛[m␣␛[34m␛[1m/tmp/tmp.pxjFyYQaGS␣␛[m␛[K␍␊
╰─␛[1m$␛[22m␛[K␍␊
␛[K␍␊

[carlos-zamora]

Looks like the bat utility inserts hard line breaks. This issue on the bat repository might be helpful: sharkdp/bat#897

Thanks for filing!

[lindhe]

It is beyond be how there can be a line wrap that affects copying but not resizing. Do you have any clue how that's possible?

The fact that the resizing "works" for you is actually a bug. When you have a line that's the exact width of the screen, it is mistakenly considered to be wrapping when the screen is resized (see issue #3088). That was supposed to have been fixed recently, but it still doesn't seem to be working correctly in Windows Terminal.

@carlos-zamora please read the above.

[DHowett]

At the very least, there are a couple issues here.

1. bat is emitting hard line breaks.
2. When you resize, sometimes Terminal unwraps a line of text even though it shouldn't. This is reproduceable with...
   • PowerShell: "a"*$host.ui.rawui.windowsize.width

for 1

When I copy text under these circumstances, it keeps the wrapping line breaks as well as he string line breaks.

This is because bat is emitting hard line breaks, visible below:

for 2

Now, we should not treat hard wrapped lines as wrapped text under any circumstances. However, we used to have an issue where text would unwrap on resize even though we knew it had a hard line break¹

I can reproduce this in 1.17 (where you originally filed the bug):

... and in 1.18 ...

but not in 1.19 (preview) or 1.20 (canary):

If I try your repro from script (thanks for sharing it!) on 1.19, it no longer unwraps when I resize it :)

(window width 76 as specified in repro)

(window resized to larger)

That is what @j4james was saying was a bug. Resize should have never unwrapped the lines, which would have made it more obvious that bat was inserting hard line breaks. 😁

Footnotes

1. The resizing code was... uh. It was not great. It is much better now! ↩

[j4james]

@DHowett FYI, I can still reproduce the bug with the bash test case from issue #3088 in version 1.19.2682.0 of Windows Terminal. However, your PowerShell test case does not reproduce the bug for me.

Looking at the debug tap, it seems that conpty is not actually writing out the expected CR LF at the end of the line in the bash test, but it is doing so in your PowerShell test. So I think the wrapping bug has indeed been fixed, but there's potentially now a conpty EOL bug that is causing the same problem.

[lindhe]

@j4james I cannot reproduce it in 1.19.3172.0.

Pretty sure I did reproduce it in Terminal Preview before, but maybe not. ¯\_(ツ)_/¯

EDIT: I think I know what I did wrong last time I "reproduced" it in Preview! I think that was before I realized that bat actually produced a newline character. So I tried copy-pasting (which I thought was the bug), but i did not try to resize the window (which was the actual bug).