Console.Clear() doesn't clear the scrollback buffer on Unix-like platforms

[mklement0]

Console.Clear()'s documentation states (emphasis added):

Clears the console buffer and corresponding console window of display information.

Indeed, not just clearing the current terminal window (screen), but also the scrollback buffer is the typical use case, and it is how this method has always worked on Windows.^[1]

By contrast, the Unix implementation currently only clears the screen, which makes for an inconsistent cross-platform experience.

While there is no POSIX-compliant way to clear the scrollback buffer (only clearing the screen is mandated by POSIX, via tput clear), xterm-compatible terminal applications do support escape sequence <esc>[3J - see Wikipedia.

In practice, the macOS terminal application and the one on Ubuntu, for instance, do support this escape sequence - generally, terminal emulators based on the X Window System.

I don't know if there are popular terminal emulators out there that do not support it, but even a best-effort implementation would be useful.

Here's a workaround that demonstrates use of the escape sequence:

// Clears the screen and the scrollback buffer in xterm-compatible terminals.
Console.Clear(); Console.WriteLine("\x1b[3J")

---

^{[1] Whether there should be an opt-in for predictably clearing only the screen across platforms, while leaving the scrollback buffer intact, is a separate question.}

[SteveL-MSFT]

Perhaps adding an overload with a bool includeScrollBackBuffer would retain current behavior and allow for the desired behavior

[mklement0]

@SteveL-MSFT: Given that (a) clearing the buffer is documented and (b) doing so is generally the more sensible behavior, my vote is to simply change the existing behavior to align with the documentation / the behavior on Windows.

[danmoseley]

@stephentoub do you recall whether this was intentional, to not attempt to clear scrollback if supported?

[stephentoub]

do you recall whether this was intentional, to not attempt to clear scrollback if supported?

Not clearing it was not a goal. If there's a "safe" way to do it whenever it's possible (by safe I mean not emitting garbage to stdout when the particular escape code isn't supported by the terminal in use), such as relying on some reliable way to query that a compatible terminal is being used, I'm fine with doing so.

[mklement0]

Thanks, @stephentoub.

relying on some reliable way to query that a compatible terminal is being used

Based on the answers posted at https://unix.stackexchange.com/q/93376/54804, this should be as simple as:

# ... perform screen clearing (as before)
# In compatible terminals, also clear the scrollback buffer.
if (Environment.GetEnvironmentVariable("TERM").StartsWith("xterm")) Console.WriteLine("\x1b[3J");

xterm-compatible terminal emulators set env. var. TERM either to xterm or xterm-<colorCountSpec>, e.g., xterm-256color.

[stephentoub]

And presumably if we did that first and then did the current clear (rather than the opposite order), if we did end up for some reason generating garbage, it'd be mitigated by being immediately cleared? :)

[mklement0]

Exemplary garbage disposal, @stephentoub :)

[danmoseley]

@mklement0 do you wish to offer a PR?

[khellang]

You should be able to query the terminfo db for the E3 extension, no?

See user_caps(5) under "Recognized capabilities":

E3 string, tells how to clear the terminal's scrollback buffer.

When present, the clear(1) program sends this before clearing the terminal.

The command "tput clear" does the same thing.

[mklement0]

Thanks, @khellang, but in practice neither clear nor tput clear clear the scrollback buffer on macOS 10.14 and Ubuntu 18.04.

Consistent with that, the requisite escape sequence, \E[3J, is not present in the output from infocmp when invoked from an xterm-compatible terminal.

I know little about terminal-info databases, so maybe I'm missing something.

@danmosemsft: I'll give it a shot.

[khellang]

Interesting Linux SE answer; https://unix.stackexchange.com/a/375784.

[danmoseley]

@mklement0 sounds good. I sent you a collaborator invite: it's optional, if you accept it I can formally assign you. Note that accepting it switches on notifications for all of the repo, which you'll likely want to switch off again.

[mklement0]

Thanks, @danmosemsft; I've accepted the invitation.

[danmoseley]

Assigned.

[mklement0]

Thanks, @khellang; specifically, the passage of interest on the linked page is the following from this answer:

The gnome terminfo entry does not define an E3 capability, and on many systems — still! — neither does the xterm entry as this has not percolated down from Dickey terminfo. So clear just writes out the contents of the clear capability.

It sounds like the right thing to do is:

• (a) query the E3 capability in the terminfo database (based on env. var. TERM) and use the corresponding escape sequence, if defined.

• (b) if not defined, and TERM is xterm or starts with xterm- (matches ^xterm(?:-|$)), hard-code "\x1b[3J".

If performance is a concern, we could do (b) first, but I presume it won't matter.

[stephentoub]

If performance is a concern

It shouldn't be. The terminal doesn't change, so what to output for a Clear can be cached (it already is in many cases).

It sounds like the right thing to do is

Sounds reasonable.

[iSazonov]

@mklement0 If you are still working this.
We could implement the fix here
https://github.com/dotnet/corefx/blob/8ff3f535097394f602653a91f41ae4b72bc990eb/src/System.Console/src/System/ConsolePal.Unix.cs#L229

like

var clearString = TerminalFormatStrings.Instance.Clear;
if (clearString.Equal("\E[H\E[J - a standard value", StringComparison.Ordinal)
{
    clearString = "\E3J" + clearString;
}

WriteStdoutAnsiString(clearString);

[carlossanlop]

@mkelement0 have you had a chance to continue looking into this?

[janvorli]

I understand that the planned change is going to align the behavior with Windows, however, I am not aware of any text mode application on Unix that would clear the scrollback buffer. I myself would consider such a behavior of any application intrusive and unexpected. What are scenarios when clearing the scrollback buffer is beneficial?

[mklement0]

My apologies for dropping the ball on this, @danmosemsft , @carlossanlop and @iSazonov - can you please unassign me? I hope someone else will take this on.

However, it sounds like maybe there isn't consensus yet as to what to do:

@janvorli, I can see how for programmatic (as opposed to interactive use in a shell) clearing the scrollback buffer by default could be considered too invasive.

(On re-reading the documentation, it doesn't actually talk about the scrollback buffer, only about the "console buffer", which can justifiably be interpreted to mean the current screen only.)

Even though also clearing the scroll-back buffer is how it has always worked in regular console windows on Windows, I now see that in Windows Terminal it does not.

This takes us back to @SteveL-MSFT's suggestion to make buffer clearing an opt-in via a Boolean parameter.

We could therefore:

• Introduce the proposed overload, public static void Clear (bool includeScrollBackBuffer);

• make that overload work as intended when false is passed in regular console windows on Windows too (which I assume is possible)

• leave the current default behavior (parameterless overload: scrollback-buffer-too clearing in Windows console windows, current-screen clearing only on Unix and in Windows Terminals) and document the inconsistency, recommending use of the overload where you signal the intent explicitly going forward

[danmoseley]

To add a new API (rather than simply align behavior) we'd want a bit more evidence of need. What sort of scenario would lead someone to pass "true" ?

[khellang]

What sort of scenario would lead someone to pass "true" ?

I don't have a specific scenario in mind, but from researching this a while back, it seems like quite a few people are interested in clearing the buffer as well. Whether that is programmatically or interactively isn't clear (no pun intended) though.

On Windows, there's no need as the existing parameterless overload already does clear the buffer. But that overload isn't consistent across platforms. If you want to to be consistent, call the overload with either true or false and it'll work the same cross-platform 😊

That's the only way I can see this working out and not break everyone relying on the existing behavior.

[mklement0]

Good points, @khellang.

When implementing a shell or REPL the feature is of particular interest: often you want to start with a clean slate in a terminal before submitting a new command, so as not to get confused between the most recent command's output and unrelated output that preceded it.

Indeed, it was the desire for consistent cross-platform behavior of PowerShell's Clear-Host command that brought us here - PowerShell/PowerShell#8606

While PowerShell could certainly implement the behavior without using Console.Clear(), I can see other shells / REPLs benefitting from support in the CLR too - and indeed any type of application looking for consistent cross-platform behavior, notably including applications that on Windows do not want to clear the scrollback buffer.

---

As for native shell / utility behavior:

• On Windows, users of cmd.exe (cls) and PowerShell (Clear-Host, aliased to cls) are used to the scrollback-buffer getting cleared, by default and invariably - though, as stated, Windows Terminal will change that experience.

• On Linux, /usr/bin/clear is meant to clear the scrollback buffer by default - but still doesn't, as of Ubuntu 18.04, due to the terminal-capabilities database still not having been updated - see https://unix.stackexchange.com/a/375784/54804. Option -x is the opt-out that clears the current screen only.

• On macOS, /usr/bin/clear was only ever designed to clear the current screen, but the default terminal offers keyboard shortcut Command-K to clear the scrollback buffer too.

[Alex-K-O-R]

To add a new API (rather than simply align behavior) we'd want a bit more evidence of need. What sort of scenario would lead someone to pass "true" ?

I do:
https://github.com/Alex-K-O-R/NeatMenu

[Alex-K-O-R]

If it is logically correct ("clear" doesn't mean "scroll" or "rewind") and doesn't require a galactic-scale efforts, why .Clear(true) could not be done?

[hamarb123]

Hi, I've noticed this issue as well and developed a solution for my code on macOS:
On macOS, you can use Console.Write("\f\u001bc\x1b[3J"); to clear the console fully and reset the cursor to the top. I'm not sure if this is portable to other unix platforms.
This line of code properly emulates what windows does with Console.Clear() in terms of the text buffer (not sure about colours etc.).

[towianisci]

I'm new to this programming game as far as programming in C#. This was really frustrating me, and the last comment thank you. thank you. It was very annoying.
.net 7 the latest version of C# on windows
static void ClearHost()
{
Console.Write("\f\u001bc\x1b[3J");
}

This cleared the host completely now as opposed to just using Console.Clear() which appears to just clear the last buffer.

[adamsitnik]

The issue got fixed by #88487 and is included in .NET 8.

[iSazonov]

@adamsitnik It did not fix MacOS.

1. Is it possible to add the MacOS fix using direct escapes?
2. Or should we document the MacOS behavior exclusion for Console.Clear?
3. Should we reopen the issue or Console.Clear() does not clear buffers on macOS core#8260?

[kasperk81]

It did not fix MacOS.

are you sure? i tried 8.0-rtm with

Console.Clear();
Console.WriteLine("Hello, World!");

it clears the scrollback buffer on mac terminal and prints out Hello, World!
7.0 doesn't clear the scrollback buffer

[hamarb123]

Did you try printing a bunch of lines first, instead of immediately clearing @kasperk81 ?
I've not tested it with the fix yet, but plan to tomorrow when 8.0 releases (but I'm not convinced it will work for macOS until I test it myself).

[kasperk81]

of course i tested with overflown buffers and that's the difference between .net 8 vs 7

[hamarb123]

of course

How is it "of course" if it's not in the code you showed.

Edit: I don't know why you're downvoting all my comments for trying to determine if this bug is fixed or not. It wasn't "of course" because your code didn't show it, I never said anything else had an issue, other than my personal doubt that it's fixed (because it's been a bug for years), I simply asked if you checked it since it wasn't in the code (therefore I assumed you hadn't checked it - why do you think I asked). I then pointed out it wasn't "of course", since it wasn't as I just explained.

[mklement0]

I just installed the officially released 8.0.100 version and I can confirm @iSazonov's findings: As far as I can tell, it does NOT work on macOS - neither in Terminal.app nor in popular third-party alternative iTerm2.app:

Repro code (reproduces the symptom on macOS 13.5.2):

for (var i = 0; i < 500; ++i) { Console.WriteLine(i); }; 
Console.Write("Press a key to clear the terminal"); 
Console.ReadKey(true); 
Console.Clear();  // Scroll back up afterwards - you'll see that the numbers are still there.

By contrast, adding the escape sequence that clears the scrollback buffer shown in the initial post works, in both terminal applications:

Console.Clear(); Console.WriteLine("\x1b[3J");

Shell alternatives:

Note: /usr/bin/clear on macOS clears only the current screen, not the scrollback buffer.

• Bash

/usr/bin/clear; printf '\x1b[3J'

• PowerShell

/usr/bin/clear; printf "`e[3J"

[kasperk81]

Repro code (reproduces the symptom on macOS 13.5.2):

i'm on 14 (sonoma) and i don't see the numbers afterwards

[hamarb123]

Can confirm the following for Terminal.app:

• Fixed on macOS 14.1
• Broken on macOS 13.6.1

We should fix this since macOS 10.15+ is supported for .NET 8.

[adamsitnik]

The referenced fix (#88487) checks the terminfo for an E3 capability and uses it (not a hardcoded "\x1b[3J") when it's available.

I would love to fix it for all terminals, but how can we do that reliably if they don't define this capability? Sharing https://unix.stackexchange.com/questions/375743/why-clear-do-not-clear-whole-screen/375784#375784 for reference.

[mklement0]

@adamsitnik: Given @hamarb123's summary above, there's a pragmatic solution:

Instead of:

runtime/src/libraries/System.Console/src/System/TerminalFormatStrings.cs

Lines 77 to 80 in fd4621c

	if (db.GetExtendedString("E3") is string clearScrollbackBuffer)
	{
	Clear += clearScrollbackBuffer; // the E3 command must come after the Clear command
	}

use:

if (
  (
    RuntimeInformation.IsOSPlatform(OSPlatform.OSX) && Environment.OSVersion.Version.Major < 14 ?
      "\x1b[3J"
      :
      db.GetExtendedString("E3")
  ) is string clearScrollbackBuffer
)
{
    Clear += clearScrollbackBuffer; // the E3 command must come after the Clear command
}

Windows is not affected, and I'm not aware of issues on Linux distros, so this should be the only workaround required.

Once .NET no longer supports macOS 13 and below, this workaround can be removed.

[hamarb123]

I think something like the above is a good solution. (OperatingSystem.IsMacOS() && !OperatingSystem.IsMacOSVersionAtLeast(14) seems better suited though)

Note that with dotnet/roslyn#70497, we should be able to even write \e[3J with C# next :)

If it concerns you that it might print something random out after it @adamsitnik, you could also re-append TermInfo.WellKnownStrings.Clear (or something similar) to it again to ensure that a clear goes through for this workaround - it seems to me like this should work well in most cases.

Ie, you could write:

if (db.GetExtendedString("E3") is string clearScrollbackBuffer)
{
    Clear += clearScrollbackBuffer; // the E3 command must come after the Clear command
}
else if (OperatingSystem.IsMacOS() && !OperatingSystem.IsMacOSVersionAtLeast(14))
{
    Clear += "\e[3J" + db.GetString(TermInfo.WellKnownStrings.Clear); //use \e[3J to do a full clear on macOS terminal, and print the clear string again in case some terminal emulators don't understand this code to ensure we don't get random characters printed out
}

(note: I've not tested the above actually does what it's intended to)

I can make a PR for it if you'd like @adamsitnik

[adamsitnik]

The solution that you suggested assumes that every Terminal that runs on macOS supports "\x1b[3J". I doubt it's always true.

Don't get me wrong, I want to get all the bugs fixed. But in reliable way.

My current best idea is to recognize the Terminals that support it, but don't define that in Terminfo. Similarly to what we do in:

runtime/src/libraries/System.Console/src/System/TerminalFormatStrings.cs

Line 89 in 33d7f98

IsRxvtTerm = !string.IsNullOrEmpty(db.Term) && db.Term.Contains("rxvt", StringComparison.OrdinalIgnoreCase);

But I doubt that such a fix would be approved for backporting to .NET 8 (unless we can guarantee that older versions of this Terminals support it).

[hamarb123]

The solution that you suggested assumes that every Terminal that runs on macOS supports "\x1b[3J". I doubt it's always true.

The point of the way I suggested doing it is assuming that they don't all support it. That's why I suggested a normal clear after. Would that not work?