Buffer problems with (really) long output

[Jaykul]

In our experiment we're outputting 100k records with a format file that indicates table formatting. Long story short:

1. PoshConsole needs a max buffer size.
2. PoshConsole.Execute needs a max output setting

Because of the problem spelled out below we need to limit the total scrollback buffer, and futhermore, in PoshConsole we want to be able to invoke a command and only scroll the first 5 or 10 items before going silent (we still want all the output back in our callback, we just don't need it printed out).

We admit this problem is not completely unique to PoshConsole, even in PowerShell.exe, scrolling 100k rows takes awhile -- but it's much worse in PoshConsole, because while PoshConsole is somewhat slower than PowerShell for output ... when there are thousands of records output, PowerShell's buffer doesn't care and operates at roughly the same speed all the way through, whereas PoshConsole's buffer is unlimited, and rich-text -- so it gets slower and slower and takes up more and more memory (starts causing faults to get more memory, making it even slower, etc).

Perhaps virtualizing the output could help, but really there's probably no need to store more than a thousand, or at most the 5000 lines of output that PowerShell stores by default in PS5.

Thought: In fast-running commands, it might be better to actually buffer the output before display, and if there's more than maxbuffer ... clear the screen and output the last maxbuffer items?

NOTE: In the worst case scenario, we get this message in the attached debugger:

The CLR has been unable to transition from COM context 0x71cb94e8 to COM context 0x71cb9610 for 60 seconds. The thread that owns the destination context/apartment is most likely either doing a non pumping wait or processing a very long running operation without pumping Windows messages. This situation generally has a negative performance impact and may even lead to the application becoming non responsive or memory usage accumulating continually over time. To avoid this problem, all single threaded apartment (STA) threads should use pumping wait primitives (such as CoWaitForMultipleHandles) and routinely pump messages during long running operations.