Add support for the DECSCNM screen mode (#3817)

## Summary of the Pull Request

This adds support for the [`DECSCNM`](https://vt100.net/docs/vt510-rm/DECSCNM.html) private mode escape sequence, which toggles the display between normal and reverse screen modes. When reversed, the background and foreground colors are switched. Tested manually, with [Vttest](https://invisible-island.net/vttest/), and with some new unit tests.

## References

This also fixes issue #72 for the most part, although if you toggle the mode too fast, there is no discernible flash.

## PR Checklist
* [x] Closes #3773
* [x] CLA signed. If not, go over [here](https://cla.opensource.microsoft.com/microsoft/Terminal) and sign the CLA
* [x] Tests added/passed
* [ ] Requires documentation to be updated
* [ ] I've discussed this with core contributors already. If not checked, I'm ready to accept this work might be rejected in favor of a different grand plan. Issue number where discussion took place: #xxx

## Detailed Description of the Pull Request / Additional comments

I've implemented this as a new flag in the `Settings` class, along with updates to the `LookupForegroundColor` and `LookupBackgroundColor` methods, to switch the returned foreground and background colors when that flag is set. 

It also required a new private API in the `ConGetSet` interface to toggle the setting. And that API is then called from the `AdaptDispatch` class when the screen mode escape sequence is received.

The last thing needed was to add a step to the `HardReset` method, to reset the mode back to normal, which is one of the `RIS` requirements.

Note that this does currently work in the Windows Terminal, but once #2661 is implemented that may no longer be the case. It might become necessary to let the mode change sequences pass through conpty, and handle the color reversing on the client side.
 
## Validation Steps Performed

I've added a state machine test to make sure the escape sequence is dispatched correctly, and a screen buffer test to confirm that the mode change does alter the interpretation of colors as expected.

I've also confirmed that the various "light background" tests in Vttest now display correctly, and that the `tput flash` command (in a bash shell) does actually cause the screen to flash.

src/host/getset.cpp

@@ -1291,6 +1291,35 @@ void ApiRoutines::GetConsoleDisplayModeImpl(ULONG& flags) noexcept
     return STATUS_SUCCESS;
 }
 
+// Routine Description:
+// - A private API call for changing the screen mode between normal and reverse.
+//    When in reverse screen mode, the background and foreground colors are switched.
+// Parameters:
+// - reverseMode - set to true to enable reverse screen mode, false for normal mode.
+// Return value:
+// - STATUS_SUCCESS if handled successfully. Otherwise, an appropriate error code.
+[[nodiscard]] NTSTATUS DoSrvPrivateSetScreenMode(const bool reverseMode)
+{
+    try
+    {
+        Globals& g = ServiceLocator::LocateGlobals();
+        CONSOLE_INFORMATION& gci = g.getConsoleInformation();
+
+        gci.SetScreenReversed(reverseMode);
+
+        if (g.pRender)
+        {
+            g.pRender->TriggerRedrawAll();
+        }
+
+        return STATUS_SUCCESS;
+    }
+    catch (...)
+    {
+        return NTSTATUS_FROM_HRESULT(wil::ResultFromCaughtException());
+    }
+}
+
 // Routine Description:
 // - A private API call for making the cursor visible or not. Does not modify
 //      blinking state.

src/host/getset.h

@@ -29,6 +29,8 @@ void DoSrvPrivateSetDefaultAttributes(SCREEN_INFORMATION& screenInfo, const bool
 [[nodiscard]] NTSTATUS DoSrvPrivateSetCursorKeysMode(_In_ bool fApplicationMode);
 [[nodiscard]] NTSTATUS DoSrvPrivateSetKeypadMode(_In_ bool fApplicationMode);
 
+[[nodiscard]] NTSTATUS DoSrvPrivateSetScreenMode(const bool reverseMode);
+
 void DoSrvPrivateShowCursor(SCREEN_INFORMATION& screenInfo, const bool show) noexcept;
 void DoSrvPrivateAllowCursorBlinking(SCREEN_INFORMATION& screenInfo, const bool fEnable);
 

src/host/outputStream.cpp

@@ -345,6 +345,19 @@ bool ConhostInternalGetSet::PrivateSetKeypadMode(const bool fApplicationMode)
     return NT_SUCCESS(DoSrvPrivateSetKeypadMode(fApplicationMode));
 }
 
+// Routine Description:
+// - Connects the PrivateSetScreenMode call directly into our Driver Message servicing call inside Conhost.exe
+//   PrivateSetScreenMode is an internal-only "API" call that the vt commands can execute,
+//     but it is not represented as a function call on our public API surface.
+// Arguments:
+// - reverseMode - set to true to enable reverse screen mode, false for normal mode.
+// Return Value:
+// - true if successful (see DoSrvPrivateSetScreenMode). false otherwise.
+bool ConhostInternalGetSet::PrivateSetScreenMode(const bool reverseMode)
+{
+    return NT_SUCCESS(DoSrvPrivateSetScreenMode(reverseMode));
+}
+
 // Routine Description:
 // - Connects the PrivateShowCursor call directly into our Driver Message servicing call inside Conhost.exe
 //   PrivateShowCursor is an internal-only "API" call that the vt commands can execute,

src/host/outputStream.hpp

@@ -93,6 +93,8 @@ class ConhostInternalGetSet final : public Microsoft::Console::VirtualTerminal::
     bool PrivateSetCursorKeysMode(const bool applicationMode) override;
     bool PrivateSetKeypadMode(const bool applicationMode) override;
 
+    bool PrivateSetScreenMode(const bool reverseMode) override;
+
     bool PrivateShowCursor(const bool show) noexcept override;
     bool PrivateAllowCursorBlinking(const bool enable) override;
 

src/host/settings.cpp

@@ -53,6 +53,7 @@ Settings::Settings() :
     _fUseWindowSizePixels(false),
     _fAutoReturnOnNewline(true), // the historic Windows behavior defaults this to on.
     _fRenderGridWorldwide(false), // historically grid lines were only rendered in DBCS codepages, so this is false by default unless otherwise specified.
+    _fScreenReversed(false),
     // window size pixels initialized below
     _fInterceptCopyPaste(0),
     _DefaultForeground(INVALID_COLOR),
@@ -394,6 +395,15 @@ void Settings::SetGridRenderingAllowedWorldwide(const bool fGridRenderingAllowed
     }
 }
 
+bool Settings::IsScreenReversed() const
+{
+    return _fScreenReversed;
+}
+void Settings::SetScreenReversed(const bool fScreenReversed)
+{
+    _fScreenReversed = fScreenReversed;
+}
+
 bool Settings::GetFilterOnPaste() const
 {
     return _fFilterOnPaste;
@@ -942,7 +952,14 @@ COLORREF Settings::CalculateDefaultBackground() const noexcept
 COLORREF Settings::LookupForegroundColor(const TextAttribute& attr) const noexcept
 {
     const auto tableView = std::basic_string_view<COLORREF>(&GetColorTable()[0], GetColorTableSize());
-    return attr.CalculateRgbForeground(tableView, CalculateDefaultForeground(), CalculateDefaultBackground());
+    if (_fScreenReversed)
+    {
+        return attr.CalculateRgbBackground(tableView, CalculateDefaultForeground(), CalculateDefaultBackground());
+    }
+    else
+    {
+        return attr.CalculateRgbForeground(tableView, CalculateDefaultForeground(), CalculateDefaultBackground());
+    }
 }
 
 // Method Description:
@@ -955,7 +972,14 @@ COLORREF Settings::LookupForegroundColor(const TextAttribute& attr) const noexce
 COLORREF Settings::LookupBackgroundColor(const TextAttribute& attr) const noexcept
 {
     const auto tableView = std::basic_string_view<COLORREF>(&GetColorTable()[0], GetColorTableSize());
-    return attr.CalculateRgbBackground(tableView, CalculateDefaultForeground(), CalculateDefaultBackground());
+    if (_fScreenReversed)
+    {
+        return attr.CalculateRgbForeground(tableView, CalculateDefaultForeground(), CalculateDefaultBackground());
+    }
+    else
+    {
+        return attr.CalculateRgbBackground(tableView, CalculateDefaultForeground(), CalculateDefaultBackground());
+    }
 }
 
 bool Settings::GetCopyColor() const noexcept

src/host/settings.hpp

@@ -52,6 +52,9 @@ class Settings
     bool IsGridRenderingAllowedWorldwide() const;
     void SetGridRenderingAllowedWorldwide(const bool fGridRenderingAllowed);
 
+    bool IsScreenReversed() const;
+    void SetScreenReversed(const bool fScreenReversed);
+
     bool GetFilterOnPaste() const;
     void SetFilterOnPaste(const bool fFilterOnPaste);
 
@@ -231,6 +234,7 @@ class Settings
     DWORD _dwVirtTermLevel;
     bool _fAutoReturnOnNewline;
     bool _fRenderGridWorldwide;
+    bool _fScreenReversed;
     bool _fUseDx;
     bool _fCopyColor;
 

src/host/ut_host/ScreenBufferTests.cpp

@@ -182,6 +182,7 @@ class ScreenBufferTests
 
     TEST_METHOD(ScrollLines256Colors);
 
+    TEST_METHOD(SetScreenMode);
     TEST_METHOD(SetOriginMode);
 
     TEST_METHOD(HardResetBuffer);
@@ -4501,6 +4502,34 @@ void ScreenBufferTests::ScrollLines256Colors()
     }
 }
 
+void ScreenBufferTests::SetScreenMode()
+{
+    auto& gci = ServiceLocator::LocateGlobals().getConsoleInformation();
+    auto& si = gci.GetActiveOutputBuffer();
+    auto& stateMachine = si.GetStateMachine();
+
+    const auto rgbForeground = RGB(12, 34, 56);
+    const auto rgbBackground = RGB(78, 90, 12);
+    const auto testAttr = TextAttribute{ rgbForeground, rgbBackground };
+
+    Log::Comment(L"By default the screen mode is normal.");
+    VERIFY_IS_FALSE(gci.IsScreenReversed());
+    VERIFY_ARE_EQUAL(rgbForeground, gci.LookupForegroundColor(testAttr));
+    VERIFY_ARE_EQUAL(rgbBackground, gci.LookupBackgroundColor(testAttr));
+
+    Log::Comment(L"When DECSCNM is set, background and foreground colors are switched.");
+    stateMachine.ProcessString(L"\x1B[?5h");
+    VERIFY_IS_TRUE(gci.IsScreenReversed());
+    VERIFY_ARE_EQUAL(rgbBackground, gci.LookupForegroundColor(testAttr));
+    VERIFY_ARE_EQUAL(rgbForeground, gci.LookupBackgroundColor(testAttr));
+
+    Log::Comment(L"When DECSCNM is reset, the colors are normal again.");
+    stateMachine.ProcessString(L"\x1B[?5l");
+    VERIFY_IS_FALSE(gci.IsScreenReversed());
+    VERIFY_ARE_EQUAL(rgbForeground, gci.LookupForegroundColor(testAttr));
+    VERIFY_ARE_EQUAL(rgbBackground, gci.LookupBackgroundColor(testAttr));
+}
+
 void ScreenBufferTests::SetOriginMode()
 {
     auto& gci = ServiceLocator::LocateGlobals().getConsoleInformation();

src/terminal/adapter/DispatchTypes.hpp

@@ -82,6 +82,7 @@ namespace Microsoft::Console::VirtualTerminal::DispatchTypes
     {
         DECCKM_CursorKeysMode = 1,
         DECCOLM_SetNumberOfColumns = 3,
+        DECSCNM_ScreenMode = 5,
         DECOM_OriginMode = 6,
         ATT610_StartCursorBlink = 12,
         DECTCEM_TextCursorEnableMode = 25,

src/terminal/adapter/ITermDispatch.hpp

@@ -54,6 +54,7 @@ class Microsoft::Console::VirtualTerminal::ITermDispatch
     virtual bool SetCursorKeysMode(const bool applicationMode) = 0; // DECCKM
     virtual bool SetKeypadMode(const bool applicationMode) = 0; // DECKPAM, DECKPNM
     virtual bool EnableCursorBlinking(const bool enable) = 0; // ATT610
+    virtual bool SetScreenMode(const bool reverseMode) = 0; //DECSCNM
     virtual bool SetOriginMode(const bool relativeMode) = 0; // DECOM
     virtual bool SetTopBottomScrollingMargins(const size_t topMargin, const size_t bottomMargin) = 0; // DECSTBM
     virtual bool WarningBell() = 0; // BEL

src/terminal/adapter/adaptDispatch.cpp

@@ -933,6 +933,9 @@ bool AdaptDispatch::_PrivateModeParamsHelper(const DispatchTypes::PrivateModePar
     case DispatchTypes::PrivateModeParams::DECCOLM_SetNumberOfColumns:
         success = _DoDECCOLMHelper(enable ? DispatchTypes::s_sDECCOLMSetColumns : DispatchTypes::s_sDECCOLMResetColumns);
         break;
+    case DispatchTypes::PrivateModeParams::DECSCNM_ScreenMode:
+        success = SetScreenMode(enable);
+        break;
     case DispatchTypes::PrivateModeParams::DECOM_OriginMode:
         // The cursor is also moved to the new home position when the origin mode is set or reset.
         success = SetOriginMode(enable) && CursorPosition(1, 1);
@@ -1079,6 +1082,18 @@ bool AdaptDispatch::DeleteLine(const size_t distance)
     return _pConApi->DeleteLines(distance);
 }
 
+// Routine Description:
+// - DECSCNM - Sets the screen mode to either normal or reverse.
+//    When in reverse screen mode, the background and foreground colors are switched.
+// Arguments:
+// - reverseMode - set to true to enable reverse screen mode, false for normal mode.
+// Return Value:
+// - True if handled successfully. False otherwise.
+bool AdaptDispatch::SetScreenMode(const bool reverseMode)
+{
+    return _pConApi->PrivateSetScreenMode(reverseMode);
+}
+
 // Routine Description:
 // - DECOM - Sets the cursor addressing origin mode to relative or absolute.
 //    When relative, line numbers start at top margin of the user-defined scrolling region.
@@ -1475,6 +1490,12 @@ bool AdaptDispatch::HardReset()
         success = _EraseScrollback();
     }
 
+    // Set the DECSCNM screen mode back to normal.
+    if (success)
+    {
+        success = SetScreenMode(false);
+    }
+
     // Cursor to 1,1 - the Soft Reset guarantees this is absolute
     if (success)
     {

src/terminal/adapter/adaptDispatch.hpp

@@ -68,6 +68,7 @@ namespace Microsoft::Console::VirtualTerminal
         bool SetCursorKeysMode(const bool applicationMode) override; // DECCKM
         bool SetKeypadMode(const bool applicationMode) override; // DECKPAM, DECKPNM
         bool EnableCursorBlinking(const bool enable) override; // ATT610
+        bool SetScreenMode(const bool reverseMode) override; //DECSCNM
         bool SetOriginMode(const bool relativeMode) noexcept override; // DECOM
         bool SetTopBottomScrollingMargins(const size_t topMargin,
                                           const size_t bottomMargin) override; // DECSTBM

src/terminal/adapter/conGetSet.hpp

@@ -57,6 +57,8 @@ namespace Microsoft::Console::VirtualTerminal
         virtual bool PrivateSetCursorKeysMode(const bool applicationMode) = 0;
         virtual bool PrivateSetKeypadMode(const bool applicationMode) = 0;
 
+        virtual bool PrivateSetScreenMode(const bool reverseMode) = 0;
+
         virtual bool PrivateShowCursor(const bool show) = 0;
         virtual bool PrivateAllowCursorBlinking(const bool enable) = 0;
 

src/terminal/adapter/termDispatch.hpp

@@ -48,6 +48,7 @@ class Microsoft::Console::VirtualTerminal::TermDispatch : public Microsoft::Cons
     bool SetCursorKeysMode(const bool /*applicationMode*/) noexcept override { return false; } // DECCKM
     bool SetKeypadMode(const bool /*applicationMode*/) noexcept override { return false; } // DECKPAM, DECKPNM
     bool EnableCursorBlinking(const bool /*enable*/) noexcept override { return false; } // ATT610
+    bool SetScreenMode(const bool /*reverseMode*/) noexcept override { return false; } //DECSCNM
     bool SetOriginMode(const bool /*relativeMode*/) noexcept override { return false; }; // DECOM
     bool SetTopBottomScrollingMargins(const size_t /*topMargin*/, const size_t /*bottomMargin*/) noexcept override { return false; } // DECSTBM
     bool WarningBell() noexcept override { return false; } // BEL

src/terminal/adapter/ut_adapter/adapterTest.cpp

@@ -162,6 +162,13 @@ class TestGetSet final : public ConGetSet
         return _privateSetKeypadModeResult;
     }
 
+    bool PrivateSetScreenMode(const bool /*reverseMode*/) override
+    {
+        Log::Comment(L"PrivateSetScreenMode MOCK called...");
+
+        return true;
+    }
+
     bool PrivateShowCursor(const bool show) override
     {
         Log::Comment(L"PrivateShowCursor MOCK called...");

src/terminal/parser/ut_parser/OutputEngineTest.cpp

@@ -678,6 +678,7 @@ class StatefulDispatch final : public TermDispatch
         _isAltBuffer{ false },
         _cursorKeysMode{ false },
         _cursorBlinking{ true },
+        _isScreenModeReversed{ false },
         _isOriginModeRelative{ false },
         _warningBell{ false },
         _carriageReturn{ false },
@@ -857,6 +858,9 @@ class StatefulDispatch final : public TermDispatch
         case DispatchTypes::PrivateModeParams::DECCOLM_SetNumberOfColumns:
             fSuccess = SetColumns(static_cast<size_t>(fEnable ? DispatchTypes::s_sDECCOLMSetColumns : DispatchTypes::s_sDECCOLMResetColumns));
             break;
+        case DispatchTypes::PrivateModeParams::DECSCNM_ScreenMode:
+            fSuccess = SetScreenMode(fEnable);
+            break;
         case DispatchTypes::PrivateModeParams::DECOM_OriginMode:
             // The cursor is also moved to the new home position when the origin mode is set or reset.
             fSuccess = SetOriginMode(fEnable) && CursorPosition(1, 1);
@@ -920,6 +924,12 @@ class StatefulDispatch final : public TermDispatch
         return true;
     }
 
+    bool SetScreenMode(const bool reverseMode) noexcept override
+    {
+        _isScreenModeReversed = reverseMode;
+        return true;
+    }
+
     bool SetOriginMode(const bool fRelativeMode) noexcept override
     {
         _isOriginModeRelative = fRelativeMode;
@@ -999,6 +1009,7 @@ class StatefulDispatch final : public TermDispatch
     bool _isAltBuffer;
     bool _cursorKeysMode;
     bool _cursorBlinking;
+    bool _isScreenModeReversed;
     bool _isOriginModeRelative;
     bool _warningBell;
     bool _carriageReturn;
@@ -1290,6 +1301,25 @@ class StateMachineExternalTest final
         pDispatch->ClearState();
     }
 
+    TEST_METHOD(TestScreenMode)
+    {
+        auto dispatch = std::make_unique<StatefulDispatch>();
+        auto pDispatch = dispatch.get();
+        auto engine = std::make_unique<OutputStateMachineEngine>(std::move(dispatch));
+        StateMachine mach(std::move(engine));
+
+        mach.ProcessString(L"\x1b[?5h");
+        VERIFY_IS_TRUE(pDispatch->_isScreenModeReversed);
+
+        pDispatch->ClearState();
+        pDispatch->_isScreenModeReversed = true;
+
+        mach.ProcessString(L"\x1b[?5l");
+        VERIFY_IS_FALSE(pDispatch->_isScreenModeReversed);
+
+        pDispatch->ClearState();
+    }
+
     TEST_METHOD(TestOriginMode)
     {
         auto dispatch = std::make_unique<StatefulDispatch>();