Scenario: Add support for 3rd-party extensions

[zadjii-msft]

Updated thread from #555

Now that we have a better understanding of what work will be needed to support 3rd party extensions ("plugins"), this thread is the megathread to track all the work that's needed to support them.

Before any real work can begin here, we're waiting on some support from the Windows operating system that will be coming Soon^TM. Fundamentally, third-party code extensions for packaged applications (like us) isn't supported currently.

This is not work slated for 1.0. It may not even land in 2.0. This work will likely span multiple releases. Whenever this work does end up starting formally, we'll likely have a number of minor versions with prototypes of the extensibility model, with breaking changes up until the major version release of the Terminal. So if extensions are scheduled for 3.0, then 2.4 might introduce bts of the extensibility model that are later broken in 2.5, 2.6, etc.

What should extensions be able to do?

• Add their own profiles via Dynamic Profile Generators
• Customize the rendering of the Terminal, with things like:
  • custom shaders, font renderers
  • custom formatting of the buffer (like hyperlinks)
• Add their own Color Schemes, keymaps
• Modify the UI theme (for adjusting appearance of tabs, scrollbars, etc.)
• Modify the layout of the UI
  • Add elements to the UI like status bars, toolbars, minimaps
  • reposition existing elements of the UI? Tabs on bottom for example.
• Create Panes with their own UI elements
• Read the contents of the text buffer, for parsing things like hyperlinks
• When extensions modify the UI, they'll need to adjust how large the initial window size is
• Add commands to the Command Palette, including nested commands
• Add commands to various context menus (such as the TermControl context menu, TabViewItem menu).

Work needed to support extensions

Some of this work is more directly related to extensions, while others on the list are simply adding features to the terminal to provide points for extensibility in the future.

• Make Profile, ColorScheme, GlobalAppSettings proper WinRT types Make Profile, ColorScheme, GlobalAppSettings proper WinRT types #3998
• Make Tab, Pane proper WinRT types Make Pane a proper WinRT type #3999, Bind TerminalPage's TabView to TerminalPage::_tabs #3922
• Support Panes with non-terminal content A pane doesn't necessarily need to host a terminal. #997 (spec: First draft of a spec for panes with non-terminal content #1080)
• Add an optional right-click menu to the TermControl (Request: Right-click menu inside TerminalControl (w/ Copy & Paste?) #3337)
• Add the command palette Feature Request: Command Palette #2046 (spec: Add a spec for the Command Palette #2193)
• Enable nested commands in the command palette Add nested commands to the Command Palette #3994
• Add comprehensive "XAML" theming functionality Add comprehensive XAML "theming" functionality #3327
• Design Extensibility model - this covers actually figuring out the API surface by which developers can write extensions for. This is most of the work
  • This will involve developing a couple extensions in-house and partnering with some community members to explore extensibility model

Mega-list of extension ideas

Dynamic Profiles / settings

• Add Developer PowerShell and Developer Command Prompt Generator #3821 Add Developer PowerShell and Developer Command Prompt Generator
• Upon installation, add a Git Bash profile, if Git for Windows is present #1394 Upon installation, add a Git Bash profile, if Git for Windows is present
• Feature Request: Dedicated serial port connection settings #1280 Feature Request SSH-Telnet-Serialport connection
• Device portal in Windows Terminal #5049 Device Portal
• Include a profile that will connect to Visual Studio Codespaces #5900 Include a profile that will connect to Visual Studio Codespaces
• Add WSL distro default desktop wallpapers as default background #6105 Add WSL distro default desktop wallpapers as default background
• Extension: Ask for parameter for profile #8339 "ask parameter" for profile. profile with variable that is prompted-for
• Auto generate profile from .ssh/config #9031 Auto generate profile from .ssh/config
• Importing PuTTY sessions #10943 Importing PuTTY sessions
• [Extension Idea] Password Manager integration #11641
  • This isn't "dynamic profiles", it's more "commands that are provided to the command palette via an API NOT a plaintext file"
• Dynamic Profiles for Hyper-V VMs #12773

Additional Connection types

• Plugin: add support for [XYZ]MODEM file transfers #1999 Plugin: add support for [XYZ]MODEM file transfers
• Suggestion: "One-click & snap" connect to bluetooth/serial devices and network hosts using QR Code and Code 128 #694 Suggestion: "One-click & snap" connect to bluetooth/serial devices and network hosts using QR Code and Code 128
• Add support for tmux Control Mode #3656 Add support for tmux Control Mode
• Support HTM (headless terminal multiplexer) for remote pane/tab management #5321 Support HTM (headless terminal multiplexer) for remote pane/tab management
• Bastion shell similar to azure cloud shell #11064 Bastion shell similar to azure cloud shell
• AWS/GCP Cloud Shell Connector #13245

Buffer Parsing/Manipulation

• Feature Request: Smart Double-click Selection (regeces?) #3196 Feature Request: Smart Double-click Selection (regeces?)
• Feature Request: link generation for files + other data types #2671 Feature Request: link generation for files + other data types
• Design Clickable Links & Link preview feature/extension #574 Design Clickable Links & Link preview feature/extension
• Feature request: Parenthesis matching in text #5226 Parenthesis matching in text
• Add support for syntax highlighting #6297 Highlighting matches in all history
• Triggers(Including Text Strings) and Actions (internal or external calls)  #5916 Triggers(Including Text Strings) and Actions (internal or external calls)
• ITerm2-like terminal autocomplete #6632 ITerm2-like terminal autocomplete
• ability to pipe scrollback buffer to a pager or terminal editor like kitty #7749 ability to pipe scrollback buffer to a pager or temrinal editor

Suggestions

• Allow 3p suggestion sources #17344
  • that pretty much tracks all of it.

UI Elements

App elements

• Add an optional status bar to the bottom #3459 Add an optional status bar to the bottom
• Feature Request: allow to pick the color of the tabs #2994 Feature Request: allow to pick the color of the tabs
  • We actually did this ourselves!
• Feature Request: Icon buttons to start relevant shell types #2934 Feature Request: Icon buttons to start relevant shell types
  • (Maybe something like a toolbar? See also Add Toolbar for WSL OS instances #4531, Favorites/Shortcuts/Quick List #5273, Quick Command as same as [XShell - Quick Command] #7207)
• Feature Request - Scripts Panel #1595 Feature Request - Scripts Panel
  • A panel for saved commands A panel for saved commands #11270
  • Favorites/Shortcuts/Quick List #5273 Favorites/Shortcuts/Quick List
    • Also maybe implies some sort of integration of storing their own settings that are linked to Terminal profiles?
• Feature Request: optional splitting of view into scroll pane and static pane while scrolling #2601 Feature Request: optional splitting of view into scroll pane and static pane while scrolling
• Feature Request: An advanced tab switcher #1502 Feature Request: An advanced tab switcher
  • We actually did this ourselves!
• Feature request: Enable customization for tabs on bottom/right/left #835 Feature request: Enable customization for tabs on bottom
• Feature Request: /help commands output to collapsible pane #644 Feature Request: /help commands output to collapsible pane
• Tab name on fullscreen view #4444 It would be nice to have a kind of popup with a tab name when you switch tabs with CTRL+TAB in fullscreen mode.
  • We actually did this ourselves!
  • Let's add a setting for this as well, this sounds like a not so terrible idea.
• "Peek" button to show screen behind Terminal #5426 "Peek" button to show screen behind Terminal
• Add support for Azure Cloud Shell code Editor #5636 Add support for Azure Cloud Shell code Editor
  • In this case, a particular connection might need to be able to spawn a new pane, with HTML content in it
• Add support for side panels #16387
  • We closed this cause we're so very unlikely to do it, but it's not a bad idea post-Feature request: Enable customization for tabs on bottom/right/left #835
• [feature request] Add distro-specific configuration options in profile configuration (e.g. the login user for WSL profiles) #16420
  • "Extensions should be able to add settings UI elements"
• more...

Control Elements

• Enhance shell autocompletion with a cool new user interface and shell completion protocol #3121 Enhance shell autocompletion with a cool new user interface and shell completion
• Feature Request: ⬇️ button to appear when scrolled many pages up #2471 Feature Request: ⬇️ button to appear when scrolled many pages up
• Feature Request: Scrollable map view for each tab [minimap] #2226 Feature Request: Scrollable map view for each tab [minimap]
• FR: IDE-style marks on scrollbar #1527 FR: IDE-style marks on scrollbar
• Provide a 'Paste mode' to allow a paste keybinding to be 'smart'. #5278 Provide a 'Paste mode' to allow a paste keybinding to be 'smart'.
  • A single keybinding that lets the user chose between "paste", "send ^V", or "Cancel".
  • Might require adding a custom argument to a ShortcutAction we've defined? Or its own ShortcutAction.
• FPS overlay for performance debugging #6025 FPS overlay for performance debugging
• Multi-click selection granularity #6511 Control how multi-click selection operates
• Feature: Set text to display in the background of a terminal instead of just an image #6979 Have text as the background instead of just an image
• mjpeg streaming for terminal control background Feature: mjpeg streaming for terminal control background #6242
• Feature Request: Have desktop image shine through #6497
• Background image alignment relative to desktop #15837
  • similar to the above
• Display a indicator on the cursor's line Display a indicator on the cursor's line #9993
• search web for highlighted text feature? #10175
  • Hey we did this in 1.19!
• rainbow cursor? Extensions should support a very cool rainbow cursor effect #10442
• Show and allow editing of typeahead Show and allow editing of typeahead #10690
• Paste with promptable sections (powered by AI) #15104

Other Elements

• Opening Terminal inside Windows Explorer as a pane (or vice-versa) #5591 Opening the windows terminal inside Windows Explorer rather than from windows Explorer
• Feature request: Background image for tab, not per split pane #6028 Background image for tab, not per split pane

Custom Rendering

• Feature request: 'age' output visually  #3520 Feature request: 'age' output visually
• Glow text exploration #781 Glow text exploration
• Feature Request: Allow background image to be pixel/fragment shader specified by me #7013 Allow background image to be pixel/fragment shader specified by me
• Add built-in D2d effects  #7380 Add built-in D2d effects
• 👁🍬 Smooth cursor animation/transition when moving  #9221 Smooth cursor animation/transition when moving

Advanced Settings

• Rolling background image like Windows Themes #3900 Rolling background image like Windows Themes
• Change tab colour automatically based on tab title text using preset regex in settings #7906 Change tab colour automatically based on tab title text using preset regex in settings

Miscellaneous

• Feature Request - Live Share support #2516 Feature Request - Live Share support
• Feature request: Recording #469 Feature request: Recording
  • GIF Capture tool for the Windows Terminal [Feature Request ⚙] GIF Capture tool for the Windows Terminal  #8098
• Voice Command  #5434 Voice Command
• Notification after a long running command finishes #6372 Notification after a long running command finishes
• Feature Request: Randomize background images from an array of filesystem paths #7279 Randomize background images from an array of filesystem paths
• Auto display readme`s #8647 Auto display readme`s
• Extension: Pipe output to command palette in Terminal #8797 Extension: Pipe output to command palette in Terminal
• More elaborate SSH features (like PuTTY)
  • See: Cluster SSH Features #4653, ssh features #11034

---

@ future me, 10/14/2021

I tried this in dev/migrie/fhl/adaptive-card-extension, but ran into a number of unfixable build issues that made me hate the world. I couldn't get another package built to be able to load that into the terminal at all. Maybe next month I'll try again. For now, it's on to honks.

[WSLUser]

You should add #3468 that needs to be brought into an extension instead of living in the main code base.

[gbritton1]

Why on earth would you need support from the WIndows team? That doesn't apply to VS code, e.g. in any environment. Can't we just do it the same way?

[DHowett]

@gbritton1 Terminal is (for better or for worse) deployed as a "MSIX" package using the new Windows packaging/installer infrastructure; VSCode is not. That infrastructure is missing some features that would allow for:

• "In-process" extensions, which...
• ...support being updated while they're in use, and...
• ...can be installed for multiple users trivially.

Had we chosen to install using MSI, traditionally, into C:\Program Files... yeah, we wouldn't need anything else from the package subsystem. We'd also lose(1) support for easy, atomic updates for the entire Terminal. 🤷🏻

We've since learned that there's a way forward where we don't need in-process extensions, but we do want to make sure that it will perform adequately for things that impact the rendering engine or the text buffer.

(1): "Lose" insofar as it is easy. We could always write a bunch of code to support it, but on the balance we decided to invest most of our effort in making the Terminal and Console Host itself better. :)

[gbritton1]

Would it be simpler to change the packaging?

[ntindle]

I'm glad you put this out there. I honestly had the same opinion on why it required updates on the windows side. Thanks for clarifying that.

[zadjii-msft]

Notes comment:

See https://github.com/microsoft/terminal/tree/dev/migrie/fhl/dyndep.

microsoft/WindowsAppSDK#66

Todo

• Find 3p app using extension catalog
• Register it as a dependency using dyndep
• Be able to instantiate a WinRT class implemented in that package which implements an interface we declare.
• Figure out why the SampleApp package always wants to include MUX 2.7 in its package. Just manually deleting its activatableClasses from the Appxmanifest.xml worked but, sheesh
  • I didn't hit this in my public/terminal clone, only the private/Openconsole one. So maybe it's just a dirty repo.
• Actually communicate the class name in the properties of the extension implementation
• Create some sort of "smart dyndep scope" class / struct, to keep the package dependency alive...
  • ... and release the dependency when no longer needed
• Try to get the extension to return a XAML object, created in codebehind
  • 🎉 This is like a pretty big milestone IMO 🎉
• Try to get the extension to return a XAML object, as defined in XAML
  • Possibly impossible: WinUI 3 Cant instantiate a UserControl from a location outside the application package or directory (XamlParseException) microsoft-ui-xaml#6299
  • 950311c
• Try to get the extension to implement a "HackerTyperConnection : ITerminalConnection"
• Clean up the way the extension projects build. Right now they're all bog-standard so they scatter shit all over the build tree. use the common props instead.
• Can an extension that creates a XAML button that gets added to the UI tree then somehow iterate upwards and get at the XamlRoot (and therefore, the whole Terminal UI tree?)
  • Or, acutely, just get at the XAML root even without user interaction like that?
• C#?
  • This was pain
• more...

---

Notes circa May 2023 Hackathon

• DynDep works perfectly.
• You can totally have an extension host instantiate classes that are hosted in another package.
  • You absolutely need to make sure they are declared as activatableClasses in the extension's manifest. This will only happen if you have a separate component (dll) that implements it. Hence ExtensionHost and ExtensionComponent. Component actually does the hard work.
• Extensions can build WinUI types at runtime, and add them to the host's UI: e0c8135
• You can even have the extension return a WebView2! That's awesome. 497fea8
• You need to make sure to keep a reference to the extension class, obviously, but even WinRT events work too: 82dc26f
• C# might be Hard to do. I don't think I understand how it works with WinRT anymore. I was able to convince the tooling that my class implemented the interface from the C++ project, but I don't know how to package it up to make it callable. And IIRC the new C#/WinRT stuff is all WASDK which is all WUI3 which is incompatible with WUI2.

---

Now, the questions are:

• How do we feel about in proc extensions?
  • Kinda a reliability nightmare. They crash? we crash.
  • Their code is literally running in our app context. Proposal: Allow hosting of UWP apps inside UWP apps WindowsAppSDK#66 (comment).
  • Literally no security guarantees anymore. They can do whatever they want.
• What do we want to expose as extensibility points?
  • Additional actions
  • Additional profiles / dynamic profile sources
  • Send actions to the Terminal
  • Custom actions
  • Custom panes / UI surfaces
  • Custom suggestion sources
  • Custom connection types

[jonwis]

@zadjii-msft before you merge this we should chat. See my comment here - microsoft/WindowsAppSDK#66 (comment)

[zadjii-msft]

@jonwis haha don't worry - that was firmly FHL code, not real shipping code. I use FHL to push the boundaries of what the platform will let me do. But I'm absolutely going to make sure there's proper boundaries set up here before doing anything like this in real Terminal code 😆

[jonwis]

(Also, don't get me wrong - I love it when we figure out the power latent in the platform. We've done sooo much here over the years that compiles together well but we don't have enough time or hands to document all the possible combinations. :) )