Feature: mjpeg streaming for terminal control background

[sipsorcery]

This PR adds a rudimentary mjpeg receiver to allow the terminal control background to display a stream.

Update:

Motivation for this feature

The example screenshots and video are a little bit gimmicky but there could be some very useful scenarios for the ability to employ an information rich terminal background:

• Network monitoring tools could cause the background to display results, e.g. a ping could cause a host in a network map to change colour,

• Quick video calls (my own goal). For developers who spend a lot of time in a console it's advantageous to avoid context switches to other applications for minor tasks. If a quick video call could be done "in-terminal" that could be a small efficiency gain,

• Infrequent graphical notifications. A video stream of 1 frame/second or less could be used to show notifications for configurable events. Again this could be a context switch saving for people that permanently have a terminal open. GitHub notifications would be a good use case. I do 90% of my git interactions from the command line. If the results of commit CI jobs were automatically displayed in the terminal background (that I keep open solely for git interactions) it could save me reading an email and visiting a web page.

It's perhaps easier to illustrate than to explain. The two screenshots below show stills of two different streams. Small mp4 captures are here and here (the second one's a bit spooky).

To test out the feature the steps are:

• Set the backgroundImage to a file path with ;mjpeg:1234 appended. The 1234 represents a loopback port that the mjpeg receiving socket will listen on. For example:

"backgroundImage": "C:/Users/aaron/Pictures/gravatar_400x400.png;mjpeg:10010"

• Start a source stream to the listening port. For example:

ffmpeg -re -f lavfi -i testsrc=size=1280x720:rate=10 -vf "drawtext=fontfile=/Windows/Fonts/arial.ttf: expansion=normal: text='%{localtime}': r=25: x=1100: y=50: fontcolor=black" -vcodec mjpeg -pix_fmt yuvj420p -huffman 0 -f rtp rtp://127.0.0.1:10010

Known Issues:

• Only one terminal control can listen on a port at a time so starting a second terminal with the same backgroundImage setting will not be able to create a listening socket.

[github-actions]

New misspellings found, please review:

• Clauson
• closesocket
• CSRC
• ctime
• deserialise
• deserialised
• DGRAM
• DHP
• dht
• DQT
• dri
• EOI
• ffmpeg
• hsample
• htonl
• htons
• huffman
• icase
• INADDR
• inband
• INET
• iphlpapi
• IPPROTO
• IPv
• IStorage
• jfif
• jpegtables
• libavcodec
• libavformat
• MAKEWORD
• mbz
• mjpeg
• posn
• qtable
• recv
• recvfrom
• rtp
• rtpdec
• rtpsocket
• shutsdown
• sipsorcery
• sockaddr
• SOI
• stdint
• tcpip
• timeval
• usec
• vsample
• winsock
• wsa
• WSADATA

To accept these changes, run the following commands

perl -e '
my @expect_files=qw('".github/actions/spell-check/whitelist/alphabet.txt
.github/actions/spell-check/whitelist/web.txt
.github/actions/spell-check/whitelist/whitelist.txt"');
@ARGV=@expect_files;
my @stale=qw('"AAD abcd abe abec acb afae afceeeaa baac bbd bca bcb bcc bda bfb BGRA bitfield caa carlos cbb dcf ddb dfa eba ebce EBFB ECFB eeb eee Emoji Emojis fbb fbd FBE fcc fdb fdd ffc HREF memcpying OUTPATHROOT PREMULTIPLIED storageitems textblock usr vpack zamora "');
my $re=join "|", @stale;
my $suffix=".".time();
my $previous="";
sub maybe_unlink { unlink($_[0]) if $_[0]; }
while (<>) {
  if ($ARGV ne $old_argv) { maybe_unlink($previous); $previous="$ARGV$suffix"; rename($ARGV, $previous); open(ARGV_OUT, ">$ARGV"); select(ARGV_OUT); $old_argv = $ARGV; }
  next if /^($re)(?:$| .*)/; print;
}; maybe_unlink($previous);'
perl -e '
my $new_expect_file=".github/actions/spell-check/whitelist/eacbc106c4603f6128d726ec0fdcb1cbfc6cab94.txt";
open FILE, q{<}, $new_expect_file; chomp(my @words = <FILE>); close FILE;
my @add=qw('"ABE bgra Clauson closesocket CSRC ctime deserialise deserialised DGRAM DHP dht DQT dri EEB EOI FDD ffmpeg hsample htonl htons huffman icase INADDR inband INET iphlpapi IPPROTO IPv IStorage jfif jpegtables libavcodec libavformat MAKEWORD mbz mjpeg posn premultiplied qtable recv recvfrom rtp rtpdec rtpsocket shutsdown sipsorcery sockaddr SOI stdint tcpip timeval usec vsample winsock wsa WSADATA "');
my %items; @items{@words} = @words x (1); @items{@add} = @add x (1);
@words = sort {lc($a) cmp lc($b)} keys %items;
open FILE, q{>}, $new_expect_file; for my $word (@words) { print FILE "$word\n" if $word =~ /\w/; };
close FILE;'
git add .github/actions/spell-check/whitelist || echo '... you want to ensure .github/actions/spell-check/whitelist/eacbc106c4603f6128d726ec0fdcb1cbfc6cab94.txt is added to your repository...'

✏️ Contributor please read this

• If the items listed above are names, please add them to .github/actions/spell-check/dictionary/names.txt.
• If they're APIs, you can add them to a file in .github/actions/spell-check/dictionary/.
• If they're just things you're using, please add them to an appropriate file in .github/actions/spell-check/whitelist/.
• If you need to use a specific token in one place and it shouldn't generally be used, you can
  add an item in an appropriate file in .github/actions/spell-check/patterns/.

See the README.md in each directory for more information.

⚠️ Reviewers

At present, the action that triggered this message will not show its ❌ in this PR unless the branch is within this repository.
Thus, you should make sure that this comment has been addressed before encouraging the merge bot to merge this PR.

[jsoref]

You're going to want to rebase and push before you take advice from the bot, otherwise you're going to hit a conflict.

The conflicts shouldn't be hard to resolve whitelist was renamed to expect -- If you need help, I can help on Sunday.

[github-actions]

New misspellings found, please review:

• Clauson
• closesocket
• CSRC
• ctime
• deserialise
• deserialised
• DGRAM
• DHP
• dht
• DQT
• dri
• EOI
• fd
• ffmpeg
• hsample
• htonl
• htons
• huffman
• icase
• INADDR
• inband
• INET
• iphlpapi
• IPPROTO
• IPv
• jfif
• jpegtables
• libavcodec
• libavformat
• MAKEWORD
• mbz
• mjpeg
• posn
• qtable
• recv
• recvfrom
• rtp
• rtpdec
• rtpsocket
• shutsdown
• sipsorcery
• sockaddr
• SOI
• stdint
• tcpip
• timeval
• usec
• vsample
• winsock
• wsa
• WSADATA

To accept these changes, run the following commands

perl -e '
my @expect_files=qw('".github/actions/spell-check/expect/alphabet.txt
.github/actions/spell-check/expect/expect.txt
.github/actions/spell-check/expect/web.txt"');
@ARGV=@expect_files;
my @stale=qw('"atg BGRA BKMK consoleaccessibility PREMULTIPLIED shobjidl "');
my $re=join "|", @stale;
my $suffix=".".time();
my $previous="";
sub maybe_unlink { unlink($_[0]) if $_[0]; }
while (<>) {
  if ($ARGV ne $old_argv) { maybe_unlink($previous); $previous="$ARGV$suffix"; rename($ARGV, $previous); open(ARGV_OUT, ">$ARGV"); select(ARGV_OUT); $old_argv = $ARGV; }
  next if /^($re)(?:$| .*)/; print;
}; maybe_unlink($previous);'
perl -e '
my $new_expect_file=".github/actions/spell-check/expect/591465ef359ea29860bc95328fab3a7e5fe7c802.txt";
open FILE, q{<}, $new_expect_file; chomp(my @words = <FILE>); close FILE;
my @add=qw('"bgra Clauson closesocket CSRC ctime deserialise deserialised DGRAM DHP dht DQT dri EOI fd ffmpeg hsample htonl htons huffman icase INADDR inband INET iphlpapi IPPROTO IPv jfif jpegtables libavcodec libavformat MAKEWORD mbz mjpeg posn premultiplied qtable recv recvfrom rtp rtpdec rtpsocket Shobjidl shutsdown sipsorcery sockaddr SOI stdint tcpip timeval usec vsample winsock wsa WSADATA "');
my %items; @items{@words} = @words x (1); @items{@add} = @add x (1);
@words = sort {lc($a) cmp lc($b)} keys %items;
open FILE, q{>}, $new_expect_file; for my $word (@words) { print FILE "$word\n" if $word =~ /\w/; };
close FILE;'
git add .github/actions/spell-check/expect || echo '... you want to ensure .github/actions/spell-check/expect/591465ef359ea29860bc95328fab3a7e5fe7c802.txt is added to your repository...'

✏️ Contributor please read this

By default the command suggestion will generate a file named based on your commit. That's generally ok as long as you add the file to your commit. Someone can reorganize it later.

⚠️ The command is written for posix shells. You can copy the contents of each perl command excluding the outer ' marks and dropping any '"/"' quotation mark pairs into a file and then run perl file.pl from the root of the repository to run the code. Alternatively, you can manually insert the items...

If the listed items are:

• ... misspelled, then please correct them instead of using the command.
• ... names, please add them to .github/actions/spell-check/dictionary/names.txt.
• ... APIs, you can add them to a file in .github/actions/spell-check/dictionary/.
• ... just things you're using, please add them to an appropriate file in .github/actions/spell-check/expect/.
• ... tokens you only need in one place and shouldn't generally be used, you can add an item in an appropriate file in .github/actions/spell-check/patterns/.

See the README.md in each directory for more information.

🔬 You can test your commits without appending to a PR by creating a new branch with that extra change and pushing it to your fork. The :check-spelling action will run in response to your push -- it doesn't require an open pull request. By using such a branch, you can limit the number of typos your peers see you make. 😉

⚠️ Reviewers

At present, the action that triggered this message will not show its ❌ in this PR unless the branch is within this repository.
Thus, you should make sure that this comment has been addressed before encouraging the merge bot to merge this PR.

[github-actions]

New misspellings found, please review:

• closesocket
• CSRC
• ctime
• deserialise
• deserialised
• DGRAM
• DHP
• dht
• DQT
• dri
• EOI
• fd
• ffmpeg
• hsample
• htonl
• htons
• huffman
• icase
• INADDR
• inband
• INET
• iphlpapi
• IPPROTO
• IPv
• jfif
• jpegtables
• libavcodec
• libavformat
• MAKEWORD
• mbz
• mjpeg
• posn
• qtable
• recv
• recvfrom
• rtp
• rtpdec
• rtpsocket
• shutsdown
• sipsorcery
• sockaddr
• SOI
• stdint
• tcpip
• timeval
• usec
• vsample
• winsock
• wsa
• WSADATA

To accept these changes, run the following commands

perl -e '
my @expect_files=qw('".github/actions/spell-check/expect/alphabet.txt
.github/actions/spell-check/expect/expect.txt
.github/actions/spell-check/expect/web.txt"');
@ARGV=@expect_files;
my @stale=qw('"atg BGRA BKMK consoleaccessibility PREMULTIPLIED shobjidl "');
my $re=join "|", @stale;
my $suffix=".".time();
my $previous="";
sub maybe_unlink { unlink($_[0]) if $_[0]; }
while (<>) {
  if ($ARGV ne $old_argv) { maybe_unlink($previous); $previous="$ARGV$suffix"; rename($ARGV, $previous); open(ARGV_OUT, ">$ARGV"); select(ARGV_OUT); $old_argv = $ARGV; }
  next if /^($re)(?:$| .*)/; print;
}; maybe_unlink($previous);'
perl -e '
my $new_expect_file=".github/actions/spell-check/expect/9c67742e0698c991de13eac4326833e48448acc7.txt";
open FILE, q{<}, $new_expect_file; chomp(my @words = <FILE>); close FILE;
my @add=qw('"bgra closesocket CSRC ctime deserialise deserialised DGRAM DHP dht DQT dri EOI fd ffmpeg hsample htonl htons huffman icase INADDR inband INET iphlpapi IPPROTO IPv jfif jpegtables libavcodec libavformat MAKEWORD mbz mjpeg posn premultiplied qtable recv recvfrom rtp rtpdec rtpsocket Shobjidl shutsdown sipsorcery sockaddr SOI stdint tcpip timeval usec vsample winsock wsa WSADATA "');
my %items; @items{@words} = @words x (1); @items{@add} = @add x (1);
@words = sort {lc($a) cmp lc($b)} keys %items;
open FILE, q{>}, $new_expect_file; for my $word (@words) { print FILE "$word\n" if $word =~ /\w/; };
close FILE;'
git add .github/actions/spell-check/expect || echo '... you want to ensure .github/actions/spell-check/expect/9c67742e0698c991de13eac4326833e48448acc7.txt is added to your repository...'

✏️ Contributor please read this

By default the command suggestion will generate a file named based on your commit. That's generally ok as long as you add the file to your commit. Someone can reorganize it later.

⚠️ The command is written for posix shells. You can copy the contents of each perl command excluding the outer ' marks and dropping any '"/"' quotation mark pairs into a file and then run perl file.pl from the root of the repository to run the code. Alternatively, you can manually insert the items...

If the listed items are:

• ... misspelled, then please correct them instead of using the command.
• ... names, please add them to .github/actions/spell-check/dictionary/names.txt.
• ... APIs, you can add them to a file in .github/actions/spell-check/dictionary/.
• ... just things you're using, please add them to an appropriate file in .github/actions/spell-check/expect/.
• ... tokens you only need in one place and shouldn't generally be used, you can add an item in an appropriate file in .github/actions/spell-check/patterns/.

See the README.md in each directory for more information.

🔬 You can test your commits without appending to a PR by creating a new branch with that extra change and pushing it to your fork. The :check-spelling action will run in response to your push -- it doesn't require an open pull request. By using such a branch, you can limit the number of typos your peers see you make. 😉

⚠️ Reviewers

At present, the action that triggered this message will not show its ❌ in this PR unless the branch is within this repository.
Thus, you should make sure that this comment has been addressed before encouraging the merge bot to merge this PR.

[miniksa]

This is quite interesting. A few quick reads on it:

1. I don't really like that the port thing can conflict. But I can't really think of another transport for this to use that wouldn't conflict. I guess technically the receiver could operate at a higher level above multiple tabs and shunt out the jpegs to any tab subscribed so you can have the same content on multiple.
2. I would really hope that there would be some lower-level registration inside the XAML/MUX platform that could pick up the streams and update the jpegs instead of pushing them into the property from the top on a loop. But maybe that sort of extensibility doesn't exist?
3. I generally like the idea. I was wondering at some point if we couldn't just have a full blown mp4 stream running as the background to feed in whatever live content you want. This is definitely in that vein.
4. I'm concerned about the licensing/derivation on mjpeg.h. We would have to have that reviewed.

I want to discuss this with @zadjii-msft and @DHowett before going much further forward on this.

[sipsorcery]

Thanks for taking a look.

1. I don't really like that the port thing can conflict. But I can't really think of another transport for this to use that wouldn't conflict. I guess technically the receiver could operate at a higher level above multiple tabs and shunt out the jpegs to any tab subscribed so you can have the same content on multiple.

I can think of a few ways to deal with it but they increase the scope of the PR and the original intent was to see if this feature was likely to be accepted before adding extras. If it was a blocker I could fix it.

Solutions off the top of my head (in increasing order of scope creep):

• Use an OS ephemeral port for each pane by binding against port 0. Once the socket has bound print it out above the terminal pane's initial prompt,
• Same as above but rather than printing to the terminal pane add a key binding command that can print out the listening socket for the current pane,
• Integrate a lightweight ASP.Net Web API that can be queried to list the available terminal panes and their listening sockets.

I don't think the shunting idea would be ideal. Displaying the same mjpeg feed in multiple panes doesn't seem that useful and could get annoying depending on the feed.

2. I would really hope that there would be some lower-level registration inside the XAML/MUX platform that could pick up the streams and update the jpegs instead of pushing them into the property from the top on a loop. But maybe that sort of extensibility doesn't exist?

If there is such a property I've never found it and if it does exist I'd love to hear about it. I have a demo VoIP softphone application that's been floating around for years and I've always updated the video stream using the exact same approach of setting a bitmap image source.

3. I generally like the idea. I was wondering at some point if we couldn't just have a full blown mp4 stream running as the background to feed in whatever live content you want. This is definitely in that vein.

This PR goes further than streaming an mp4 or other static content (which can already be achieved with the existing MediaPlayElement) and allows the streaming of live content. The simplest live content is mjpeg but if that was incorporated and useful then it could be the initial step in VoIP, WebRTC and other live streaming sources.

4. I'm concerned about the licensing/derivation on mjpeg.h. We would have to have that reviewed.

Yes same, hence the header comments. It's always a quandary when "leveraging" existing source code. I do think in this case the scope of the changes I've made to the various ffmpeg source blocks make the mjpeg.h a new work. Plus the ffmpeg sources are C and I've employed C++ idioms. The Huffman tables are part of the JPEG spec so they wouldn't count as GPL licensed. If any other large blocks were of concern I could most likely re-write them.

[WSLUser]

@DHowett Has there been any progress for the legal aspects of this PR? I'd love to start using this.

[zadjii-msft]

Hey, sorry it's been like half a year since we took a look at this. We discussed this a bit as a team recently. The consensus was that as neat as this feature is, we're not really comfortable shipping it directly in the Terminal. It probably makes the most sense as an extension (#4000), but we're not confident that we could support this adequately in the Terminal itself.

I've added this to the mega list of extension ideas in #4000. We'll make sure to keep this PR in mind when we do get to extensions, because it's a GREAT example of something that developers should be able to do.

Sorry again for leaving you on Read for so long!