Streamdeck Plus

[c0nnex]

Recently got my Streamdeck Plus (ProductID 0x0084)
It features 8 normal buttons , 4 ecoders with images (imagesize unknown yet, but rectangle not square)

I'll play with it the next days and capture some reports.

UsbDeview report
report.zip

[c0nnex]

Normal Buttons: JPEG, Size 100x100 (Same palyoad paket as for XL)
Encoders: JPEG, Size 100x200 (Payload Paket Header is 16 Bytes)
Will need some rewrite of the internal code to reflect diffrent x/y dimensions. Might come along with a PR on the weekend

SideNote: I switched to MAGICK.net for all the image composition/resizing etc to bypass windows System.Drawing thread-limitations, but I'll do the PR for Graphics-implementation

[c0nnex]

Got the Streamdeck Plus implemented.
But it's some fiddling, because i did not want to touch OpenMacroBoard.SDK
I'll publish my branch in my Fork later today, and you can have a look if you want to do it that way (and i'llopen a pull request), or want adopt the OpenMacroBoard.SDK
Generally the currently layout of device data handling is not very flexible and i would suggest to redesign it.

[c0nnex]

OK after working deeper into it, it turns out that what i want to do with the plus ( handle encoder-image as one big image, like the SD does) would result in a major rewrite of all handling, so i skipped the OpenMacroBoard approach.
However, if anyone, is interested I'll push my "4x 200x100"-Images OMB-Implementation after code clean up, but it's based on an old branch (non-ImageSharp)

For anyone also working on this, here's the usb payload info:

Input-PayLoad:

           /* byte ReportId - 0x01
             * byte ReportType - 0x00 ButtonReport , 0x03 EncoderReport
             * ushort ReportSize - Number of state bytes following.            
             * <ReportSize> bytes with states. For Encoder Report first byte is Encoder-Turn flag
             * <Padding to ReportLength>
             */

If is EncoderReport AND first DataByte is 0x01:
following Databytes are signed bytes (- CountClockWise + Clockwiset turns, Abs(0 .. 16) , giving the speed of turning)
Else databytes are just 0/1 for release/press of the respective encoder (offset - 1 )

Output-Payload:
All Images for SD Plus (buttons and encoder) are jpeg encoded , not flipped.
Output report for the 8 buttons : identical to Streamdeck XL but 120x120 pixels

Output report for the encoder image (800x100 in total):
This is a "image difference rectangle report"
HeaderSize 16

/* 
0x02 // id
0x0c // Image Rectangle
ushort Left
ushort Top
uShort width
ushort height
byte isFinalPayLoad (0 / 1 )
ushort payLoadIndex (0 ... x)
ushort imagePayloadLength
byte padbyte (0)
<payload>
<padding to 1024>
*/
                var leftPixel = Encoders.KeyWidth * encoderId; // 200 * x ... Elgato itself handles the whole pic as one 800x100 picture and just sends diffrence rectangles. Pretty nice and fast.
                // Report ID
                data[0] = 2; 
                // Image-Rectangle Jpeg
                data[1] = 0x0c; 

                //Left
                data[2] = (byte)(leftPixel & 0xff);
                data[3] = (byte)(leftPixel >> 8);
                // Top
                data[4] = 0;
                data[5] = 0;
                // Width
                data[6] = (byte)(Encoders.KeyWidth & 0xff); // defaults to 200
                data[7] = (byte)(Encoders.KeyWidth >> 8);
                // Height
                data[8] = (byte)(Encoders.KeyHeight & 0xff); // defaults to 100
                data[9] = (byte)(Encoders.KeyHeight >> 8);

                data[10] = (byte)(isLast ? 1 : 0);

                data[11] = (byte)(pageNumber & 0xff);
                data[12] = (byte)(pageNumber >> 8);

                data[13] = (byte)(payloadLength & 255);
                data[14] = (byte)(payloadLength >> 8);

                // Padding
                data[15] = 0;

[Mutex666]

Hi @c0nnex , thanks for your job on the Stream Deck Plus. I just received mine and I would be curious to dig into your code to see how things goes. I looked to implement it myself in the Streamdecksharp code but hardware dialog is quite new for me so I'm struggling a little to see how I can get report of the HID device and so on.

[Necromantic]

Thanks to @c0nnex for his ground work.

I've just implemented this on my side. Had to mess with IMacroBoard, add more event/delegate types, additional hardware information and a few method overloads with different information. It didn't take all that much work, but it's still a bit messy. As was said before this might need to be redesigned to be a bit more dynamic.
I've implemented the touchscreen output with a rectangle (roi) approach independent of the encoders, so I can draw images over any area of any size I want on the screen.

For the additional input reports the information is as follows (reduced to the important data).
Byte 2 (ReportType): 0x00 ButtonReport, 0x02 TouchReport, 0x03 EncoderReport

TouchReport:
Byte 5 (EventType): 0x01 TapEvent, 0x02 HoldEvent, 0x03 SwipeEvent

TapEvent:
Byte 7 & 8: Touch X-Position in pixels (unsigned short)
Byte 9 & 10: Touch Y-Position in pixels (unsigned short)

HoldEvent:
Byte 7 & 8: Touch X-Position in pixels (unsigned short)
Byte 9 & 10: Touch Y-Position in pixels (unsigned short)

SwipeEvent:
Byte 7 & 8: Start X-Position in pixels (unsigned short)
Byte 9 & 10: Start Y-Position in pixels (unsigned short)
Byte 11 & 12: End X-Position in pixels (unsigned short)
Byte 13 & 14: End Y-Position in pixels (unsigned short)

[Mutex666]

Wow , fantastic news @Necromantic. Thanks for your work on the subject. Can't wait to test this :-)

[jpk6789]

Can't wait to implement this in our software. Would be nice to use the encoders as another input besides the buttons of the other stream decks. Are there any updates on the implementation? :)

[wischi-chr]

I'm currently planning a redesign of the interfaces in OpenMacroBoard, because I also want to support devices like Streamdeck Plus and Loupedeck CT which looks like this:

[flat-eric147]

I'm currently planning a redesign of the interfaces in OpenMacroBoard, because I also want to support devices like Streamdeck Plus and Loupedeck CT which looks like this:

I would love to see you implementing the Loupedeck. I used their plugin api and it's quite limiting. Having some low level control on the Loupedeck would be a big plus for me. Of course I would also love to see the StreamDeck + implemented here!

[Necromantic]

I second that. A redesign for future "macro boards" seems to be needed.
I've used my hacky implementation of the Stream Deck Plus to control an interactive media installation from a Raspberry Pi.
I might open up a fork with that implementation but because it's so hacky I've refrained from it so far.

I'm currently looking at the Stream Deck Plus vs. the Loupedeck Live/CT for various future projects. Currently I'm leaning towards the Stream Deck Plus because of this repository and the possibilities it gives me. But if this would expand to more generic systems with these new inputs/outputs (Touch Displays, Encoders etc.) this would open up many more options

@wischi-chr Has there been any progress/effort from your part towards this goal or is that just a future plan?
Have you put any thought into the abstractions of the redesign?

[wischi-chr]

Sadly no, there hasn't been much progress since. I still want to implement this but I don't want to make any promises when I will find time to to that.

I'm currently not sure what I want the interface to look like from a developers perspective that uses the library.

I'd like it to be intuitive an yet strictly typed but also reduce the number of different types needed. For example I plan to represent the color LED buttons also as "BitmapKeys" but with single 1x1 pixel "bitmap". This way all the same method and factories to create and clear keys can still be used exactly like the other keys.

I recently looked into how some operating systems designed their interfaces for gamepads, but the situation is a bit different because most gamepads don't have visual feedback/outputs like displays or LEDs.

[wojciechsura]

Do you plan the library to be cross-platform? If not, I would drop SixLabors library for images and just stick with GDI+ Bitmap - it is more than sufficient to handle everything needed for the *decks.

I'd be also glad to see LoupeDeck implementation - the device is really cool, but unfortunately LoupeDeck's software lags behind their fantastic hardware. I could also add support for LoupeDeck in my OpenMacroKeyboard project.

I can help somewhat in the implementation, but I'm not that good in hardware RE, so I'd probably need some guidance on how to do that.

[Necromantic]

I have it running on a Raspberry Pi for a project. So the cross-platform libraries do have a use.

[wischi-chr]

@wojciechsura it actually already is cross-platform and that's the reason why it depends on ImageSharp (because the library internally needs to encode jpegs and resize images, etc.) - I'm using it on windows and linux (Ubuntu) myself.

I have a LoupeDeck already in my drawer but haven't found the time to implement it. The reverse engineering part isn't actually the most difficult part.

I have pretty strong opinions about what the code and the API should look like and would a "comfortable" API would be. I thought about how to "generalize" the API for devices with rotary engoder knobs, sliders, Display-Buttons (like the Stream Deck) but also allow users to handle touch events (the LoupeDeck buttons is actually a touch screen), etc. and all that without breaking (at least not completely changing) the way current users work with their "simple" macro boards.

It's still "on my list" but sadly it's a pretty long list and I don't want to make more promises I can't keep.

[wojciechsura]

@Necromantic, @wischi-chr - fair enough, all is clear now. As a side note, I have spotted some helper methods for GDI in the OpenMacroBoard project, but I couldn't find those in the library nor in the Nuget repository. Are these available anywhere?

And by the way, that's a ton of good work you did on the library. Kudos, man. Your work gives people potential to implement amazing things for the *deck devices and allows disentangling from the software provided by vendors.

Also, I wrote an email to you about my Open Macro Keyboard. I used the email address available on the GitHub. Did you have chance to read it?

[wischi-chr]

The GDI methods were originally removed, but than added back as a separate crate (as discussed here: OpenMacroBoard/OpenMacroBoard.SDK#20) you can find the nuget package here: https://www.nuget.org/packages/OpenMacroBoard.SDK.KeyBitmap.GDI

The email address you found was the correct one - I read it and skimmed the code a bit but haven't had the time to reply because I wanted to look at it in depth (like checking out the code and actually trying it).

It would be really great to have some alternative for some other closed source macro board tools that could actually by usable by consumers instead of just developers like these libraries but I'm pretty sure I won't find the term to maintain that long term. I also have pretty strong opinions about how I would design and implement such a tool and it wouldn't be really fair to just throw some ideas around and letting other people figure out how to actually implement that.

For example I'd try write it in a way so it would be runtime extensible with plugins - maybe with webassembly or something like that and the user can't just set some action to some button, but define regions and assign the region to a plugin/action-instance.
The plugin (when registered) would tell the Plugin-Host which dimensions it supports (think of it like android widgets) and the UI makes sure that the user defines such a reason.

This would bring great flexibility to users and plugin-developers. You could develop games (like minesweeper, xorflip, memory, etc.).

But as I said - those are just ideas and I currently don't really have the time to implement something like that. I'm also a pretty "visual" guy, so it has to look decent. For example I'd like it to look a little bit like real devices. This for example is was the VirtualBoard I wrote looks like (can be used for debugging for example):

[PhillyNJ]

Hi - Has anyone implemented Stream Deck Plus? I have downloaded the project, made some modifications and I am able update the top 8 keys on the Stream Deck Plus. From reading the posts here, its seams that the report for updating the LCD screen and reacting to the encoders takes a new report format and implementing new eventhandlers.

I've written apps and programmed hardware (SAMD21) using the HID protocol for my own projects. However, I need to see the Stream Deck reports or format.

Any help or guidance is appreciated.

[Necromantic]

I have implemented it in a hacky way for a project on a Raspberry Pi a year ago. A proper way would require the discussed restructuring. I basically had to modify multiple layers of the project hierarchy to add support for the encoders and the touch screen input and output. I'd have to find the project files and look at it myself again to see what exactly I did and how. I didn't host it anywhere because I felt it's too hacky to contribute to anything.

The report structures are basically covered in C0nnex and my early posts.

[PhillyNJ]

Thanks - C0nnex's work was helpful! Using wireshark, I am able to sniff the packets and the data matches. I appreciate the effort and work he put into it,