IPTS on Surface Gen7 #4
Comments
|
Example for the new 07 reports: |
|
Patch for not sending From af13ea8a753cbe3f4e366a8a927c5f1892360843 Mon Sep 17 00:00:00 2001
From: Dorian Stoll <dorian.stoll@tmsp.io>
Date: Wed, 3 Jun 2020 15:41:15 +0200
Subject: [PATCH] no set-mode
---
receiver.c | 25 ++-----------------------
1 file changed, 2 insertions(+), 23 deletions(-)
diff --git a/receiver.c b/receiver.c
index ab28399..9884750 100644
--- a/receiver.c
+++ b/receiver.c
@@ -51,7 +51,8 @@ static void ipts_receiver_handle_get_device_info(struct ipts_context *ipts,
static void ipts_receiver_handle_clear_mem_window(struct ipts_context *ipts,
struct ipts_response *msg, int *cmd_status, int *ret)
{
- struct ipts_set_mode_cmd sensor_mode_cmd;
+ int i;
+ struct ipts_set_mem_window_cmd cmd;
if (msg->status != IPTS_ME_STATUS_TIMEOUT &&
msg->status != IPTS_ME_STATUS_SUCCESS) {
@@ -68,25 +69,6 @@ static void ipts_receiver_handle_clear_mem_window(struct ipts_context *ipts,
ipts->status = IPTS_HOST_STATUS_RESOURCE_READY;
- memset(&sensor_mode_cmd, 0, sizeof(struct ipts_set_mode_cmd));
- sensor_mode_cmd.sensor_mode = ipts->mode;
-
- *cmd_status = ipts_control_send(ipts, IPTS_CMD(SET_MODE),
- &sensor_mode_cmd, sizeof(struct ipts_set_mode_cmd));
-}
-
-static void ipts_receiver_handle_set_mode(struct ipts_context *ipts,
- struct ipts_response *msg, int *cmd_status)
-{
- int i;
- struct ipts_set_mem_window_cmd cmd;
-
- if (msg->status != IPTS_ME_STATUS_SUCCESS) {
- dev_err(ipts->dev, "0x%08x failed - status = %d\n",
- msg->code, msg->status);
- return;
- }
-
memset(&cmd, 0, sizeof(struct ipts_set_mem_window_cmd));
for (i = 0; i < 16; i++) {
@@ -206,9 +188,6 @@ static int ipts_receiver_handle_response(struct ipts_context *ipts,
ipts_receiver_handle_clear_mem_window(ipts, msg,
&cmd_status, &ret);
break;
- case IPTS_RSP(SET_MODE):
- ipts_receiver_handle_set_mode(ipts, msg, &cmd_status);
- break;
case IPTS_RSP(SET_MEM_WINDOW):
ipts_receiver_handle_set_mem_window(ipts, msg, &cmd_status);
break;
--
2.26.2 |
|
My earlier experiments with sending HID GET_FEATURE reports using host2me feedback: https://gist.github.com/StollD/89f1c8d094c41e6ab5e3bbbe0dd0335b Raw HID requests using host2me (aka. hid2me) feedback from the old Intel driver: |
|
Unfortunately it does not work without |
|
@StollD I've set up qemu windows 10 and forwarded IPTS device to windows 10 I can see data flowing when I touch screen, |
|
Nice! We want to figure out the initialization sequence that the new IPTS uses, so we need to look for that. It will be encapsulated by an unknown amount of wrapper data, because IPTS doesn't talk to the PCI device that you stubbed directly, it talks to the virtual MEI bus. There is a IPTS specific UUID, so we might be able to look for that. Can you upload an example of the calls that are logged? Never messed with this myself, would be useful to know how that looks like. |
|
Sure. This one is unprocessed, it should contain both initialization and several pen/finger touches |
|
Alright, there are some interesting things. https://gist.github.com/StollD/e2fa122521bef3241782ef827285d998#file-trace-001-L3049-L3053 This is the command that sets the mode of the IPTS hardware. The parameters are all zeros, which, on the old IPTS, meant singletouch mode. If the first zero would be a one, it would be multitouch mode. If windows initializes the hardware in singletouch mode and gets multitouch events, and we initialize it in multitouch mode and get singletouch, it might be that Intel swapped the meaning of these two values. However, the device otherwise seems to behave exactly like the old singletouch mode: It doesn't signal a new report by increasing the number in the doorbell buffer but uses the For example:
One issue with that theory though is that we already tested it: When the SL3 was just released, @archseer and I tried exactly that (swapping the meaning of single- and multitouch). The hardware returned an invalid-params error. Though it is possible that that was changed again in the SB3, or in a firmware revision. I also don't know what @archseer exactly changed back then, so it probably won't hurt to try it again. https://gist.github.com/StollD/e2fa122521bef3241782ef827285d998#file-trace-001-L2874-L2878 IPTS sends a 0x9 command as part of the initialization sequence. According to the old driver from intel, 0x9 means
https://gist.github.com/StollD/e2fa122521bef3241782ef827285d998#file-trace-001-L3001-L3007 There is a new 0xf command that is not documented in the old intel driver. Could be related, although I doubt it. It seems to carry a memory address as a parameter. Feedback sending is...weird. It only seems to be sent for buffer 0xA and 0x2, and there are two instances of "special" feedback (with buffer ID 0x10).
That special feedback could be used to send HID feature reports to the hardware, but there is no way for us to know, because these are stored in a seperate buffer and don't go over the PCI bus. But we also don't know if that special feedback is new, because we have no trace from a gen4-6 device to compare against. |
|
@StollD I can get the same trace from gen6 (I believe) device (surface book 2) if it helps Also if it will help in research I could record session with "pen only", "multitouch" and "touch" |
That would be great actually! I was planning to research how to setup the passthrough myself, but if you already know how it works, it will probably be faster.
Not really. What I call multitouch covers all three of these. It is simply how we (and intel) call the mode that IPTS runs in. And windows doesn't allow changing it AFAIK. Regarding my multitouch vs. singletouch theory: It should actually be enough if you loaded the master branch of this repo with the $ sudo modprobe -r ipts
$ sudo modprobe ipts singletouch debugThen simply try to use the pen (or touch the screen), and watch dmesg. You actually need to use the master version though, the UAPI version for iptsd won't work, because it doesn't support changing the mode. |
|
@StollD unfortunately no reaction on pen / multitouch with singletouch option (i see logs on single touches so it was loaded correctly) |
|
Out of curiosity could you post the dmesg? Maybe I can find some clue in there. If simply changing the parameter for |
|
Sure! I've also tried to record another very short session trace on qemu, touching device only with pen, that I will be able to get logs from Surface Book 2 somewhere this week |
|
Hmm. I'm actually very confused of size for |
Well, its part of the device initialization. That doesn't care about what tool you want to use on the device. ;) The actual device data is passed using memory buffers, and not through the PCI interface, so you won't see a difference if you change the tool (stylus / finger).
The IPTS interface is just one function of the PCI device you are tracing. It could provide other interfaces that are used by windows that we don't really know about. But I am confused about a thing too: Why is the linux driver logging singletouch events, if you added the singletouch option? Changing the mode should either produce an error (like when @archseer tried), or change the behaviour. IPTS shouldn't just ignore it. |
|
Double checked again, I see logs, I see reaction to single touches, no pen, no multitouch |
|
Ok. Maybe the parameter has actually lost all its meaning. Btw, I guess I know why the feedback that was sent looks weird to me. On linux we often had issues, especially on older devices, where sending feedback too often would cause the device to crash. Maybe on windows they only send feedback for every eight input (every time you read from buffer 2 or 10) to mitigate these issues and prevent bad firmware from crashing the device. That would actually be a pretty smart thing to do, so we might want to replicate that behaviour under linux, for all devices. |
|
I wrote a patch that sends the same diff --git a/control.c b/control.c
index 9179eca..1bc00a0 100644
--- a/control.c
+++ b/control.c
@@ -59,6 +59,7 @@ int ipts_control_send_feedback(struct ipts_context *ipts,
int ipts_control_start(struct ipts_context *ipts)
{
+ u8 data[16];
ipts->status = IPTS_HOST_STATUS_INIT;
if (ipts_params.singletouch)
@@ -66,7 +67,12 @@ int ipts_control_start(struct ipts_context *ipts)
else
ipts->mode = IPTS_SENSOR_MODE_MULTITOUCH;
- return ipts_control_send(ipts, IPTS_CMD(NOTIFY_DEV_READY), NULL, 0);
+ memset(data, 0, 16);
+ data[0] = 0x12;
+ data[2] = 0x64;
+ data[4] = 0x1E;
+
+ return ipts_control_send(ipts, IPTS_CMD(SET_POLICIES), data, 16);
}
void ipts_control_stop(struct ipts_context *ipts)
diff --git a/protocol/events.h b/protocol/events.h
index f8b771f..b50d399 100644
--- a/protocol/events.h
+++ b/protocol/events.h
@@ -24,6 +24,7 @@ enum ipts_evt_code {
IPTS_EVT_FEEDBACK,
IPTS_EVT_CLEAR_MEM_WINDOW,
IPTS_EVT_NOTIFY_DEV_READY,
+ IPTS_EVT_SET_POLICIES,
};
#endif /* _IPTS_PROTOCOL_EVENTS_H_ */
diff --git a/receiver.c b/receiver.c
index ab28399..c0f60c2 100644
--- a/receiver.c
+++ b/receiver.c
@@ -10,6 +10,20 @@
#include "protocol/responses.h"
#include "resources.h"
+static void ipts_receiver_handle_set_policies(struct ipts_context *ipts,
+ struct ipts_response *msg, int *cmd_status)
+{
+ if (msg->status != IPTS_ME_STATUS_SENSOR_FAIL_NONFATAL &&
+ msg->status != IPTS_ME_STATUS_SUCCESS) {
+ dev_err(ipts->dev, "0x%08x failed - status = %d\n",
+ msg->code, msg->status);
+ return;
+ }
+
+ *cmd_status = ipts_control_send(ipts,
+ IPTS_CMD(NOTIFY_DEV_READY), NULL, 0);
+}
+
static void ipts_receiver_handle_notify_dev_ready(struct ipts_context *ipts,
struct ipts_response *msg, int *cmd_status)
{
@@ -196,6 +210,9 @@ static int ipts_receiver_handle_response(struct ipts_context *ipts,
int ret = 0;
switch (msg->code) {
+ case IPTS_RSP(SET_POLICIES):
+ ipts_receiver_handle_set_policies(ipts, msg, &cmd_status);
+ break;
case IPTS_RSP(NOTIFY_DEV_READY):
ipts_receiver_handle_notify_dev_ready(ipts, msg, &cmd_status);
break; |
|
No effect unfortunately. Still modprobes, still reports single touches only 😢 (with or without singletouch option) |
|
Damm. I had really hoped that would work. The other new command (the 0xf one) seems to contain a memory address, so sending that ourselves might not be that simple. It's either that, or changing the modes really happens through special feedback, which will be pretty hard for us to replicate. |
|
Sad. I'm trying to get myself familiar with dtrace for windows introduced by Microsoft not so long ago - it works extremely low level so theoretically it should be possible to trace special feedback function calls in windows
|
|
Additionally, I've put |
|
Sorry for "blogging". It seems that initial assumption about additional DLL for processing is correct. Surface is installing Unfortunately this DLL also includes just HidP_GetCaps and HidP_GetValueCaps calls |
|
Some new dumps from surface book 2 The first one is pretty funny. I've just copied qemu image with my surface book 3 driver to surface book 2. It initialized, but reported only single touches. Also I've noticed that same updates arrive when we're in (for example) boot sequence, so it seems like it's just a default mode for touch screen Second one is with sb2 driver |
|
Thank you! Your trace-clean log confuses me a bit. Were you able to use the pen with that? Because the commands in the log are the ones that are used for singletouch. |
|
I have played with it a bit myself, and I don't think it will be possible to enable multitouch mode on gen4-6 within the VM, at least not with a custom build of QEMU. Multitouch mode on the old IPTS depends on binary firmwares that are run on the GPU. The firmware to use is selected by querying the name of the But even then it will probably not work, because we don't pass through the GPU, so the driver won't be able to process touch data there. Both, the TSML ACPI device, and the touch processing on the GPU are not used on gen7 though. So it should be less of an issue there. Just to be sure, @xanf when you tried it on gen7, was the hardware actually in multitouch mode? If it is in multitouch mode, either the pen works, or nothing works (because no touch processing driver is installed). If it detects single touches, and nothing else, it is in singletouch mode, which would match the results from our earlier experiments on linux. |
|
Also, I would be curious about how your VM was set up. I just made a generic windows install, and tried to install the surface driver package. But that refused to run, because it didn't detect an actual SB2. I resorted to unpacking the installer and installing the drivers by hand, but there are so many, and we have no idea what is really needed. Also, at least on SB2, the installer seems to do some more setup in the registry, so we loose that too. I am wondering if we might get better results if we create a disk image of an actual windows install, where all these drivers are installed properly, and use that for the VM. I've seen that microsoft has recovery images for the surfaces, maybe that can also be of use. |
|
On surface book 3 system was reacting with reads / writes in log to pen,
I've checked this
I've also resorted to manual installation of unpacked driver, for surface
book 3 it seems two devices are sufficient - intel precise touch driver
(and installing this alone enables touchscreen to "receive" pen events and
something ( i don't remember exact name, does not have sb3 right now with
me) called surface touch processor - it was exactly the device which had
non-typical log, sin, cos functions in disassembly.
I've figured required driver by disabling them one by one in windows and
checking when touch stops working
|
|
I think on the SB3, access to the GPU shouldn't be required as it looks like touch processing happens in the Regarding ACPI, its possible to add/load additional ACPI tables in qemu (should be |
Yes. I am seeing a new report type (code 08) associated with the tap events, as you can see in this sample. There are still plenty of heat maps but with these shorter reports mixed in. Has there been any effort to extract a report descriptor? It would really interesting to see the device's take on the data it's spewing out! Thanks for taking a look at my sandbox! I was thinking that one could do very well to apply something like Guo's algorithm to get a linear fit, and from there use an optimized vector library to efficiently manage all the regression operations with vectorized instructions. As I think you've discovered, the trickiest part seems to be just identifying the peaks, and choosing what data to include in your fit. However, I think I have something that works pretty well now for two fingers. I'll ping everyone here once I've posted a proper demo. |
I think a right click is usually done by a long one finger press. The best DE for using touch is probably GNOME, because their UX is somewhat gesture / touch oriented, and because they have a pretty solid Wayland implementation. If you dont like GNOME, I am running KDE pretty successfully at the moment, but its Wayland support is still a work in progress so you might have issues.
Yeah, there are commands in the IPTS interface that return the report descriptor. @archseer also dumped his descriptor from Windows (on an SL3). https://gist.github.com/archseer/80e33f90b02db83da0d88d95af1eb3a9 As far as I know, the values that come out of the processing are not different from what we are already receiving / sending on older devices. |
|
Any updates on this? |
|
Unfortunately not. We'll post an update here if we have something. |
|
I will have a try on Martin's work. :) |
|
I've spent some time figuring out how the reports for the pen data are formatted. Together with the report names that @qzed already figured out, maybe someone has any ideas about the data in the reports https://gist.github.com/StollD/ae901f46bf693a0c5355cc048cb21073 |
|
Can confirm two-finger gestures are working on SB3 now. Great work! |
|
I've also dumped some data: https://gist.github.com/yatli/f248517c414cef8d460e08bab260c793 Data size goes [2000 -> 1500 -> 2000 -> 1500 -> 7488] then repeats this pattern. "12 07 ... 0b 07 ...": "ba 07 ... b3 07" ... where |
Peek.2022-02-21.03-59.mp4 |
|
This is great news. Thank you to all of you putting in such hard and great work |
|
Wait, what was figured out? Is it related to the Surface Gen7 models at all? |
|
TL;DR |
|
@yatli Yeah, the messages can have a variable length, depending on the reports that they contain. The size of the outer HID report doesn't really matter. Since HID does not allow variable length reports, Microsoft had to define a bunch of reports for different sizes, so they could always send the one that fits best. Kinda like owning a 8GB, 16GB and 32GB USB drive and always using the smallest one possible to store your data. (Also I think your dumping program has buffer and type swapped ;)) |
|
This is the ImHex version of @StollD 's hand-annotation. Tested a few reports of different sizes and it generalizes.
|
|
@StollD how did you figure out the names and the boundaries of the report slices? :O |
|
@qzed figured out the names and report IDs, in the windows touch processing dll. (see here: #4 (comment)) As for the report slices, the same format is used in the data that we get on older gens, and we already figured that out. I first looked for a report ID, and then checked if I would get another report ID if I interpreted the next two bytes as a length value and skipped that many bytes. If it had been a different format, it would have been for difficult. After that I just had to figure out the length of the header, by following the same scheme. |
|
Okay so we now need to figure out the content format and what it means. Time to move to #14 :p |
|
Is anyone else having trouble with the thumb not reading as a touch with the libqzed version of iptsd and mult-touch |
|
Possibly you've already seen this, but I ran across some interesting comments regarding Surface HID: https://lore.kernel.org/all/87im072pfw.fsf@kernel.org/
So it seems like the recommended way to send heatmap data to userspace is v4l (VFL_TYPE_TOUCH, /dev/v4l-touchX)? Not sure how I feel about that. Also, I can't actually find any userspace components which use that interface. (Haven't looked very hard though.) Maybe it's just closed source Android stuff? I don't think v4l would be a good fit for the pen data though. |
|
I mean, I guess it makes sense. The heatmap we receive is not different from a 80x72 (idk if thats the right dimension) video. But for touch processing, you dont really want extensive parallelization (otherwise the inputs might jump around), and in our case, the data is already a HID report. The kernel maintainer probably thought that MS used HID to get raw data from the device to userspace, and not that the device already produces HID data. |
|
@quo Oh nice! Haven't seen that before. It reads like the Duo also needs user-space processing, which is somewhat unexpected to me.
That'd complicate things. As mentioned it'd be possible to implement it in some HID device driver and let that do the basic heatmap frame parsing. Question is what to do with the whole other bunch of metadata being sent. It's not just the raw touch frame after all... also as you rightly mention, it doesn't particularly suit pen data well either. Also as @StollD writes: Some assumptions without actually knowing the protocol, I guess. If you purely look at "it's a heatmap", which is essentially some weird video format, v4l kinda makes sense. Just that in our case everything is already HID and it's not just a pure heatmap. For the time being, I'd try to go the hidraw route, simply because it seems easiest. Ultimately, it'd probably be best if we then follow whatever MS decides to do. I suspect if they do need pen support, they'd probably at least handle that via hidraw (or maybe eBPF?). It's quite nice that they actually try to upstream their SPI-HID driver, that's a big plus for the Pro X and AMD devices (although AMD devices still need ACPI support in that driver...).
I might try looking for that later... but yeah, I'm not sure it's open-source :/ Might need to dig for custom roms, they'd need to include that somehow... |
Now I kinda want to try and find the system image of the duo and find the service that they use for processing the data. It will be completely useless as is, because it will be an arm binary, but maybe it responds better to decompilation than the windows version xD |
|
Yeah, I agree, hidraw is easier for now. I was just looking at the HID feature reports. It looks like report 6 (digitizer usage 0x63) contains the metadata that iptsd currently needs the config files for. Format is as follows (values from my SP7+): struct touch_metadata_feature_report {
u8 report_id; // 6
struct {
u32 size; // 119
u8 zero, type, unknown; // 00 00 00
struct {
u32 size; // 112
u8 zero, type, unknown; // 00 02 01
struct touch_metadata container_data;
} container;
} root_container;
}
struct touch_metadata {
u32 rows; // 44
u32 columns; // 64
u32 screen_width; // 25978 (mm*100)
u32 screen_height; // 17319 (mm*100)
u8 unknown; // 1
// transform matrix to convert row/col to physical screen coords
struct {
f32 xx; // -412.3492126464844 = -width/(columns-1)
f32 yx; // 0.0
f32 tx; // 25978.0 = width
f32 xy; // 0.0
f32 yy; // -402.7674560546875 = -height/(rows-1)
f32 ty; // 17319.0 = height
} transform;
// unknown floats, possibly tilt transform?
f32 a[4]; // 178.0, 182.0, 180.0, 1.0
f32 b[4]; // 178.0, 182.0, 180.0, 1.0
f32 c[4]; // 90.0, 171.0, 100.0, 20.0
f32 d[4]; // 172.0, 177.0, 175.0, 2.0
} |
|
so now multitouch is enabled by default or need extra steps ？ |
|
Not really. You still need to build the kernel module (IPTS or ITHC, depending on your device) and the iptsd daemon from source. But we are getting there. |
|
I'm wondering whether we can put some AVX512 in the kernel. There is precedence for that. If we do the processing in the kernel and Microsoft does it in userspace, then we might be able to beat them in speed! This would just be like the HTTP server wars with Linux's TUX and Microsoft Windows's IIS http.sys |
|
Doable? Yes (you'll need to set up some additional registers and restore them before/after AVX and other vector or floating-point instructions). Accepted by upstream: Very likely not. Too complex for kernel code, too little benefits, and too hard to maintain. See the whole graphics stack which essentially lives in user-space beyond "simple" buffer allocation and command submission. |
|
Even if it is doable, the userspace approach allows us to support all MS devices with a single implementation. If we put the processing into the kernel we need to add or link it to three seperate drivers, and one is not even developed by us (spi-hid). |
and others
On the surface gen7 devices, IPTS has undergone several changes:
The binary firmware that was used for previous models does not exist anymore. Due to the changes in the GuC firmware it seems like Intel / Microsoft are now forwarding the data to userspace(?) using HID reports, and parsing it using a daemon (or another driver?). The HID descriptor is also not loaded from a binary anymore and needs to be dumped in Windows to analyze it.
The
SET_MODEcommand will not accept any parameter except forIPTS_SENSOR_MODE_MULTITOUCH. Other parameters will result in the statusIPTS_ME_STATUS_INVALID_PARAMSbeing returned.Even though we can only initialize the device in "multitouch" mode, once the initialization was finished, the only that that is being sent are HID reports: the ones for singletouch, as well as a new report (with report ID 07). Other than that the device behaves exactly expected for multitouch mode (doorbell is incremented instead of signaling new data via. MEI responses).
This issue will be for tracking the progress on full IPTS support for gen7 surface devices.
We suspect that the new IPTS needs HID feature reports to enable proper multitouch mode. The old driver from intel has some hints towards sending HID reports using a special type of feedback, but some quick tests didn't appear to be working. It also has no proper documentation so it is mostly just guessing how the different parts behave.
Another thing that has no been tested is simply not sending the
SET_MODEevent and directly sendingSET_MEM_WINDOW. Maybe there was a change so that multitouch mode is the default, and sending anything will change it to singletouch mode? (i.e redefiningSET_MODEas sth. likeSET_SINGLETOUCH_MODE)The text was updated successfully, but these errors were encountered: