Does not work on reMarkable 2

[Utopiah]

Worked well with reMarkable 1. Tried with reMarkable and despite no error and a window appearing it remains black. I tried with the version I received the device with 2.2.1.88 then latest update 2.3.1.27 but to no avail.

Happy to provide logs or any other information that could help.

Here is the output I have without ffmpeg banner.

fabien@fabien-CORSAIR-ONE-i160:~/Prototypes/reStream$ ./reStream.sh 
Your remarkable does not have lz4.
Falling back to gzip, your experience may not be optimal.
Go to https://github.com/rien/reStream/#sub-second-latency for a better experience.
Input #0, rawvideo, from 'pipe:':  0KB vq=    0KB sq=    0B f=0/0   
  Duration: N/A, start: 0.000000, bitrate: 1054310 kb/s
    Stream #0:0: Video: rawvideo (Y1[0][16] / 0x10003159), gray16le, 1408x1872, 1054310 kb/s, 25 tbr, 25 tbn, 25 tbc

[Eeems]

We don't yet know how to work with the rM2's framebuffer, so all older software that interacts with the screen will not work. You can find more information on the current research here Eeems-Org/oxide#48 and here koreader/koreader#6792

[rien]

This script makes use of the standardized /dev/fbX to read the framebuffer, so it is weird it doesn't work. Can you try reading the framebuffer manually (cat /dev/fb0) and say if there is any output? It could be the case that they output less data or another encoding.

I don't have an rM2, so I won't be able to do troubleshooting myself unfortunately. Good luck with the research.

[Eeems]

This script makes use of the standardized /dev/fbX to read the framebuffer, so it is weird it doesn't work. Can you try reading the framebuffer manually (cat /dev/fb0) and say if there is any output? It could be the case that they output less data or another encoding.

I don't have an rM2, so I won't be able to do troubleshooting myself unfortunately. Good luck with the research.

It's not a standard framebuffer, @ddvk has been having quite the time trying to figure out how it even works.

[rien]

I've got my hands on a framebuffer capture and I was not able to figure out which pixel format is used. I do have the following conclusions of my analysis:

• While one framebuffer dump is 32MB in size, the data is all zeroes after 24893440 bytes.
• The data seems to be grouped by 4 bytes.
• Each of these groups range from 0000 0041 to 0000 0061 in the capture I have. The last byte only has the values 0x41, 0x51 0x53, 0x61, maybe this is dark - light grey - dark grey - white?

[Eeems]

@ddvk is this in line with your findings?

[rien]

I've got a second dump, where there are more values for the last byte. But they are in the same range (0x41 - 0x61).

[ddvk]

@Eeems it is complicated, but the waveforms are there somewhere

[raisjn]

@rien: from a trace that i saw, the FB reports as rgba32 when queried with fbioget_vscreeninfo ioctl during xochitl startup

(FBIOGET_VSCREENINFO)
xres		= 260
yres		= 1408
xres_virtual	= 260
yres_virtual	= 23936
xoffset		= 0
yoffset		= 0
bits_per_pixel	= 32	grayscale	= 0
red	: offset = 16,	length =8,	msb_right = 0
green	: offset = 8,	length =8,	msb_right = 0
blue	: offset = 0,	length =8,	msb_right = 0
transp	: offset = 0,	length =0,	msb_right = 0
nonstd		= 0
activate	= 0
height		= 0x0
width		= 0x0
accel_flags(OBSOLETE) = 0
pixclock	= 28800
left_margin	= 1
right_margin	= 1
upper_margin	= 1
lower_margin	= 143
hsync_len	= 1
vsync_len       = 1
sync		= 0
vmode		= 0
rotate		= 0
colorspace 	= 0
reserved 	= 4387708
-----
(FBIOPUT_VSCREENINFO)
xres		= 260
yres		= 1408
xres_virtual	= 260
yres_virtual	= 23936
xoffset		= 0
yoffset		= 22528
bits_per_pixel	= 32	grayscale	= 0
red	: offset = 16,	length =8,	msb_right = 0
green	: offset = 8,	length =8,	msb_right = 0
blue	: offset = 0,	length =8,	msb_right = 0
transp	: offset = 0,	length =0,	msb_right = 0
nonstd		= 0
activate	= 0
height		= 0x0
width		= 0x0
accel_flags(OBSOLETE) = 0
pixclock	= 28800
left_margin	= 1
right_margin	= 1
upper_margin	= 1
lower_margin	= 143
hsync_len	= 1
vsync_len       = 1
sync		= 0
vmode		= 0
rotate		= 0
colorspace 	= 0
reserved 	= 4387708

rgba32 is easier to deal with than rgb565, as you've noticed it's just groups of 4 bytes :-D i am surprised you have data in your framebuffer based on what other people have reported. if rgba32 works, i'll be pretty happy.

btw, what's the md5sum or sha256 of the buffer?

[raisjn]

regarding the bytes filled with zero:

on the original rM, the framebuffer is twice as large as it needs to be (the height is 2x what is actually shown on screen), so seeing all zeroes after a certain point is not surprising to me. i think the reason for 2x is usually for double buffering purposes (write to second screen, then swap it in all at once)

[thomsten]

I've got access to a reMarkable 2 as well if anyone needs some commands run.

btw, what's the md5sum or sha256 of the buffer?

You mean?

reMarkable: ~/ md5sum /dev/fb0 
480a37f069653d6a99aa603492eb7d66  /dev/fb0
reMarkable: ~/ sha256sum /dev/fb0 
3b5db951652767e4e1abaa964107d5e37cd1ea972297c0b273c1b1dcb8bb1d27  /dev/fb0

[ddvk]

yoi are running version 2.2.1.82?

…

On Fri, 23 Oct 2020 at 11:30 Thomas Stenersen ***@***.***> wrote: I've got access to a reMarkable 2 as well if anyone needs some commands run. btw, what's the md5sum or sha256 of the buffer? You mean? reMarkable: ~/ md5sum /dev/fb0 480a37f069653d6a99aa603492eb7d66 /dev/fb0 reMarkable: ~/ sha256sum /dev/fb0 3b5db951652767e4e1abaa964107d5e37cd1ea972297c0b273c1b1dcb8bb1d27 /dev/fb0 — You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub <#28 (comment)>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/AIYZFLQBYZDX4E6S3CFOOSLSMFEJXANCNFSM4ST2JWJA> .

[thomsten]

2.3.1.27

[ddvk]

What does xochitl's log say, e.g: the line:
Reading waveforms from /usr/share/remarkable/320_R292_AFBC21_ED103TC2M1_TC.wbf

[thomsten]

This part?

Oct 21 06:42:04 reMarkable systemd[1]: Started reMarkable main application.
Oct 21 06:42:04 reMarkable xochitl[240]: Registering exit handlers
Oct 21 06:42:04 reMarkable xochitl[240]: 42:04.816                          we're running on an epaper device (int main(int, char**) /usr/src/debug/xochitl/2.3+gitAUTOINC+3ccb7cc15b-r0/git/src/main.cpp:139)
Oct 21 06:42:06 reMarkable xochitl[240]: Reading waveforms from /usr/share/remarkable/320_R292_AFBC21_ED103TC2M1_TC.wbf
Oct 21 06:42:06 reMarkable xochitl[240]: Running INIT (111 phases)
Oct 21 06:42:06 reMarkable xochitl[240]: 42:06.719                          SWTCON initialized \o/
Oct 21 06:42:06 reMarkable xochitl[240]: 42:06.748                          EPD platform plugin loaded!

[ddvk]

ah, sorry,my mistake, didn't see that it is sha2 and not 1, mine is the same, the framebuffer does not change.

sha256sum /dev/fb0
3b5db951652767e4e1abaa964107d5e37cd1ea972297c0b273c1b1dcb8bb1d27 /dev/fb0

[rien]

The framebuffers I was sent were also the same, so I fear there is no useful information in there.

Has somebody tried strace xochitl to look which syscalls happen?

[ddvk]

yes, fbpan and sometimes fbblank

…

On Fri, 23 Oct 2020 at 12:50 Rien ***@***.***> wrote: The framebuffers I was sent were also the same, so I fear there is no useful information in there. Has somebody tried strace xochitl to look which syscalls happen? — You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub <#28 (comment)>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/AIYZFLWYAES6PMKVUIYEZ6DSMFNXFANCNFSM4ST2JWJA> .

[thomsten]

Attached a strace:
xochitl-strace.log

Suppose the interesting bits are here:

openat(AT_FDCWD, "/dev/fb0", O_RDWR)    = 5
ioctl(5, FBIOGET_FSCREENINFO, 0x7eac565c) = 0
ioctl(5, FBIOGET_VSCREENINFO, 0x3ee1c4) = 0
ioctl(5, FBIOPUT_VSCREENINFO, 0x3ee1c4) = 0
mmap2(NULL, 24893440, PROT_READ|PROT_WRITE, MAP_SHARED, 5, 0) = 0x7141e000
ioctl(5, FBIOPUT_VSCREENINFO, 0x3ee1c4) = 0
ioctl(5, FBIOBLANK, 0)                  = 0

[tadfisher]

Here's an strace dump using -ff, which collects separate traces for each child process. https://gist.github.com/tadfisher/a6ad91552e418077c2ed32d0a9e40a2e

trace.2415 is the main process, which does the FB setup before forking off the process in trace.2417, which appears to call FBIOPAN_DISPLAY for every screen refresh. I do see a couple of calls to FBIOPUT_VSCREENINFO which may point to actual framebuffer data.

I do not see any calls to MXCFB_* ioctls, and the driver in /sys/class/graphics/fb0 does appear to be mxs-lcdif, so they may have repurposed the mxsfb driver for framebuffer updates and are somehow setting EPDC parameters via userspace. I'm not experienced with this whatsoever, though, so I hope at least the traces can help.

[raisjn]

@tadfisher i have been examining the xochitl binary with bokluk and what you've said lines up with what we've seen. a few things:

• the fb0 seems to have waveforms placed in it when xochitl starts running (and thats why everyone has the same values in their framebuffer)

• there is a "vsync and flip thread"

• there is a "generator thread"

• the driver is mxsfb (according to the /sys/ graphics subdirectories) but i have not come across mention of mxsfb or mxcfb that treats fb0 like waveform storage (or RAM) for the display.

• there is no mxcfb_update (like in rM1)

it will be really nice when we can find the memory area that corresponds to what is being displayed on screen (for capturing purposes). it might be in xochitl's process memory (instead of system wide) though.

[PhilippeDuchon]

I don't have the technical skills to help, but I have access to both a rM1 and rM2, so if there are tests I can try and report here, let me know. I'd very much like to be able to use the script on the new device!

[tadfisher]

@raisjnn The pointer to global framebuffer data is located at address 0x003ee1c0, and the buffer itself is 2628288 bytes long. Dumping the buffer results in the raw bitmap data, which is interleaved a bit strangely, but should be trivial to reconstruct. You can open the dump directly in GIMP in "Raw image data" format and play with the sliders to get a picture. I used width=1404.

If you would like a ghidra export, I can provide one.

[raisjn]

awesome, nice work! it looks like the first half of that is something else? the image itself reminds me of when there is a bpp mismatch or resolution mismatch (it ends up printing the image a few times)

btw, are you able to find the same address in remarkable-shutdown binary?

[ddvk]

try converting it like this

convert -depth 8 -size 1872x1404+0 gray:raw out.png

[harrylepotter]

@tadfisher i'm curious how you managed to get the dump at address 0x003ee1c0. I tried using devmem and it threw me a bus error.

[tadfisher]

@harrylepotter This location is in xochitl's address space, so I'm not surprised. To grab the pointer, use dd on /dev/<pid>/mem, skipping 0x003ee1c0 bytes and with a count of 4. Translate the result from litlle-endian, which should get you an address in the heap toward the upper half of the address space. Find the memory map for that address in /dev/<pid>/maps and dump it with dd again.

[rien]

@tadfisher great you were able to dump the image!

If someone could write a script which is able to fetch the memory address and then dump that memory, I can fix it in the reStream script to work with the reMarkable 2. The following things should be done:

1. pid=$(pidof xochitl)
2. Interpret the 4 bytes in /dev/$pid/mem at 0x003ee1c0 (4121024 in decimal) as little-endian integer.
3. Use the address of the previous step to read 24893440 bytes from /dev/$pid/maps.

To check whether de data is correct, you can use @ddvk's convert command. I haven't tested it so I am not sure the pixel format is correct.

If I had a rM2 I would do it myself, but that's unfortunately not the case.

[raisjn]

i don't think this is portable across compiles of xochitl, so you'd want to make sure the memory address is correct before dumping

[rien]

We could make a mapping between the xochitl sha1 and the memory address of the framebuffer

[rien]

I have updated the binary (I've probably used an old build). Can someone test with the latest binary?

I should be dumping the right amount of bytes with the correct offset. But it's hard to test without a rM2 unfortunately.

[1dotd4]

@Foxei

How would you realize this without patching?

In the current version reMarkable 2 have stream is in beta (requires cloud), but I guess there could be a way to intercept it somewhere, just don't have the time right now to see that rabbit hole.

Maybe setting routes to the machine via usb and having a custom server that read the stream? Not sure, just thinking.

[rien]

I know the rM1 could handle color because the framebuffer was a the 'normal' one built into Linux. The rM2 uses special hardware and drives the display from user-space, so I don't think there is any color information present.

From what I remember from the cloud stream, it sends file updates (e.g. the lines you draw) instead of an actual "video stream" of the screen. If the file contains images in color, it will be rendered by your browser.

[BeneCollyridam]

@rien I just tried with the updated binary. By default it is in portrait mode, and when I add the -p flag, it is upside down.

[rien]

Thanks for trying out @BeneCollyridam, does that mean the video isn't broken anymore? How is the latency/fps?

I just tried with the updated binary. By default it is in portrait mode, and when I add the -p flag, it is upside down.

OK, I 've updated the script now. Can you try again?

[BeneCollyridam]

For me it is now in landscape when running with -p and when running without it is in portrait, but upside down.

[rien]

Ok, I've looked at @Foxei's PR and I've updated it again. I've forced pushed so you'll have to git fetch && git reset --hard feature/support-reMarkable-2 probably.

[BeneCollyridam]

It works perfectly now. I haven't measured, but it feels great (with regards to fps and latency).

[bencord0]

I've just pulled the latest commit on this branch (cf6b105 in case you force push again).

Can confirm that it works on the rm2 too.

I've only stumbled onto this project recently, and wow, great work!

[Foxei]

@rien I have to be honest, I am
A little bit proud that my work was not completely dropped. 😉 If reStream is normal working fine you should post it on Reddit. There are a bunch of people waiting for this. Just post a short
Exclamation and a gif. I read tons of complains about the live view feature of remarkable.

If you have any problem in the future just drop me a line. I am down about any research about the rM2.

[Utopiah]

Best way to start 2021!

[1ykos]

I don't know what I did or what happened, It might be that the firmware was updated to 2.5.0.27 (it is at that version now anyways), but reStream was working as intended on my reMarkable2 just a few days ago and today it stopped working.

reMarkable: ~/ ./restream 
Error: No line containing /dev/fb0 in /proc/218/maps file
reMarkable: ~/ ps | grep xochitl
  218 root      265m S    /usr/bin/xochitl --system
  925 root      2496 S    grep xochitl
reMarkable: ~/ grep "/dev/fb0" /proc/218/maps
reMarkable: ~/ 

Any Ideas what I could try? Is there a reason we are not reading from /dev/fb0 directly?

[PhilippeDuchon]

I just checked my own, it is at 2.5.0.27 and reStream works fine - both in landscape and portrait modes - as far as I can tell.

[1ykos]

I also did install rm2fb ... https://github.com/ddvk/remarkable2-framebuffer maybe that has something to do with it?

[Foxei]

Hello @1ykos,
the reason why we are not reading from /dev/fb0 directly is that the rM2 does not provide the image in this way anymore. 😢 We have to locate the frame buffer from the QtApplication itself and the dump the memory from there.

[Foxei]

If you redirected your frame buffer to /dev/fb0 you should use the old rM1 way of streaming. Hard code the rM verison to 1.0 and the image sizes to 1872px times 1404px with the corresponding number of bytes per pixel you are outputting. This should do the trick.

[1ykos]

@Foxei Ah that is what the checksum was about. Indeed /dev/fb0 does not change when the displayed image changes.

[Eeems]

I also did install rm2fb ... https://github.com/ddvk/remarkable2-framebuffer maybe that has something to do with it?

Yes, this is not compatible with rm2fb yet. See #41

[Baccanno]

I don't know what I did or what happened, It might be that the firmware was updated to 2.5.0.27 (it is at that version now anyways), but reStream was working as intended on my reMarkable2 just a few days ago and today it stopped working.

reMarkable: ~/ ./restream 
Error: No line containing /dev/fb0 in /proc/218/maps file
reMarkable: ~/ ps | grep xochitl
  218 root      265m S    /usr/bin/xochitl --system
  925 root      2496 S    grep xochitl
reMarkable: ~/ grep "/dev/fb0" /proc/218/maps
reMarkable: ~/ 

Any Ideas what I could try? Is there a reason we are not reading from /dev/fb0 directly?

Could you try with this maybe ? grep "/dev/shm/swtfb.01" /proc/$pid/maps

[Eeems]

Did you do something like install rm2fb?

[Baccanno]

Yes I installed rm2fb and in my case there where no "/dev/fb0" line in the fb maps, only a "/dev/shm/swtfb.01". I'm on 2.6 on a rm2

[Eeems]

Right, so #41 should have resolved this.