Support for RM2

[ishanina]

Hi, thanks a lot for rmview, I think it's really great! I recently got a remarkable 2, and tried to use both branches. The vnc branch didn't connect at all, but the ssh branch showed me a black screen that just flickered every time something on the screen changed. Do you know what could be causing the problem?

[fabien-roquet]

Same issue for me. It would be so nice if rmview could work for rm2!

[bordaigorl]

Hi, thanks for reporting this, it would be great if we could figure out RM2 support together.
I don't own a RM2 so I'll need somebody to do the tests for me.
Pull requests also very much appreciated.

The vnc branch is not expected to work on RM2 due to different kernel.
The first step to make the ssh branch work is to obtain a version of LZ4 that works on the RM2.
You can verify it works by connecting to the RM via ssh and calling $HOME/lz4 --version.
If that works we can start looking at whether the framebuffer stores the screen data in a different format (currently assumed: RGB16). If LZ4 does not work you need to either compile it for the RM2 platform or you need to figure out the platform and find a binary compiled for it.

[ishanina]

Thanks for the response! I tried running lz4 --version on the remarkable 2, and it gave the output:
*** LZ4 command line interface 32-bits v1.9.2, by Yann Collet ***

Also for the vnc branch, it doesn't look like it's incompatible with rm2 to me, but maybe I'm missing something. When I run rM-vnc-server, it gives the output

28/10/2020 14:44:34 Listening for VNC connections on TCP port 5900
28/10/2020 14:44:34 Listening for VNC connections on TCP6 port 5900

[bordaigorl]

The VNC branch is incompatible because of mxc_epdc_fb_damage which is a kernel module that works only with the RM1 kernel, see here.
the vnc server may start up but would not receive any update from the screen, as those are generated by the kernel module.

It seems that reStream has the same issue which makes sense since the ssh branch of rmview implements the same pipeline. It seems they made some progress there. Depending on your coding skills you could try to integrate their proposals, e.g. this and preceding posts. To do that you would need to modify workers.py to issue the right commands to the remarkable, and changing img_format to QImage.Format_Grayscale8.

[harrylepotter]

this is very bespoke (you'll need to edit workers.py line 39 using the information from the reStream issue - such as xochitl's pid and addresss space ) patch to get things up and running on an RM2 if you're overwhelmingly eager to try things out. Obviously this is not sustainable code, but being a new user i was super excited to check out @bordaigorl 's wares...

diff --git a/src/rmparams.py b/src/rmparams.py
index 2298cc2..b1fe9aa 100644
--- a/src/rmparams.py
+++ b/src/rmparams.py
@@ -1,7 +1,7 @@
-WIDTH = 1408
+WIDTH = 1404
 HEIGHT = 1872
 PIXELS_NUM = WIDTH * HEIGHT
-TOTAL_BYTES = PIXELS_NUM * 2
+TOTAL_BYTES = PIXELS_NUM
 
 # evtype_sync = 0
 e_type_key = 1
diff --git a/src/workers.py b/src/workers.py
index 6d68c37..509e542 100644
--- a/src/workers.py
+++ b/src/workers.py
@@ -17,6 +17,7 @@ from lz4framed import Decompressor, Lz4FramedNoDataError
 
 try:
   GRAY16 = QImage.Format_Grayscale16
+  GRAY8 = QImage.Format_Grayscale8
 except Exception:
   GRAY16 = QImage.Format_RGB16
 RGB16 = QImage.Format_RGB16
@@ -33,10 +34,10 @@ class FrameBufferWorker(QRunnable):
 
   _stop = False
 
-  def __init__(self, ssh, delay=None, lz4_path=None, img_format=GRAY16):
+  def __init__(self, ssh, delay=None, lz4_path=None, img_format=GRAY8):
     super(FrameBufferWorker, self).__init__()
     self._read_loop = """\
-      while dd if=/dev/fb0 count=1 bs={bytes} 2>/dev/null; do {delay}; done | {lz4_path}\
+      while dd if=/proc/237/mem bs=3647 count=721 skip=527864 2>/dev/null > /mnt/ram/temp && sleep 0.1 && dd if=/mnt/ram/temp bs=2628288 count=1 2>/dev/null; do {delay}; done | {lz4_path}\
     """.format(bytes=TOTAL_BYTES,
                delay="sleep "+str(delay) if delay else "true",
                lz4_path=lz4_path or "$HOME/lz4")
@@ -64,7 +65,10 @@ class FrameBufferWorker(QRunnable):
         while len(data) >= TOTAL_BYTES:
           pix = data[:TOTAL_BYTES]
           data = data[TOTAL_BYTES:]
-          self.signals.onNewFrame.emit(QImage(pix, WIDTH, HEIGHT, WIDTH * 2, self.img_format))
+          #self.signals.onNewFrame.emit(QImage(pix, WIDTH, HEIGHT, WIDTH * 2, self.img_format))
+          ima = QImage(pix, HEIGHT, WIDTH , self.img_format)
+          ima = ima.transformed(QTransform().rotate(-90))
+          self.signals.onNewFrame.emit(ima)
           if SHOW_FPS:
             f += 1
             if f % 10 == 0:
@@ -117,7 +121,7 @@ class PointerWorker(QRunnable):
 
   @pyqtSlot()
   def run(self):
-    penkill, penstream, _ = self.ssh.exec_command('cat /dev/input/event0 & { read ; kill %1; }')
+    penkill, penstream, _ = self.ssh.exec_command('cat /dev/input/event1 & { read ; kill %1; }')
     self._penkill = penkill
     new_x = new_y = False
     state = LIFTED

[FurryAcetylCoA]

With @harrylepotter 's information and a lots of trial and error I get a better version of line 39 as following

while dd if=/proc/219/mem bs=4096 skip=469957 count=642 2>/dev/null |   \
      tail -c+8 | head -c 2628288; do {delay}; done | \
      {lz4_path}

(other changes from #22 (comment) should also be applyed )
And src/rmparams.py as following

@@ -1,9 +1,9 @@
-WIDTH = 1408
+WIDTH = 1404
 HEIGHT = 1872
 PIXELS_NUM = WIDTH * HEIGHT
-TOTAL_BYTES = PIXELS_NUM * 2
+TOTAL_BYTES = PIXELS_NUM 

You can get those magic number by running the following script on rM2

#!/bin/sh
width=1872
height=1404
bytes_per_pixel=1

window_bytes="$((width * height * bytes_per_pixel))"
pid="$(pidof xochitl)"

skip_bytes_hex="$(grep -C1 '/dev/fb0' /proc/$pid/maps | tail -n1 | sed 's/-.*$//')"
skip_bytes="$((0x$skip_bytes_hex + 8))"
page_size=4096
window_start_blocks="$((skip_bytes / page_size))"
window_offset="$((skip_bytes % page_size))"
window_length_blocks="$((window_bytes / page_size + 1))"
echo "while dd if=/proc/$pid/mem bs=$page_size skip=$window_start_blocks count=$window_length_blocks 2>/dev/null | tail -c+$window_offset | head -c $window_bytes;do {delay} ; done |  {lz4_path}"

However as rien/reStream#31 (comment) pointed. This code can't run natively on rM2.
You should install entware on your rM2 (refer https://github.com/Evidlo/remarkable_entware) .
Then

/opt/bin/opkg update
/opt/bin/opkg upgrade
/opt/bin/opkg install coreutils-head

And remove the stock head in /usr/bin then create a new link to the new head in /opt/bin (or just copy)

Also a very bespoke patch. And involves changes in rM2's system.
At least it works fine

[pl-semiotics]

For anyone interested, I am the author of the device side of the vnc branch. I unfortunately do not have a reMarkable2, and the fact that much of the framebuffer processing now happens in userspace makes the former solution for snooping on damage tracking information perhaps somewhat unfeasible. I do expect to take a look at trying to port the software with the assistance of others who do have rM2 devices over the next few weeks and would be happy to update here with the result should I find a solution.

[bordaigorl]

@pl-semiotics that's so nice of you, thank you so much.
The reMarkable team finally released the sources of the kernel of rm2 so maybe now it's a bit more feasible to prepare a new version?

[johncokos]

I've got a new rM2 and am happy to test things out.

[ChrisPattison]

@pl-semiotics I have been looking into this a little bit (no rm2 at the moment) and maybe the links in canselcik/libremarkable#46
are helpful (especially @ddvk and @raisjn's reversing efforts). If it's unknown how to get the damage data from SWTCON, maybe the processor is fast enough to recover it by comparing frame buffers? Since my understanding is that part is known

[cyplo]

can also help testing/debugging if needed, just got my rm2

[Foxei]

Merry Christmas everyone,
I can announce that the reStream like version via ssh is working great! I have uploaded a working version (no polished but working) for the rM2 on my rmView fork (https://github.com/Foxei/rmview). It does not require you to expose the frame buffer or to use entware at all. Just copy two file to the reMarkable and you are done.

Screencast.2020-12-26.14.20.08.mp4

Thinks that remain open (feel free to add thinks to the list):

• Make the code reliable detect if a rM2 or rM1 is connected (as reStream already does)
• Test mouse features.
• Bring the ssh branche on the state of the vnc branche.

What do you think is the best way going forward from here?

[bordaigorl]

@Foxei thank you so much for this!
It really is a Christmas miracle 😉

Let me have a look and I'll get back to you.
I would integrate your edits into the current vnc version, allowing the same rmview version to seamlessly handle rm1 or rm2 automatically detecting the appropriate method to use.
I'll update this thread soon.

[pl-semiotics]

@bordaigorl Sorry for the long delay, but I've finally had a bit of time and access to someone with an RM2 for testing, so I also have a Boxing Day present in the form of a VNC server prototype that works with the RM2, finished last night! The interface is exactly the same as for RM1, so only installation would need to change. I'm just finishing cleaning it up and getting it into a distributable form---I should have it ready in the next day or two!

(It also supports turning VNC cursor events into fake input events on the device so that you can draw/turn pages/etc from VNC.)

[bordaigorl]

@pl-semiotics Oh my god that's amazing news!
Thank you so much for your work on this, it's truly a game changer for many ❤️ .
Two Christmas miracles at once!

[Foxei]

Hello it is me again. 😄
Do any of you have a rM1 and can tell me what kind of FPS the vnc version can achieve? I am working on a replacement for the dd, head and tail combination and I have a running version that achieves about 12-15fps. Is this promising and should I continue?

Untitled.mov

[bordaigorl]

@Foxei the vnc version is orders of magnitudes better than the reStream-like one, for the simple reason that it transmits only the minimal updates necessary to update the picture instead of the whole frame every time.
It does so without having to compute diffs because it just reuses the update events of the device itself, which just repaints small rectangles at every update. This way you can get the updates on rmview may even be quicker to display than on the tablet itself (because of the eInk display latency)!
Once @pl-semiotics is done, a VNC solution would be both easier to incorporate (as the interface is the same) and more efficient.
So I think maybe you should wait until the VNC version is ready to be tested for RM2, so you can compare and see if there is a reason to maybe support both...

[pl-semiotics]

@bordaigorl Sorry this ended up taking so long! After a surprising amount of time spent getting from "something that vaguely works" into "something that can hopefully be distributed", I've finally pushed rm2 support! The new -standalone executables ought to make installation a bit easier---they include all of the necessary kernel modules/etc. themselves. I also send the position of the stylus via cursor position update messages now, which might be useful for you? Feel free to ping me if there's anything in input/output handling that could be made more useful.

@Foxei I apologize, I didn't see your comment earlier. As @bordaigorl points out, the vnc server works by only sending the regions which have actually been updated, and so it does not have a static frames-per-second rate as frames without updates are never sent (and how many times a second an update is sent is of course dependent on the size of the redrawn region).

[tbabej]

@pl-semiotics Is this possibly version dependant? While this is a great progress, I'm seeing some deficiencies in the damage tracking, see the recording below. I'm on 2.5.0.27.

Lower framerate is caused by the gif-recording, the VNC server itself is quite smooth.

[Foxei]

@tbabej I just shoot in the dark as I have never used the vnc version but have you set the metadata for the rM2 screen? 1404px x1872px with Greyscale 8 bit (1 byte per Pixel). If not it could be a rendering issue caused by QT5.

[bordaigorl]

@tbabej looks like there could be some scaling issue on the client side.
(the black bar on right and bottom corners looks suspicious)
What are you using as a client? I don't recognise the cursor as rmView's...

[pl-semiotics]

@tbabej It should not not be terribly version dependent, as avoiding such was part of the motivation for the approach taken to find the information needed. It seems it is correctly finding and patching sendUpdate (and exporting the actual fb), so I am not sure that is likely to be the issue. Actually, I should have documented this---but I think there are some issues when using clients that do not support the vnc cursor position pseudo encoding, as in this case libvncserver sometimes insists on trying to draw the cursor into the framebuffer on the tablet, and this can degrade the quality as some 2048 pixels around the cursor may be affected by the hack I am using to prevent this from interfering with the framebuffer displayed on the device. I think this looks like it may be the issue? As it seems that the pixels you are missing are frequently directly around the cursor. Can you try again with a client that supports cursor position updates? (You may need to turn off an option entitled "Show remote cursor" or similar). The server should print "Enabling cursor position updates for client" when you connect.

If that doesn't work, please open an issue on the vnc server repository with logs and I'll be happy to take a look!

[tbabej]

@bordaigorl That is indeed rmview. The cursor tracking did not work out of the box with the rM2-vnc-server-standalone, unfortunately.

@pl-semiotics I will retest with a regular VNC client.

[tbabej]

Indeed it looks like @pl-semiotics is right, using a different client (in the video below it's remmina) resolves the issue.

remmina.mp4

Which means this is an issue in the VNC client implementation of the rmview. @bordaigorl if you want, I'd be happy to expose the VNC server of a connected device so that you can connect to it (and play the role of your arm when needed).

[pl-semiotics]

@bordaigorl By the way, if it's handy for testing---I've been meaning to post this somewhere but not gotten around to it---I've found that running xochitl in a VM with a virtual framebuffer works pretty well. I use this script in Fedora 32 armhp qemu VM (run in a directory where empty subdirs rmcnt, rmmnt, rmcnt-root-home exist):

#!/bin/sh
modprobe vfb vfb_enable=1 videomemorysize=40000000
losetup -f </path/to/rm2/rootfs>
partprobe /dev/loop0
mount -o ro /dev/loop0p3 rmmnt
mount -t tmpfs tmpfs rmcnt
for x in rmmnt/*; do out=rmcnt/$(basename $x); if [ -f $x ]; then touch $out; else mkdir $out; fi; mount --rbind --make-rprivate $x $out; done
for x in sys proc dev run; do mount --rbind --make-rprivate /$x rmcnt/$x; done
mount --bind rmcnt-root-home rmcnt/home/root
mount --bind /dev/fb1 rmcnt/dev/fb0
mount -t tmpfs tmpfs rmcnt/sys/power
cp /sys/power/* rmcnt/sys/power/

For stylus input to work, a uinput digitizer device needs to be added before xochitl starts up; I use rM-mk-uinput from here.

[bordaigorl]

First of all thanks so much @pl-semiotics for your work, and thanks @tbabej for testing and reporting this!
I can confirm I get the same kind of output on RM1 too with the new standalone vnc server.
From a quick look it seems it's the cursor handling indeed: my superficial diagnosis is that if the server detects the client does not support cursor events, it falls back on the rectangle updates by sending a little image of a cross moving as the cursor does (but somehow the update that should restore the underlying bit covered by the cross is not handled properly, effectively erasing the pic below).

@tbabej I could get notmal behaviour simply by adding PSEUDO_CURSOR_ENCODING, on line 44 of workers.py.
That signals to the server that cursor events are supported by the client (even if they are ignored).
Could you try that?

[pl-semiotics]

@bordaigorl Oh, I see why there was a behavior change---now that I send the wacom stylus position as cursor position updates to allow drawing a cursor, all of the problems with non-cursor-position-capable clients manifest more. (Previously, if you were never sending cursor movement information, you wouldn't get any cursor drawing). Sorry about that---I don't have any clients that are not cursor-position-capable but also do not send cursor data of their own. Yes, unfortunately if the client doesn't support the cursor position pseudoencodings there are some problems with getting back zeroes, since we have to replace the MAP_SHARED pages (needed to be able to see updates from xochitl) with something else writable in order to let libvncserver draw the cursor (and there's no reasonable way to turn off cursor drawing in the libvncserver api, as far as I can tell). If you declare the encoding, things should indeed go back to working.

I added the cursor position updates for the stylus because for my client this is sufficient for cursor rendering, but I see that you are using the in/out of prox/contact events. If you'd like to get the stylus data out of the rfb stream, it should be possible to add support for that, although I'm not sure where to put them---perhaps via different cursor shapes for different states, or via textchat messages?

[bordaigorl]

@tbabej @johncokos @Foxei
I pushed on rm2-support a version that should work on both RM1 and RM2.
I only tested it on RM1 and it works as expected.
Could you test it on RM2?
Can you also confirm the pointer tracking works as well?

[bordaigorl]

@pl-semiotics

I added the cursor position updates for the stylus because for my client this is sufficient for cursor rendering, but I see that you are using the in/out of prox/contact events. If you'd like to get the stylus data out of the rfb stream, it should be possible to add support for that, although I'm not sure where to put them---perhaps via different cursor shapes for different states, or via textchat messages?

I think that would be more complicated maybe? The current solution works pretty well and I like that I can detect proximity events too...

[pl-semiotics]

@bordaigorl

I think that would be more complicated maybe? The current solution works pretty well and I like that I can detect proximity events too...
As I was saying, I'd also love to have the prox events so I'm happy to try to find somewhere to stick them sometime in the next week or two (and may do so anyway). I was just assuming that only having one connection would make things simpler (especially since the code's there already) :), although I don't know what rfb client you're using so perhaps not.

[tbabej]

@bordaigorl Ooops, looks like we had unfortunate code collision there! 😅

I tested my PR both on rM1 and rM2, rM1 works fine but rM2 does not recognize the cursor (same cursor behaviour as in the videos above). The screen updates, however, are smooth, quick and clean (the fix with PSEUDO_CURSOR_ENCODING) worked.

[tbabej]

@bordaigorl Looks like your branch has additional changes around the pointer, I'm gonna add that and re-test.

[bordaigorl]

@tbabej oh wow! And the changes are almost identical 😄
Yes my version should correctly pick up the cursor events too. Let me know. If that works I'd merge mine if that's ok.
But thanks a lot for the input!

[tbabej]

@bordaigorl I added the support for mouse events to the PR, mirroring your implementation.

Works quite smoothly on rM1 and both rM2. Demos below:

• rM1:

rm1.mp4

• rM2:

rm2.mp4

[bordaigorl]

@tbabej that's great thanks for testing it!
I'll put together some documentation for the changes and announce the new version.
Thanks again to everyone involved in this issue for the help!

[Foxei]

Okay, now I'm impressed. This is buttery smooth! The only reason to support ssh would be for future proofing. Consider how long it has taken now for both ssh and vnc to work reliably. We are at batch 13 or even 14, so if in the future the kernel is updated or something, we would have the fallback with ssh support.

[pl-semiotics]

There should not be many issues with new kernel version, since for rm2 we do everything in userspace. If libqsgepaper.a is updated drastically it will need to be modified, but it works without modification at least on both 2.4 and 2.5. And the bulk of the work in reverse-engineering &c. is done, so hopefully it will be faster to update in the future.