tinydrm progress reports

[notro]

tinydrm is an attempt at making a drm version of fbtft.

I will post progress reports in this thread as the tinydrm work moves forward.
If you want to be notified, click the Subscribe button to the right.

One change from fbtft is that we now write drivers for displays, not controllers. The main reason for this is that we're not allowed to have the binary init sequence in the Device Tree and it's impossible to cover all the possible controller configurations with individual properties.
The controller specific code is put in a module of it's own and contains the necessary code to update the display memory.

This is an example of how a display driver might look like: ada-mipifb.c
This is how a controller module might look like (many controllers are mipi compliant): mipi-dbi.c
But this could change depending on the feedback from the drm maintainers.

I haven't posted anything on the drm mailinglist yet, just a very short discussion.

[ladyada]

(fyi i have subscribed to this thread - ready to test anything)

[notro]

It took a while, but the first patchset is away: drm: Add support for tiny LCD displays

Based on the review it seems that I still have some way to go, so no need for testing yet. Need to get the API stable first.

[ladyada]

thx for the update!

[xor-shift]

How do i setup tinydrm? (Yes im new.)

[notro]

[RFC v2 0/8] drm: Add support for tiny LCD displays

The feedback is focusing on the parts of tinydrm that will go into the core drm helpers. So I will concentrate on getting those parts accepted before I do a new tinydrm patchset. Hopefully the fbdev deferred io part will land in 4.7.

Patches can also be picked from here: https://github.com/notro/linux/commits/tinydrm-rfc2 (based on rpi-4.5.y)

How do i setup tinydrm? (Yes im new.)

See wiki.

[beardedbadger]

@notro do you have any recommendations on where to look for a jump start on interfacing a TFT with the Parallel RGB of an imx6. we're looking to add a custom display to our linux device...seems like we could use a lot of the elements of FBTFT but you're messages suggest abandoning that as the maintainer is not accepting any more drivers.

[notro]

@beardedbadger This thread is for tinydrm announcements, so I have move your question here: notro/fbtft#391

[notro]

This is just a note to say that the project is still alive. I needed a break after working on it for 6 months straight. After a 4 month break, I'm now back working on it.
Parts of tinydrm has been merged like the fbdev deferred io code and the drm_simple_display_pipe helper.

Currently I'm looking at the possibility to write userspace drm drivers: https://github.com/notro/udrm-kernel
If I can get the same performance from userspace, then I think that's a better solution. It looks promising so far.

This is the userspace driver I use for testing: https://github.com/notro/utinydrm
It's very ugly, I just copied the tinydrm driver into userspace and put up some scaffolding around it.

When I started with fbtft, I tried to make it as easy as possible for others to write drivers, even if they hadn't done any kernel coding before. But it turned out that not many actually did write drivers, so I made the drivers as flexible as possible using module parameters. This way the drivers could be used with many displays, without having to build a new kernel.
This possibility will go away with tinydrm. Because of Device Tree policy, we can't have an inititialization sequence property like fbtft does. This means that there will be one driver per display (or init sequence). Which amounts to a lot of drivers. And someone has to write those drivers and get them merged in the very busy subsystem that drm is.

So the reason I'm detouring on userspace drivers, is because I want as many people as possible to be able to write drivers for these displays.
Even if I can't get it to perform good enough for 320x240 spi displays, it should work for i2c, smaller spi displays and e-ink displays.

It would have been really cool to see a drm userspace driver for this: https://learn.adafruit.com/1500-neopixel-led-curtain-with-raspberry-pi-fadecandy/overview
That is something that would "never" happen if it is kernel only drivers.

[mattdf]

What's in the Device Tree policy prevents init sequences? Also, how much slower would the userspace driver be for SPI displays?

Currently 320x240 SPIs are barely usable as it is, anything that requires X will be unworkable if it gets any slower.

Would there be some way to write the userspace drivers in a way that they're easy to shove into the kernel if one wanted to build a custom kernel with them?

[dlech]

What's in the Device Tree policy prevents init sequences?

The current consensus among kernel maintainers is that device tree "describes the hardware" and any configuration parameters (especially anything linux-specific) are rejected. To get a sense of the bigger picture, here are some slides from the recent Linux Plumbers Conference about this.

So the reason I'm detouring on userspace drivers, is because I want as many people as possible to be able to write drivers for these displays.

Personally, I would much rather write kernel drivers so that I don't have to maintain a userspace driver on my own. In fact, if you don't mind ccing me when you submit patches for the mainline kernel, I would like to get familiar with the simple drm helpers so that I can submit drivers for the displays that I am interested in.

[notro]

Currently 320x240 SPIs are barely usable as it is, anything that requires X will be unworkable if it gets any slower.

X is one of the things that should perform better with a drm driver.
The reason for this is that fbtft relies on page faults on the video memory to know when to update. This means that all updates is as wide as the display, because it's impossible to know which part of the page that got changed.
With drm, X can tell which rectangle it did update making it possible to do smaller changes. This is a comment I made about a quick test I did:

	/*
	 * TODO: Add support for all widths (requires a buffer copy)
	 *
	 * Crude X windows usage numbers for a 320x240 (76.8k pixel) display,
	 * possible improvements:
	 * - 80-90% cut for <2k pixel transfers
	 * - 40-50% cut for <50k pixel tranfers
	 */

So if I didn't mess up anything with this test, there's quite some improvements to be made compared to fbtft.
Most controllers I have seen, support updating rectangles like this.

But full screen updates is the difficult metric, and it's not just used for video playback, because with drm it's possible to do rendering on the gpu and then scanout the resulting buffer on a spi display (PRIME). I would guess that this requires a full scanout every time, because that's what the app expects when it does buffer/page flipping.
I have only tested that I can import and show a buffer, not done any real life experiments. So I don't know if it will actually work with the low 20fps.

Would there be some way to write the userspace drivers in a way that they're easy to shove into the kernel if one wanted to build a custom kernel with them?

Not if I end up doing only userspace drivers, but that's to early to tell right now.

[notro]

Personally, I would much rather write kernel drivers so that I don't have to maintain a userspace driver on my own. In fact, if you don't mind ccing me when you submit patches for the mainline kernel, I would like to get familiar with the simple drm helpers so that I can submit drivers for the displays that I am interested in.

If we end up doing only userspace drivers, then I will add a repo to collect those drivers.
But I have to discuss this userspace approach with the drm maintainers first to see what they have to say.

What do you see as a downside with having the drivers in userspace?

The first driver to use the simple drm helper has just landed, but it doesn't use the framebuffer dirty callback: https://git.kernel.org/cgit/linux/kernel/git/next/linux-next.git/tree/drivers/gpu/drm/mxsfb

[dlech]

What do you see as a downside with having the drivers in userspace?

The main downside is that the userspace drivers will have to be packaged or manually installed on whatever system/OS you are using. So, if I am to use this in the ev3dev OS that I maintain, it is one more package for me to maintain and one more upstream project that I have to follow. Whereas, if the drivers are in the kernel, there is no extra work for me.

Also, it is just a minor thing, but with a kernel driver, you can use the display in early boot, for things like a boot splash.

Thanks for the link.

[ladyada]

as a thought for the custom initialization details: could we set up a 'generic' 320x240 & 320x480 SPI kernel driver and then have a userspace tool send the init commands over an IOCTL or similar? id say at this point, 3 or 4 display chipsets dominate the 'small display' market (ili9341, hx8357d, ili9328 etc) so you can set them up with basic generic init and then load the special init commands for gamma correction, rotation, and bias voltage later.

[notro]

@ladyada It's a possibility to have all but the init sequence in-kernel, but performance doesn't seem to be an argument.
The kernel driver uses 51ms on a full 320x240 update spi@32MHz which is ~19fps.
The userspace driver uses 53ms which is ~18fps.

[notro]

There are Raspberry Pi HAT displays with an onboard eeprom containing a Device Tree fragment.

If we end up with userspace drivers, it shouldn't be a problem with backwards compatibility:
A udev rule detects an fbtft device, say fb_ili9341, loads the udrm kernel module and starts the necessary userspace driver.
The driver can pick up properties from DT via /proc/device-tree.

If we do kernel drivers, then maybe we're allowed to add fbtft legacy support to the drivers, I don't know.

[dlech]

If we do kernel drivers, then maybe we're allowed to add fbtft legacy support to the drivers, I don't know.

I think yes. What I hear from kernel maintainers is that if there is something already in the mainline device tree, then the kernel supports it indefinitely, even if the device tree is "wrong". So, if the new drm drivers can use the same device tree bindings as fbtft, this will be the only way that fbtft can be removed from staging.

[notro]

The userspace driver approach wasn't such a good idea:

[RFC 0/6] drm: Add support for userspace drivers

> Hi,
> 
> I was previously working on tinydrm as a replacement for staging/fbtft.
> During a break from that work, I started to think about if it would be
> possible to move the drivers to userspace instead. No point in having
> them in the kernel if it's not necessary.
> 
> This patchset includes all the pieces necessary to get a userspace
> driver[1] working that is compatible with the fbtft/fb_ili9341 driver.
> It is tested with a SPI interfaced display hat/shield for the Raspberry Pi,
> which has an eeprom with a Device Tree fragment on it (merged by the
> bootloader). With the help of udev and systemd, the driver is able to
> autoload when the display is connected.
> Performance wise, the kernel driver can do ~19fps on a 320x240 spi at 32MHz
> display, whereas the userspace version does ~18fps. So performance is not
> an argument to keep the driver in the kernel.
> 
> What do you think about this approach?

It's a pretty nifty thing, and iirc some of the ideas for implementing
miracast centered around the same idea: Small userspace drm driver shim
that forwards everything to the miracast code running all in userspace.
David Herrmann looked into that years ago, not sure he ever got the full
stack up&running.

For dumb panels itself I'm not sure it's the right design. Adding a
kernel/userspace interface will make code sharing harder, and experience
also says that we'll rework kms semantics every few years. In-kernel
drivers seem to me like the better default choice, except when there's a
clear reason for why userspace is the only option. For miracast that's the
case, since porting the userspace video encode libraries to the kernel
just doesn't make sense. For SPI and other slow buses I don't see any such
compelling reason to move it into userspace.
-Daniel

It wasn't a waste of time though, because it made me rework the emulation code (byte swapping RGB565) in tinydrm (16-bit SPI not supported on Pi which is little endian) which made the code easier to read.
By accident I discovered that for 9-bit displays, if I make a copy of the backing buffer (memcpy) before transforming it byte by byte (D/C pin is a 9th bit), I can increase the framerate by 40-50% @32mhz.
I have no idea why, an extra buffer copy should decrease not increase performance. Maybe it has something to do with the source buffer being mapped into userspace or DMA write coherent, I don't know.

Edit:
Got 26% increased framerate on the common Option 3 (D/C pin) display: 1bca15e

[notro]

[PATCH 0/9] drm: Add support for tiny LCD displays

The review went well, only minor things left to do.

I have now understood why I get a speed up if I make a copy of the pixel line before swapping the bytes:
dma_alloc_wc() allocates write-combined memory which has uncached reads.
By making a copy, I'm actually caching one line, which is faster than reading 2 bytes at a time from memory.
Ref: https://fgiesen.wordpress.com/2013/01/29/write-combining-is-not-your-friend/

So now i get 27 fps on a 320x240 display @48MHz.

Edit: This is nonsense, fbtft doesn't read from the tx buffer, but tinydrm performs better than fbtft...
(fbtft currently has the same issue, but it is resolved in linux-next using kmalloc for it's tx buffer)

[pdp7]

@notro Good to hear. Thanks for all your efforts! I'd like to try out tinydrm on the BeagleBone.

Are there particular LCDs that would be good to test?

In the past with fbtft, I've used Adafruit 1.8" [PID 358], Adafruit 2.2" [PID 1480] and Adafruit 2.8" [PID 2090]

[amitesh-singh]

I have few LCDs too and i could try them on rpi2. plz do let me know if i could be a help.

currently i have 1.8" ST7735 (spi), 2.4" ili9341(spi), 3.5" ili9481 (8 bit bus)
and 3.95" ili9481 (16bit bus).

fbtft was a good excuse for me to learn abt FB drivers. (https://github.com/amitesh-singh/ldd/blob/master/tfts/tftframework/4dtft.c)

[notro]

The patchset will only apply cleanly on linux-next, there's been several changes to drm header files and other changes since 4.10 that tinydrm relies on. I'm too late for 4.11 so tinydrm will be available in 4.12.

The patchset only supports rpi-display and Adafruit PiTFT 2.8" since they both have a MI0283QT display panel.
I have updated the wiki with the DT overlay I use.

And I'm reminded that I will probably not add support for 8080 parallel bus in tinydrm. At least not initially. The reason is that driving the bus through the gpio subsystem is very slow. Some boards like Pi and BBB (I believe) has hw support for parallel bus, but there's no Linux subsystem for a parallel databus.
But, struct gpio_chip has a set_multiple() callback, so drivers that implement this may be "fast enough". See gpio_chip_set_multiple(). Currently there seem to be 11 drivers that implement this callback: https://github.com/torvalds/linux/search?utf8=%E2%9C%93&q=set_multiple

I actually don't know how widespread the use of parallel bus connected displays are. It can't compete with SPI on performance (with DMA), so why bother?

[pdp7]

@notro thanks for the insight. You linked to PID 1601 for the Adafruit PiTFT 2.8". Is that correct?

The backside product image seems to indicate that it is ILI9341.

[notro]

I can't find the notes, but IIRC 3 of the Adafruit displays use(d) the same init sequence: 1601, 2090 and one more. So 2090 will probably work with mi0283qt.
I looked at fb_ili9341 and it uses the mi0283qt init sequence, so any display that uses fb_ili9341 should be able to use mi0283qt.

Continuing on the parallel bus support:
Maybe I should add support after all. The .set_multiple() callback has the potential to satisfy my need for speed, and it probably won't be many lines of code to support it.
It has been nagging me that even I could support fbtft Device Trees, it wouldn't be full backwards compatibility without the parallel bus support.

[notro]

[PATCH v4 0/7] drm: Add support for tiny LCD displays

The only thing missing now is a DT maintainer ack.

[timonsku]

Congrats on getting accepted for Kernel 4.11!
And just to let you know that I highly appreciate the work you are doing, don't let angry mails from Linus discourage you.

[pdp7]

Hooray! Great news @PTS93. Well done @notro!

[notro]

Yes, tinydrm has crashed into Linus' tree.
I expected it to go into linux-next first and then arrive in 4.12, but here it is.

It's been a difficult task to condense the complexity of drm into something simple that the maintainers would accept. Daniel Vetter (drm maintainer) has been very helpful in guiding me along the way. So now there's a foundation for fbtft type drivers.

I have a fatigue illness that keeps getting worse each year. It's now at a point where I can't spearhead this work anymore.
It's easy to make code that works, but it takes a lot more effort to do it in a form that kernel maintainers accept. To keep going until they're satisfied.

This means that we need someone else to step up and do the actual work of converting and preparing drivers for submission.
I will continue as tinydrm maintainer and do reviews, but I can't do the work.

I have converted the fbtft module to tinydrm so the fbtft drivers do work, but that will never be accepted by maintainers.
I have more or less converted fb_ili9325 to a form that is much closer to something that can be accepted. See https://github.com/notro/tinydrm

One driver should be picked and submitted for review to find out if backwards Device Tree compatibility will be accepted.
In my experience just asking in an email hardly ever gets an answer. Patches are the lingua franca of the kernel mailing lists. It's the Device Tree maintainers that have the final say I guess.

Thanks for the encouragements along the way. It's been times that I thought of quitting, but I wanted to get it done for the maker community.
So I'm glad I made it, and I'm glad that I don't have to spend that amount of effort on this anymore.

[pdp7]

@notro thanks very much to all of your efforts!

[computermouth]

Thanks @notro!

[amitesh-singh]

Thanks @notro for your great efforts!
I have learned about linux framebuffer by reading your code.

Thanks a lot!

[notro]

For the Raspberry Pi crowd I've built a 4.9 kernel for easy testing on a stable kernel and a 4.10 kernel with headers for low entry development: https://github.com/notro/tinydrm/wiki/board:-Raspberry-Pi

[notro]

There's a new tinydrm driver for some small e-ink panels (avail in 4.14): https://cgit.freedesktop.org/drm/drm-misc/tree/drivers/gpu/drm/tinydrm/repaper.c

tinydrm has been backported to 4.9 for the Raspberry Pi: raspberrypi/linux#2119

[ladyada]

wow that repaper driver is really simple, looks awesome. nice work @notro :)

[tvoverbeek]

@notro, tested your repaper driver on a 2.7" Papirus display.
Even mapping the console to the display (/dev/fb1) works nicely.
Can even use it as a small terminal without problems.
However the gray8 to mono you use (lines 502-520 in repaper.c) does not treat colors very well.
On a console most colored items (e.g. in bash prompt) become invisible, since they map to white.
Is it possible to use Floyd-Steinberg dithering in that routine?
Would also make things like the image viewer fbi work much better on these epaper displays.
Maybe even using the RPi camera with fbcp would work.
Anyway, thanks a lot for your work on this.

[notro]

On a console most colored items (e.g. in bash prompt) become invisible, since they map to white.
Is it possible to use Floyd-Steinberg dithering in that routine?

I have considered it, but it isn't the kernel job to do stuff like this. If userspace (lib, apps) already had support for drm monochrome/greyscale, I would have used that instead, but I see no point in adding it when 'no one' will add support for such formats. So I took the best supported format XRGB8888 and did a simple brightness conversion (plymouth, boot splash, only support xrgb8888). The downside is that userspace needs to know that the display is monochrome for good result.
We can add a drm monochrome format if someone really needs it I guess.

For bash to look good, you can change the colors it uses.
For fbi, maybe you can put the image through a Floyd-Steinberg filter before giving it to fbi.

[notro]

New tinydrm driver for LEGO MINDSTORMS EV3 LCD in 4.14: https://github.com/torvalds/linux/blob/master/drivers/gpu/drm/tinydrm/st7586.c

I'm trying to make tinydrm more useful on all platforms by moving away from buffers allocated with dma_alloc_wc() to using vmalloc() instead: https://lists.freedesktop.org/archives/dri-devel/2017-October/154048.html

I'm also working on supporting USB devices: https://lists.freedesktop.org/archives/dri-devel/2017-September/152115.html

[pdp7]

@notro thanks for the update and all your efforts on fbtft and tinydrm!

[notro]

Two new tindyrm drivers by @dlech for 4.16:

• ili9225 For panels like this: https://github.com/Nkawu/TFT_22_ILI9225
• st7735r For panels like this: Adafruit 1.8"

[pdp7]

@notro @dlech thanks for your efforts!

[notro]

@anholt has just ported hx8357d.c to tinydrm (DT binding).
It's for the Adafruit 3.5" display. It will show up in linux-next in a couple of weeks I guess.

Another great think is that he's done some work to offload rendering to the vc4 driver: anholt/linux#10

[ladyada]

yay!

[notro]

tinydrm as a concept will be gone in Linux 5.4.
All the tinydrm functionality has now been moved into drm core helpers.
The folder will now contain any drm driver that is so tiny that it fits into one sourcefile:
https://cgit.freedesktop.org/drm/drm-misc/tree/drivers/gpu/drm/tiny

2 years ago I got an idea to turn a Raspberry Pi Zero into a $5 USB to HDMI/SDTV/DSI/DPI/DBI adapter. I have spent the time since to clean up tinydrm and do core prep work and now I've finally started on the driver. If all goes well, it will be a generic USB display driver. This means that it will be possible for anyone to make a USB display without having to write a kernel driver.
What I don't know yet is what kind of speeds I can get. I'm hoping that lz4 compression can help me here since it's fast and available in the kernel.

I'll post here when I got some code to show.

[ladyada]

sounds neat! i just wrote a tinydrm display driver for a new high resolution mini st7789 display cause none of the existing drivers would support it. luckily it looks like drm is not much different than tinydrm so i think my code will work just fine.
it seems fbtft is still in there in staging too :)
i did find a bug in 4.19 (raspberry pi) kernel where if you have odd-value width (e.g. this display is 135x240 pixels) it causes a kernel panic. i have no idea where the bug is :( for now i'm just adding 1 to the width to make it even

[notro]

I have finally something to show for wrt. the idea to turn a Raspberry Pi Zero into a $5 USB to HDMI/SDTV/DSI/DPI/DBI adapter.

I did end up with a generic 'Multifunction USB Device' that can support multiple functions like display, gpio, adc, etc.

[RFC 0/9] Regmap over USB for Multifunction USB Device (gpio, display, ...)

If this doesn't work out, then I'll fall back to a pure display driver.

lz4 compression does help for the display driver.
The transfers resulting from just starting X to the desktop is shown, gives a 3.8 compression boost:

# cat /sys/kernel/debug/regmap/mud-drm.0.auto/usbinfo
USB interface:     1-1.3:1.0
Regmap index:      0
Max transfer size: 4194304
Tag:               92
Number of errors:  0
Number of resets:  0
Compression:       lz4
Compression ratio: 3.8

If I disable compression I'm getting 5 fps for a 1.5MB framebuffer (7800 kB/s):

$ modetest -M mud_drm -s 35:1024x768@RG16 -v
setting mode 1024x768-60.00Hz@RG16 on connectors 35, crtc 33
freq: 5.07Hz
freq: 5.06Hz

This is slower than I hoped for. I've discovered that reading is almost 3 times faster than writing.

The zero gadget (loop testing) confirms my findings:

$ sudo ~/testusb -a -t 27 -g 64 -s 16384
/dev/bus/usb/001/010 test 27,  107.230669 secs		-> 1000 / 107 =  9MB/s
/dev/bus/usb/001/010 test 28,   37.791292 secs		-> 1000 / 37  = 27MB/s
[73983.796552] usbtest 1-1.3:3.0: TEST 27: bulk write 1000Mbytes
[74205.060204] usbtest 1-1.3:3.0: TEST 28: bulk read 1000Mbytes

$ sudo ~/testusb -a -t n -g 64 -s 16384
/dev/bus/usb/001/010 test 5,  107.421535 secs
/dev/bus/usb/001/010 test 6,   38.189712 secs
[74893.204170] usbtest 1-1.3:3.0: TEST 5:  write 1000 sglists 64 entries of 16384 bytes
[75012.592222] usbtest 1-1.3:3.0: TEST 6:  read 1000 sglists 64 entries of 16384 bytes

I have tried Pi1 and Pi4 as host (2 different controllers) and Pi Zero and Beaglebone Black as device, getting the same result.
I will look more into this when I get my hands on a usb3 capable device (if I can afford it).

[notro]

The USB display driver is now included in Linux 5.13: https://github.com/notro/gud/wiki

With this driver I'm winding down my involvement into Linux display drivers and move more into maintainance mode.

It's been an interesting and rewarding ride starting in 2012 with a Raspberry Pi and an Adafruit 2.2" display which resulted in fbtft and 5 years later I was doing refactoring work in the Linux DRM subsystem. You never know what the future holds in store for you :)

Thanks for all encouragement along the way.