Picframe Overlay

[helgeerbe]

Actual I use electron to create an overlay for picframe. It's a transparent window running in the foreground and rendering e. g. a weather forecast (https://github.com/helgeerbe/picframe-overlay).

This is a little shaky, because it depends on the order you have to start picframe and electron.

@paddywwoof added a text file input to the clock.

I'm wondering now, how performant it would be if electron would render a bitmap (https://www.electronjs.org/docs/latest/tutorial/offscreen-rendering). Picframe would read this file and render it on top of the actual image.

In this way, you can render any website on top of picframe (weather, calendar, analog clock, .....). This would be pretty cool.

[paddywwoof]

I suppose it would be quite easy to make an overlay that would read an image file. The complexity would be how much flexibility to allow. Maybe a starter would be to simply look in /dev/shm/overlay.png but some way of signalling when to reload that and where to position it might also be minimal requirements. I will think about it a little more.

[helgeerbe]

from PIL import Image

background = Image.open("test1.png")
foreground = Image.open("test2.png")

background.paste(foreground, (0, 0), foreground)
background.show()

Could do the job. Maybe with size and offset info for the foreground image. So it doesn't have to be the same size.

Electron saves the bitmap on changes of the website with a frame rate of up to 240. So this should be hopefully fast enough. Even for an analog clock (counting seconds).

[helgeerbe]

I will prepare a small electron clock app, writing the image to an file.

[paddywwoof]

I suppose the simplest signalling would be to check the os last modified time. Having an image reloaded every second would be quite a load but once a minute should be fine. Also the image could be much lower resolution than the display then scaled up by pi3d using the GPU.

[helgeerbe]

For a static overlay this works pretty well.

foreground = Image.open("/Users/herbe/Development/picframe-overlay/ex.png")
            foreground = foreground.resize(im.size)
            im.paste(foreground, (0, 0), foreground)
            tex = pi3d.Texture(im, blend=True, m_repeat=True, free_after_load=True)

Have to find out how to do the blending within pi3d mapping 2 textures.

[paddywwoof]

Hi, using the CPU (pillow) to do all the blending and resizing will create quite a load on the little RPi. Using the GPU (pi3d) will have scarcely any additional load beyond rendering the existing layers of picframe. However loading from disk has to use pillow so that will always take time. For an analogue clock I might use rotating sprites for the hands.

[helgeerbe]

Hi Paddy,

I agree that for a simple clock sprites would be the better idea. But for me, it's more a speed test.

My intention is to use a smart speaker and mqtt to show some dashbords. Like weather forecast, the family calendar, dates for waste collection, gasoline price, energy state of my solar battery, urgent messages of my home automation …..

It’s quite easy to render everything into a nice png images. And updating this images only every minute should be fine. But I’m curious how much image refreshes my pi 3 can take. For my 1920x1200 images it has a load of 0.2. (all logs are on ramdisk). So there is still some capacity left.

[paddywwoof]

Helge, I also think the best route for doing the overlay is rendering to an image file. That way all the variable complexity can be kept out of picframe - there's always a danger of enhancements making it bloated and un-maintainable.

With pi3d, the image texture dimensions can be different from the pixel dimensions of the plane that is being rendered and the scaling happens automatically and very efficiently (i.e. as objects move around a scene in a computer game). The other efficiency is that the default pi3d.Texture loading will set the blend=False which means that if the texture has an alpha channel, the GPU will discard pixels that are sufficiently transparent without wasting any effort rendering them. The flip side of this is that the pixels that are visible will be drawn using depth priority. If you render a foreground before a background then the pixels that would be masked in the background get discarded. This sometimes causes odd edges - you will see what I mean if you change the constructor argument in the example you posted above (is one of the foreground objects in your im.paste the background object?)

Should I add some extra functionality to the viewer_display.py dev branch, to do the overlay or have you already solved it?

[helgeerbe]

It would be great if you can provide a function which is doing the overlay.

Guess this function has to be called in slideshow_is_running() just before self.__slide.draw() not to affect the blending.

How to handle auto scale, or explicit placing? Adding a parameter [x,y,width,height]? I would start with a full resolution overlay, but maybe a quarter sized overlay image with scaling is good enough?

[paddywwoof]

OK I will knock something up. To start it will just be full size. The quad that the texture is drawn onto will be the size of the display but the texture can be any size. If the aspect ratio is wrong there will be distortion and if the resolution is too low then it won't be sharp.

I will try it just in front of the main image - i.e. behind clock and picture info text. See if blend mode makes the fading jerky.

[helgeerbe]

Perfect, Thanks Paddy

[paddywwoof]

I've pushed an addition to the dev branch 05feb15 which simply pastes whatever image is located at /dev/shm/overlay.png over the picture being displayed. It is behind the clock and info text.

I haven't included any config aspects at the moment, it just takes the image and fits it to the screen. Also the blend mode is False which doesn't give any feathering for blurred edges. Possibly many other things that could be improved.

Paddy

EDIT - I see, just checking the code, that I pushed a version with blend=True but it seems to work well with False so I might change it to that to save a bit of work. Most overlay images will be transparent pixels so this might make a difference.

[helgeerbe]

IMG_5894_small.mov

Thanks Paddy, this is so cool! This is on my Mac. Hope I will find time on the weekend to do some Pi testing.

By the way:

 self.__display = pi3d.Display.create(x=self.__display_x, y=self.__display_y,
                                             w=self.__display_w, h=self.__display_h, frames_per_second=self.__fps,
                                             display_config=pi3d.DISPLAY_CONFIG_HIDE_CURSOR,
                                             background=self.__background, use_glx=self.__use_glx, use_sdl2=self.__use_sdl2)

sdl2 is not released yet for pi3d?

[paddywwoof]

Hi, impressive updating every second! Yes sdl2 is in pi3d main branch but I did find that the pip install of pysdl2 hadn't pulled in the libsdl2-dev so had to manually apt install it on this RPi.

PS I tried an overlay 25% of the screen size (1/16th area) and it looked fine across the room, or even close up.

[helgeerbe]

I did some tests on my pi3 with https://github.com/helgeerbe/picframe-overlay. The overall performance is quite good.
I rendered the clock with 1 frame per second in 1920x1200 and picframe was capable to show it with a load below 2.

[paddywwoof]

That is good; kudos to pillow for being able to load and unpack a png into numpy array so efficiently!

I will make a couple of tweaks so that the brightness also works on the clock and overlay, then, provided it works OK, I will pull into main.

[helgeerbe]

I add a small plugin to picframe-overlay to display the image meta info.

[paddywwoof]

Very neat!
Is that running separately from picframe - just getting the exif info, then generating and writing to /dev/shm/overlay.png? In which case should we pull the changes from dev into main? I've been running dev here for the last week on a Pi3 and a Pi5 and it hasn't had any problems.

Paddy

[helgeerbe]

What I want to test:

1. Using Redis Cache instead of File - Interface.
2. Findout how performant picframe would be (framerate) in sending numpy arrays -> redis -> electron.
   In that way, electron could handle all the user interaction while picframe renders the background.