Stuttering and off-sync rendering when other encodings than jpeg are used

[basilgello]

As previously mentioned in xpra bytes and strings conversiin issue, #20 broke smooth playback of JPEG-encoded stream in HTML5 client.

On my device (Samsung Galaxy Tab S 2014 tablet running LineageOS 14.1 and Chromium for Android 92), JPEG was far tge best encoding option that allowed me to watch even Youtube videos remotely nearly as smooth as they play natively in the same browser. The last commit working properly was b28eadd and I verified that an issue started around a0f50fd (so no lz4.js was removed yet!)

Launching the HTML5 client with draw and network debug options, I noticed that the stutter and the following spike in the network bandwidth usage (~6000kB/sec against ~300-1200kB/sec during normal operation) spews the following in Xpra server's debug log:

2021-08-18 18:35:44,595 client   2 draw may_paint_now() paint pending= 0 , paint queue length= 0
2021-08-18 18:35:44,621 Warning: client decoding error:
2021-08-18 18:35:44,621  unknown cause
2021-08-18 18:35:44,622 client   2 draw animation frame: 1 windows to paint, processing delay 24 ms
2021-08-18 18:35:44,623 Warning: client decoding error:
2021-08-18 18:35:44,624  unknown cause
2021-08-18 18:35:44,642 Warning: client decoding error:
2021-08-18 18:35:44,642  unknown cause
…
2021-08-18 18:35:47,541 Warning: client decoding error:
2021-08-18 18:35:47,541  unknown cause
2021-08-18 18:35:48,001 client   2 draw do_paint( undefined  bytes of   bitmap:jpeg  data  642 x 361  at  100 , 154 ) focused= true

So the issue is that a malformed packet is sent and immediately a spinner or notification pops up.

@totaam ?

[totaam]

I haven't tested much apart from Chrome and Firefox on my local desktop.
It is possible that some browser implementations (especially mobile ones) choke on some of this new code.
Ideally, the decode worker would run a smoke test before responding to the check sent by the client, for now this will do: abfccbd.

[basilgello]

choke on some of this new code.

That's why I am smoking this now ;)

[basilgello]

592da90 reverts isMobile ;) Adding it back restores normal playback.

Now I am going to dump packets that fail to decode so we can investigate what's wrong with them.

[totaam]

592da90 reverts isMobile ;)

Ooops! Thanks, reverted the revert: e9bcce5

[totaam]

Now I am going to dump packets that fail to decode so we can investigate what's wrong with them.

Or you could just add code to the error handlers in DecodeWorker.js.
There's only a two that are relevant for jpeg draw packets:

• xpra-html5/html5/js/DecodeWorker.js

  Line 192 in e9bcce5

  }, function(e) {

• xpra-html5/html5/js/DecodeWorker.js

  Line 254 in e9bcce5

  catch (e) {

[basilgello]

2021-08-18 20:36:13,456 Warning: client decoding error:
2021-08-18 20:36:13,457  failed to create image bitmap from jpeg [object Blob]: InvalidStateError: The source image could not be decoded.

[totaam]

Are you going through a proxy?
I don't see how but perhaps the pixel data arrives as something other than Uint8Array and so data.buffer is undefined, the Blob is empty, createImageBitmap therefore rightfully fails on it.
The old path used a very inefficient base64 string, so it probably still worked OK with that.

Anyhow, I've tested with and without the proxy, with and without rencodeplus... and I can't get it to misbehave.
Only difference is that I'm not on a mobile device.

If you can still trigger errors, perhaps we should save the bytearray of the offending image that fails to decode so we can inspect it.
Please make sure to record the full packet encoder settings used, end-to-end: server connection, both proxy connections (to server and to client) and the html5 client itself.

[basilgello]

Please make sure to record the full packet encoder settings used, end-to-end: server connection, both proxy connections (to server and to client) and the html5 client itself.

I have no python3-rencode package installed so it must be eitger rencodeplus or old bencode everywhere. How do I check which enckder is used server-to-proxy?

[totaam]

How do I check which encoder is used server-to-proxy?

Easiest way is to check the client connection on the server. The proxy is the server's client. So:

xpra info | grep client.connection.encoder=

[totaam]

Oh, but I forgot, you're forcing jpeg-only somehow, right?
Please always specify the full command lines.

[totaam]

So, I'm taking a random Blob from your log file and doing this with it in Python:

data=[255,216,255,224,0,16,74,70,73,70,0,1,1,0,0,1,0,1,0,...]
b=bytearray(data)
with open("test.jpeg", "wb") as f:
    f.write(b)

And although it looks wrong when I open it with an image viewer (only the very top of the image is present), none of the tools complain about it:

from PIL import Image
i=Image.open("test.jpeg")
print(i.size)
(853, 480)

Or even directly to pillow:

data=[255,216,255,224,0,16,74,70,73,70,0,1,1,0,0,1,0,1,0,...]
b=bytearray(data)
from PIL import Image
from io import BytesIO
print(Image.open(BytesIO(b)).size)
(853, 480)

The fact that the top of the image looks OK makes me think that the jpeg data is correct (the header is correct for a start) but perhaps the packet is incomplete somehow?
I'm not sure if it is Javascript that's truncating the bytearray (perhaps only when displaying it in the console log output?) or if it is already truncated on the wire.

Can you try with a desktop browser to confirm that the problem only occurs with mobile devices?
I have tested with all the browsers I have and none of them are having a problem with it.

[basilgello]

Oh, but I forgot, you're forcing jpeg-only somehow, right?

I do specify "JPEG" in HTML5 advanced settings "Encoding" drop-down list

[basilgello]

Can you try with a desktop browser to confirm that the problem only occurs with mobile devices?

Sure!

On desktop Chrome 92 there are no issues. Trying to test with Firefox inside Firefox using profiles :)

[basilgello]

The fact that the top of the image looks OK makes me think that the jpeg data is correct (the header is correct for a start) but perhaps the packet is incomplete somehow?

My bet is that browser codes treat any inconsistency in untrusted input as security risk and discard the whole image. Let me check why some NAL is broken.

[totaam]

FYI: you can get the server to save all screen updates to file using:

xpra start --env=XPRA_SAVE_TO_FILE=1 ...

It will then save a sequence of jpeg files with the exact contents as the draw packets.

[basilgello]

Let me check why some NAL is broken.

Which image editor did you use? Mine is Eye of Mate (eom) on Debian bookworm and it displays every file properly. Same with desktop Firefox. Now I will check them on mobile Chromium that initially failed decoding them.

[basilgello]

Well, recent commits introduced scroll issues if decoder worker is used. Recent, I mean yesterday ones. This manifests only on decoder worker on Android tab, and does not manifest if isMobile check is in place.

[totaam]

Well, recent commits introduced scroll issues if decoder worker is used.

What sort of problem? Errors? Visual corruption?

My guess is that there's something odd going on with your setup and the scroll packets aren't skipped as they should be here:

xpra-html5/html5/js/Client.js

Lines 2785 to 2792 in ad665ea

	if (coding!="scroll") {
	if (!(img_data instanceof Uint8Array)) {
	//the legacy bencoder can give us a string here
	img_data = Utilities.StringToUint8(img_data);
	packet[7] = img_data;
	}
	raw_buffers.push(img_data.buffer);
	}

You could try logging the coding value here.
Like I said, I've tried with bencode, rencode, proxy, no-proxy... and I'm not seeing any problems.

[basilgello]

Visual corruption?

Yes, the part of scrolled-out region rendered in the visible viewport, quite occasionally. Let me capture screenshots when I get back to that network (in 3-4 hours) and log the coding stuff.

As for images failing to decode, the strange thing that every image displays properly even in Chromium for Android whose decoder produced the error output. i.e file:///sdcard/temp/temp01.jpeg renders full image without issues.

As a last resort I can try grabbing a logcat to see if there are any line numbers preceding the image decoding error.

[basilgello]

Yes, the part of scrolled-out region rendered in the visible viewport, quite occasionally.

As expected, visual glitches are result of image decoding errors during fast scroll. I guess I need to build a debug flavor of Chromium to see what triggers the exact error.

[basilgello]

Meh! Firefox 91.2.0 for Android fails to decode every image supplied, the client window rendered has only borders & header and log file is full 9f "failed to decode image".

[basilgello]

What can cause this error:

2021-09-21 15:54:34,626  failed to create image bitmap from png [object Blob], data=137,80,78,71,13,10,26,10,0,0,0,13,73,72,68,82,0,0,0,150,0,0,0,100,8,2,0,0,0,235,57,94,167,0,0,9,213,73,68,65,84,120,94,237,157,71,104,85,205,27,198,141,5,69,197,46,98,141,36,1,11,26,21,43,162,18,52,184,83,80,177,128,27,221,4,116,23,68,55,217,184,16,81,92,186,145,8,234,194,133,5,27,89,136,16,33,162,96,119,161,96,65,77,98,137,13,43,216,219,247,253,62,238,159,203,253,231,206,204,57,83,207,13,204,93,207,60,239,243,60,239,57,231,206,153,51,243,78,183,110,241,23,29,136,14,68,7,162,3,209,129,232,64,116,32,58,16,29,136,14,68,7,162,3,209,129,232,64,23,118,160,44,0,247,158,61,123,150,151,151,143,30,61,250,229,203,151,237,237,237,191,126,253,10,16,52,171,16,189,122,245,26,63,126,252,200,145,35,59,58,58,158,60,121,242,251,247,111,223,76,188,164,176,71,143,30,203,150,45,171,173,173,173,170,170,170,172,172,36,127,8,203,41,65,18,194,90,91,91,31,61,122,212,210,210,114,234,212,169,174,158,81,164,173,88,177,162,166,166,6,177,21,21,21,136,229,146,205,137,69,26,98,31,63,126,140,216,230,230,230,166,166,166,63,127,254,248,206,168,45,254,208,161,67,183,109,219,214,214,214,246,79,186,223,243,231,207,27,26,26,134,15,31,110,27,56,139,254,208,134,60,18,210,105,253,7,91,48,7,139,178,32,155,34,230,196,137,19,27,27,27,191,124,249,146,82,79,97,179,239,223,191,31,58,116,168,186,186,58,69,156,146,104,2,85,8,67,219,64,44,22,97,20,118,149,132,146,60,137,186,186,186,111,223,190,25,232,41,236,242,243,231,207,250,250,250,178,50,47,207,118,87,126,65,15,146,80,181,20,139,93,152,230,138,149,21,78,255,254,253,15,31,62,108,169,167,176,59,255,142,131,6,13,178,226,228,173,51,196,160,231,80,44,214,97,160,55,190,41,128,167,79,159,126,255,254,125,135,146,114,80,12,118,102,205,154,149,34,126,208,38,80,130,152,115,177,24,136,141,65,149,228,131,173,89,179,198,254,225,41,115,132,191,153,141,27,55,102,35,76,20,21,50,102,255,124,105,82,142,141,152,25,90,236,226,197,139,127,252,248,145,134,159,113,27,94,63,150,47,95,30,90,152,40,30,52,32,99,44,36,77,71,204,196,210,112,98,167,76,153,242,241,227,199,52,204,44,219,48,120,155,49,99,70,56,97,162,72,16,48,27,102,235,106,199,82,140,13,33,150,151,161,135,15,31,234,242,51,110,255,236,217,179,49,99,198,132,16,38,138,65,104,8,24,147,215,237,136,177,222,95,145,251,246,237,123,229,202,21,93,102,150,237,111,223,190,61,112,224,192,240,89,36,40,161,45,201,235,118,199,94,76,246,40,246,248,241,227,186,156,156,180,63,123,246,172,71,85,18,104,130,58,33,175,11,130,201,190,196,46,93,186,84,151,141,195,246,171,87,175,246,37,76,132,75,56,135,228,117,161,176,218,189,88,38,115,239,221,187,167,75,197,97,123,230,139,253,62,97,10,60,35,16,225,28,146,215,133,194,234,252,135,1,103,185,220,178,101,139,46,15,231,237,183,111,223,238,76,143,18,136,64,206,201,235,2,98,184,75,177,35,70,140,248,244,233,147,46,9,231,237,25,220,143,27,55,206,165,48,17,22,33,194,188,69,168,253,193,112,108,119,38,118,255,254,253,206,243,97,6,120,236,216,49,103,170,36,64,132,48,227,230,188,23,182,187,17,203,187,45,31,42,157,243,51,6,92,184,112,161,27,97,34,20,192,141,137,57,239,136,237,105,102,54,186,39,218,177,105,211,166,238,221,147,155,37,226,184,106,176,121,243,102,87,80,197,56,94,193,117,105,99,59,230,235,246,234,220,158,177,89,41,252,11,22,94,224,76,10,15,30,60,216,86,152,168,63,176,254,38,238,205,238,81,204,79,28,135,39,220,94,43,87,174,28,48,96,128,15,191,140,49,251,244,233,179,110,221,58,227,238,138,142,192,2,238,3,217,24,19,243,73,129,113,247,255,58,158,63,127,222,236,242,241,218,235,250,245,235,86,170,36,157,129,245,74,219,12,156,20,152,139,101,49,93,73,13,100,10,45,152,58,117,170,185,48,81,79,0,205,44,246,221,139,20,144,8,133,88,213,131,148,239,156,37,53,144,41,148,177,97,195,6,183,41,116,14,232,138,30,41,48,255,250,205,2,72,223,151,152,49,254,235,215,175,29,94,94,64,1,104,76,198,119,71,18,97,114,65,240,169,204,55,51,75,252,201,147,39,155,8,19,245,1,202,146,140,239,238,138,143,166,210,7,233,188,121,243,100,6,253,253,251,247,242,229,203,172,106,101,58,159,53,88,124,110,118,101,165,22,206,156,57,115,180,218,43,26,59,132,114,69,169,19,142,34,29,210,136,187,119,239,22,94,89,172,242,88,178,100,73,97,183,126,253,250,133,255,14,12,183,125,251,246,185,242,11,40,217,109,244,244,233,211,61,123,246,204,159,63,127,246,236,217,187,118,237,202,234,207,133,116,104,139,189,112,225,66,177,42,238,191,245,235,215,23,99,5,94,141,145,35,118,235,214,45,109,85,146,14,64,9,83,88,188,96,128,79,249,87,175,94,245,253,216,44,198,39,29,122,98,217,216,33,156,173,223,187,119,175,12,136,21,230,129,223,64,216,116,146,56,115,145,70,54,32,64,21,187,38,91,182,147,201,245,74,58,242,187,109,210,136,234,54,109,218,52,225,133,54,115,230,76,69,255,139,23,47,6,190,60,23,45,90,148,74,143,178,17,32,66,218,108,206,146,245,147,117,241,42,159,164,8,249,136,135,51,194,63,79,118,88,221,188,121,83,225,198,145,35,71,236,13,213,66,112,50,12,17,130,188,127,255,254,220,185,115,50,50,151,46,93,98,67,147,22,85,251,198,178,17,141,56,133,147,38,77,42,14,121,242,228,73,53,143,19,39,78,216,19,213,66,112,242,94,33,4,97,239,4,123,95,100,100,24,19,4,248,114,217,41,186,48,41,180,17,167,112,200,144,33,197,236,63,124,248,160,246,247,213,171,87,90,9,176,111,236,228,147,133,16,132,20,170,233,157,62,125,218,158,191,22,130,48,41,122,41,212,138,23,166,177,76,149,86,116,33,8,95,121,212,32,225,223,134,245,82,88,186,59,81,255,223,215,97,195,134,105,101,75,216,216,9,136,61,141,68,4,89,82,196,15,82,225,179,133,87,120,117,152,240,251,2,253,61,72,19,13,13,223,64,38,86,227,191,48,113,159,17,85,3,2,11,243,151,194,177,99,199,170,181,36,54,112,110,133,222,191,134,240,85,151,151,30,245,14,113,70,225,94,95,140,132,224,150,111,247,116,23,194,38,14,191,217,107,31,88,172,70,105,16,62,246,203,200,237,216,177,67,118,113,141,26,53,74,150,120,175,82,41,103,99,115,189,211,93,72,143,13,161,138,21,39,189,123,247,206,100,73,145,144,146,222,210,180,173,91,183,10,119,50,114,45,115,217,234,205,0,217,24,239,191,47,73,82,124,4,102,149,77,169,45,41,234,108,137,226,126,98,48,221,105,166,135,66,65,103,206,156,241,122,171,41,192,61,61,72,137,40,219,121,59,119,238,220,76,150,123,107,60,72,201,231,155,55,111,20,174,125,254,252,153,185,180,181,107,215,34,102,231,206,157,25,110,151,97,205,160,253,13,169,88,120,88,124,189,78,152,48,65,109,142,191,75,153,184,66,177,226,34,47,148,97,128,171,189,59,190,17,40,234,198,127,176,101,148,23,47,94,80,50,77,6,194,92,26,95,67,153,59,228,38,88,181,106,213,130,5,11,120,234,88,70,52,235,254,224,193,3,225,112,242,127,213,194,58,129,38,206,165,153,145,112,222,203,9,79,64,20,41,100,89,13,223,123,249,57,39,175,11,40,19,43,30,206,188,123,247,78,55,64,38,237,223,190,125,107,31,215,9,136,61,141,68,4,89,82,196,41,228,83,75,34,98,41,52,112,194,211,9,72,0,55,100,60,187,118,10,93,61,72,3,36,192,62,132,94,10,25,100,218,135,12,128,112,247,238,93,251,40,78,64,236,105,36,34,200,146,34,190,11,25,131,37,34,150,66,131,107,215,174,217,211,112,2,98,79,35,17,65,150,20,241,75,5,243,44,105,182,69,37,70,245,218,128,162,90,172,39,251,250,245,171,101,148,220,6,188,18,159,90,66,38,98,133,101,162,197,119,33,77,111,220,184,97,105,141,239,238,119,238,220,177,207,31,36,1,1,202,55,91,75,124,210,33,43,243,45,157,35,45,253,103,169,195,7,160,67,40,203,84,201,186,43,210,33,77,161,167,61,124,14,21,58,188,200,28,66,57,20,88,8,165,72,71,188,11,255,51,170,75,223,133,170,139,38,171,253,3,105,102,138,227,230,180,124,230,84,223,11,217,181,228,233,177,96,15,11,55,38,160,237,113,114,8,64,149,184,88,67,165,113,163,118,154,231,129,239,54,137,27,181,19,178,27,203,37,248,206,80,34,126,98,185,132,132,133,23,7,15,30,52,188,133,125,118,59,112,224,128,15,120,79,176,150,84,109,83,16,75,7,37,222,37,94,27,164,153,35,75,184,11,153,185,56,122,244,168,229,117,228,182,59,155,25,156,124,160,40,102,5,108,248,157,18,106,115,48,223,193,12,84,44,163,231,245,62,83,128,167,44,163,151,234,14,137,197,44,51,201,162,179,98,150,36,57,150,148,13,159,66,199,37,101,201,98,44,236,28,56,139,142,11,59,147,194,88,94,61,100,10,189,148,87,39,139,241,144,131,96,89,244,114,200,65,110,216,19,143,26,9,144,69,143,71,141,144,194,120,224,143,239,20,122,63,240,135,44,6,46,156,19,143,221,74,245,218,167,219,40,30,126,231,227,94,12,119,248,93,46,223,241,8,74,183,89,12,125,4,101,46,139,241,32,88,87,89,204,230,32,216,92,22,227,113,204,246,89,204,242,56,230,92,22,227,161,232,54,89,204,254,80,244,252,56,168,174,174,206,254,144,14,106,158,213,215,215,151,149,137,23,152,235,142,185,60,181,135,30,36,161,106,147,57,250,98,23,166,121,34,105,8,203,6,212,198,198,70,179,77,232,148,151,160,6,72,117,117,181,97,236,224,221,160,10,97,179,99,210,177,8,163,212,229,95,130,11,42,8,72,145,41,106,118,183,181,181,165,188,72,41,9,217,208,208,224,253,12,98,63,150,64,27,242,72,72,41,22,91,48,199,121,113,52,47,79,45,118,163,83,143,181,182,182,182,170,170,170,178,178,178,188,188,60,127,38,38,27,3,56,160,179,181,181,149,234,166,45,45,45,20,28,212,43,228,224,39,25,54,168,72,163,236,85,77,77,13,98,43,42,42,16,155,223,97,131,52,196,178,28,23,177,205,205,205,77,77,77,124,197,181,137,37,236,235,37,133,157,34,33,9,97,20,233,161,186,65,123,123,123,87,207,153,58,7,100,148,197,155,108,222,239,232,232,32,127,178,189,44,206,19,25,1,163,3,209,129,232,64,116,32,58,16,29,136,14,68,7,162,3,209,129,232,64,116,32,58,16,29,232,162,14,252,11,108,15,96,163,199,145,116,124,0,0,0,0,73,69,78,68,174,66,96,130: TypeError: WorkerGlobalScope.createImageBitmap: 2 is not a valid argument count for any overload.

with xpra-html5 master using Firefox 88 for Linux?

[totaam]

What can cause this error:

I don't know, but it is a good thing that we have the data, we can feed it to the decoder as a smoke test and disable the worker if it fails to process this test.
Do you have any other failing samples?

[basilgello]

I can collect some in the runs tonight!

[basilgello]

Yes, it is still disabled. Let me monitor the situation.

[basilgello]

#110 works smoothly but xpra-org/xpra-html5@master stutters without any error messages in server.log

Selecting WebP dies not stutter but instead of stuttering once in ~3s (the same interval as stuttering, related to timer?) the pixelated frame is displayed.

[totaam]

pixelated frame is displayed

You could run your server with -d compress to see which encodings are used, perhaps these pixelated frames correspond to a specific encoding.

[basilgello]

You could run your server with -d compress to see which encodings are used, perhaps these pixelated frames correspond to a specific encoding.

You are right! The root cause is the selection of available encodings.

Working revision selects jpeg as the only available encoding when I select "JPEG" in advanced optiobs:

2021-11-05 22:37:34,686 setting keyboard layout to 'us'
2021-11-05 22:37:34,919  client root window size is 1280x680 with 1 display:
2021-11-05 22:37:34,920   HTML (339x180 mm - DPI: 96x96)
2021-11-05 22:37:34,921     Canvas
2021-11-05 22:37:35,427 server virtual display now set to 1280x680
2021-11-05 22:37:35,438  automatic picture encoding enabled, also available:
2021-11-05 22:37:35,439   jpeg
2021-11-05 22:37:35,472 debug enabled for xpra.server.window.window_source / ('window', 'compress')

while the latest revision gives all supported encodings to the server:

2021-11-05 22:31:58,222  client root window size is 1280x680 with 1 display:
2021-11-05 22:31:58,224   HTML (339x180 mm - DPI: 96x96)
2021-11-05 22:31:58,224     Canvas
2021-11-05 22:31:58,730 server virtual display now set to 1280x680
2021-11-05 22:31:58,742  using jpeg as primary encoding, also available:
2021-11-05 22:31:58,742   png, rgb32, rgb24
2021-11-05 22:31:58,774 debug enabled for xpra.server.window.window_source / ('window',

so that server sends a mix of encodings. Some full-screen PNGs take up to 123ms compare to 30ms for JPEG for exactly same frame size!
And WebP are the pixelated ones :(

[totaam]

I don't understand why you have png but not webp.
webp is a lot more efficient than png and would not take as much bandwidth or cpu.

Selecting jpeg as the only encoding is problematic because jpeg is lossy.

[basilgello]

I don't understand why you have png but not webp.
webp is a lot more efficient than png and would not take as much bandwidth or cpu.

I can investigate this.

Selecting jpeg as an only encoding…

is fine if user did it explicitly. User knows better, isnt he?

[totaam]

User knows better

Absolutely not. Users have zero clue which encodings are best. The encodings switch is the most misused feature ever added.
In this case, jpeg works better because it uses less bandwidth and less CPU overall, but it is also lossy and doesn't deal with transparency at all.

It makes zero sense to only use jpeg: very small regions will use less CPU and bandwidth with plain rgb. Small-ish regions can be sent as webp without incurring the bandwidth and CPU problems.
IIRC, newer versions always assume plain-rgb is supported.

We need to offer a different UI, one that will allow users to select jpeg behaviour but without doing the dumb thing and using jpeg when it's not suitable.

[basilgello]

For me ot is the only way to watch videos from a remote computer smoothly. I'd be glad if the new combination of webp + rgb delivers same encoding stability and same or better quality but for now I'd like to keep using jpeg.

[basilgello]

I ran the following experiment. I use auto encoding on HTML5 side but vary encodings on server side playing the same YouTube video.

xpra start --encodings=jpeg --start=firefox --daemon=no --debug=compress 2>&1 | grep "using" > /tmp/server-jpeg.txt is the report for "good" playback, while
xpra start --start=firefox --daemon=no --debug=compress 2>&1 | grep "using" > /tmp/server-auto.txt is more laggy and colors are sometimes distorted.

[basilgello]

server-jpeg.txt
server-auto.txt

[totaam]

for now I'd like to keep using jpeg

You misunderstand: I'm not saying that we should not use jpeg but that as long as the selection is betweenjpeg or others, then we're left with bad compromises.

I'd be glad if the new combination of webp + rgb delivers

This is not implemented yet, see also Xpra-org/xpra#3337.
The solution is to use jpeg more with the html5 client (and also clients with poor bandwidth) but not using jpeg when it's just the wrong choice.

[basilgello]

I don't understand why you have png but not webp.
webp is a lot more efficient than png and would not take as much bandwidth or cpu.

webp support is detected in DecodeWorker only, right? And it is disabled on mobile browsers.

[basilgello]

I moved webp temporarily to this.supported_encodings and things got slightly better on speed but pixelation is not cool :(

Why dont we run encoding tests on server start? ie encode a set of predefined images just like in html5 client and build the codec preference list dynamically?

[totaam]

webp support is detected in DecodeWorker only, right?

Correct.

And it is disabled on mobile browsers.

Sort of. The check in DecodeWorker should take care of that since it is disabled on mobile.
But perhaps we can re-instate it now?

I moved webp temporarily to this.supported_encodings and things got slightly better on speed but pixelation is not cool :(

Are you running xpra master?
There are a lot of fixes for webp and latency, with details here: Xpra-org/xpra#3337
I have not seen any pixelation. Can you attach a screenshot?
I assume that this is on mobile otherwise webp would have been enabled automatically?

Why dont we run encoding tests on server start?

Why bother sending samples when we can just embed them in the html5 client? For what benefit?

[basilgello]

But perhaps we can re-instate it now?

After extensive testing :)

Are you running xpra master?

Yes, built this morning

Why bother sending samples when we can just embed them in the html5 client? For what benefit?

Not sending anything. Just bake image sample into server asset and try encoding parts of it of various XxY size with different available encoders to measure encoding time and size on server.

[totaam]

Why dont we run encoding tests on server start?

The server already validates all of its encoders and csc modules on start.
Look for the selftest function in each module.

to measure encoding time and size on server

This can't be used to ascertain the relative performance of various encoders, not reliably.
As for the client side, if it can decode the sample image we bundle, chances are it can decode them all.

[basilgello]

I reverted the configuration changes introduced in Xpra-org/xpra@39c8838 and testing with auto encoding. At least, pixelation on full screen videos is gone, but I have to test scrolling yet.

[totaam]

@basilgello I would much rather not revert the changes to min-quality and min-speed because this would just hide one or more underlying bugs leading to sub-par behaviour. It would be much better to fix those bugs:

• I think that the actual perceived quality ends up being too low for a specific encoding (in which case we may just need to normalize the quality input values for each encoder better) - I will try to check. Which encoding was pixelated for you? can you post the -d compress, xpra.server.source.source_stats around that time?
• quality value swings too much and ends up being too low - I have seen this. I assume that your connection is good enough and doesn't need the server to make major quality adjustments to cope?

[basilgello]

I would much rather not revert the changes

I meant I reverted them in local config file, testing different options :)

I think that the actual perceived quality ends up being too low for a specific encoding

jpeg with quality < 20. Let me send you the log where there will be pixels.

[basilgello]

I assume that your connection is good enough and doesn't need the server to make major quality adjustments to cope?

Yes, my network is stable

[basilgello]

error.log

I would much rather not revert the changes

I meant I reverted them in local config file, testing different options :)

I think that the actual perceived quality ends up being too low for a specific encoding

jpeg with quality < 20. Let me send you the log where there will be pixels.

sorry for delay, here is compress debugging log for Xpra-org/xpra@39c8838 and trunk html5 client with min-speed and min-quality equal to default value 1. especially pixelation grows on full screen video playback in the second part of log.

[totaam]

TILs:

48,512 compress:   6.9ms for 1280x720  pixels at    0,40   for wid=2     using      jpeg with ratio   0.5%  ( 3600KB to    17KB), sequence  1622, client_options={'quality': 1}, options={'quality': 1, 'speed': 1, 'scroll': True}

55,333 compress:  12.9ms for 1280x770  pixels at    0,0    for wid=2     using      jpeg with ratio   7.8%  ( 3850KB to   298KB), sequence   452, client_options={'quality': 95}, options={'quality': 95, 'speed': 36, 'scroll': True}
55,395 compress:   7.3ms for 1280x770  pixels at    0,0    for wid=2     using      jpeg with ratio   2.1%  ( 3850KB to    81KB), sequence   453, client_options={'quality': 35}, options={'quality': 35, 'speed': 1, 'scroll': True}
55,539 compress:   8.1ms for 1280x770  pixels at    0,0    for wid=2     using      jpeg with ratio   1.4%  ( 3850KB to    52KB), sequence   454, client_options={'quality': 15}, options={'quality': 15, 'speed': 1, 'scroll': True}
55,764 compress:  11.4ms for  794x447  pixels at   24,165  for wid=2     using      jpeg with ratio   0.5%  ( 1386KB to     7KB), sequence   455, client_options={'quality': 1}, options={'quality': 1, 'speed': 1, 'scroll': True}
55,938 compress:   7.5ms for  794x447  pixels at   24,165  for wid=2     using      jpeg with ratio   0.5%  ( 1386KB to     7KB), sequence   456, client_options={'quality': 1}, options={'quality': 1, 'speed': 1, 'scroll': True}
56,124 compress:   8.2ms for  794x447  pixels at   24,165  for wid=2     using      jpeg with ratio   1.4%  ( 1386KB to    19KB), sequence   457, client_options={'quality': 44}, options={'quality': 44, 'speed': 13, 'scroll': True}

02,704 compress:  49.6ms for 1280x356  pixels at    0,212  for wid=2     using       png with ratio  18.2%  ( 1780KB to   323KB), sequence   606, client_options={}�[0m
02,724 compress:   3.9ms for  794x447  pixels at   24,165  for wid=2     using      jpeg with ratio   2.2%  ( 1386KB to    29KB), sequence   606, client_options={'quality': 80}, options={'quality': 80, 'speed': 34, 'scroll': True}
02,887 compress:   7.8ms for  794x447  pixels at   24,165  for wid=2     using      jpeg with ratio   0.7%  ( 1386KB to     9KB), sequence   607, client_options={'quality': 13}, options={'quality': 13, 'speed': 1, 'scroll': True}
03,034 compress:  14.9ms for  794x447  pixels at   24,165  for wid=2     using      jpeg with ratio   0.7%  ( 1386KB to     9KB), sequence   608, client_options={'quality': 13}, options={'quality': 13, 'speed': 1, 'scroll': True}

So the pattern seems to be that in between lots of scroll updates using jpeg for the new data, we get a png screen update that uses ~8 times more bandwidth then the quality drops suddenly (probably as a result of slow packet echo and / or decoding).
Another pattern in the logs is both quality and speed dropping very low with a 1280x720 region (720p video?).

I'm still doing lots of fixing in this area, bear with me.

[basilgello]

Another pattern in the logs is both quality and speed dropping very low with a 1280x720 region (720p video?).

Yes, 1280x720 :)

[totaam]

@basilgello is this still an issue?
There are many major fixes that may impact this ticket in Xpra-org/xpra#3337

[basilgello]

is this still an issue?

From my testing yesterday, it is. Let me grab a -d compress log to see what changed.

Right now I do use completely stock configuration (i.e no min-quality and min-speed overrides), and stock HTML5 client on Chromium for Android.

[totaam]

.. for Android

So that's going to be without webp then since that's only enabled via the decode worker.
It should still work fine.

[totaam]

Now that the decode worker is enabled on mobile again (0f51cab), you should be getting webp for those lossless screen updates.
It's still a lot more costly than jpeg, but it should help.

[totaam]

Should be much improved thanks to #198