Encoder improvements. 118 KB/s #54

sz3 · 2021-03-05T13:55:19Z

switched cimbar.js to the "shaky cam" animation to allow quicker detection (and rejection) of in-between frames by the decoder
- rotate wasn't cutting it. A lot of CPU cycles were getting burned trying to decode unusable input images.
since it's now the default, I improved the shaky cam animation to be a bit less visually jarring (humans have to look at it too, not just the computer)
added a mostly-white background to cimbar.js/.org to help auto-exposure behavior
- it's silly how much this helps -- it emphasizes that there are likely plenty of performance wins that would come from writing a better android app than cfc
improved frame-pacing of cimbar.js
- probably need to port the improvement cimbar_send at some point
added full screen menu option to cimbar.js

The result of the frame pacing fix + auto-exposure jedi-mind-trick is some new benchmark numbers:

943+ kilobits/s (~118 KB/s) for 8-color cimbar.
838+ kilobits/s (~104 KB/s) for 4-color cimbar.

https://github.com/sz3/libcimbar/blob/2800ed433f8133312e0b188df47e2931aedf32d5/README.md

(still using the shaky_cam object, though its days are likely numbered)

Not sure where the magical "shake" calculation should live -- it's tied to the shader, but feels weird to prop out into the window_* classes. Might pass in the dimensions to gl_2d_display(), it kinda makes sense for it to have them

It's slightly more data to copy to the GPU (1040x1040px vs 1024x1024), but everything else is much more sane...

I kinda want to leave rotate as not implemented? Maybe I'll add a todo

... not an interpretive barcode dance. Less snarkily, I'm trying to find the right balance between "decoder can discard bad frames" and "user gets a headache looking at the barcode".

It'd be nice to have a test to catch this 🤔

The idea here is to improve frame pacing.

= cleaner frames = better transfer rates. Checking in a cool gradient, because I can't resist

8-color probably needs higher ECC... 🤔

100 KB/s is defeated

sz3 · 2021-03-05T13:58:29Z

PERFORMANCE.md


 ## Current sustained benchmark

 * 4-color cimbar with ecc=30:
- * 2,980,556 bytes (after compression) in 36s -> 662 kilobits/s (~82 KB/s)
+ * 4,717,525 bytes (after compression) in 45s -> 838 kilobits/s (~104 KB/s)


4-color being over 100 KB/s is very nice.

sz3 · 2021-03-05T13:59:46Z

PERFORMANCE.md


+* other notes:
+ * having better lighting in the frame often leads to better results -- this is why cimbar.org has a (mostly) white background. cfc uses android's auto-exposure, auto-focus, etc (it's a very simple app). Good ambient light -- or a white background -- can lead to more consitent quality frame capture.


Misc notes for how to achieve peak throughput. There may be other stuff, but this is what I know.

sz3 · 2021-03-05T14:04:48Z

src/exe/cimbar_send/send.cpp

 return true;
 };

 Encoder en(ecc, cimbar::Config::symbol_bits(), colorBits);
 while (running)
 for (const string& f : infiles)
- en.encode_fountain(f, draw, compressionLevel);
+ en.encode_fountain(f, draw, compressionLevel, 8.0, window_size);


I may go a different direction with this at some point. We're passing in the size here to embed the 1024x1024 image in a larger image -- which will then get passed to the cimbar::window (which won't have to worry about resizing it).

This is somewhat silly -- the window should be able to render an image on a larger background.

But you know what else is silly? Trying to get transpose and scaling to work in a GLSL shader without hitting weird half-pixel blurring issues.

sz3 · 2021-03-05T14:08:21Z

src/lib/encoder/test/EncoderTest.cpp

@@ -56,7 +55,6 @@ TEST_CASE( "EncoderTest/testFountain", "[unit]" )
 {
 std::string path = fmt::format("{}_{}.png", outPrefix, i);
 cv::Mat img = cv::imread(path);
- cv::cvtColor(img, img, cv::COLOR_BGR2RGB);


Here's some silliness -- image_hash doesn't care about color.

sz3 · 2021-03-05T14:08:56Z

src/lib/gui/gl_2d_display.h

+ {zero, zero},
+ {zero-shake, zero-shake},
+ {zero, zero},
+ {zero+shake, zero+shake}


Here is the shakycam pattern.

sz3 · 2021-03-05T14:09:15Z

src/lib/gui/gl_2d_display.h

 in vec4 vert;
 out vec2 texCoord;
 void main() {
 gl_Position = vec4(vert.x, vert.y, 0.0f, 1.0f);
 vec2 ori = vec2(vert.x, vert.y);
 ori *= rot;
 texCoord = vec2(1.0f - ori.x, 1.0f - ori.y) / 2.0;
+ texCoord -= tform;


baby's first shader

sz3 · 2021-03-05T14:11:23Z

src/lib/gui/shaky_cam.h

 }

 img.copyTo(_frame(cv::Rect(offsetX, offsetY, img.cols, img.rows)));
+ cv::cvtColor(_frame, _frame, cv::COLOR_BGR2RGB);


It's not really clear where this should live, but for now it's here. We spit out RGB images, opencv wants BGRs, at some point before we render to screen (or to file) with opencv we need to do the conversion.

sz3 · 2021-03-05T14:12:24Z

src/wasm/cimbar_js/wasm.cpp

@@ -39,34 +41,43 @@ int initialize_GL(int width, int height)
 return 1;
 }

+// render() and next_frame() could be put in the same function,
+// but it seems cleaner to split them.
+// in any case, we're concerned with frame pacing (some encodes take longer than others)


sz3 · 2021-03-05T14:14:44Z

src/wasm/cimbar_js/wasm.cpp

 unsigned required = _fes->blocks_required();
 if (required > cimbar::Config::fountain_chunks_per_frame())
- required = required*4;
+ required = required*8;


Decided to increase the amount of fountain chunks before we loop around. There's a somewhat squishy point at which weird things happen (large file -> 8x blocks -> exceed wirehair's uint16_t limit for block count -> disaster???) that needs further investigation, and probably a sanity check.

sz3 · 2021-03-05T14:16:52Z

web/index.html

@@ -14,7 +14,8 @@
 }

 body {
- background-color: black;
+ background-color: white;
+ background-image: radial-gradient(circle at top left, rgb(7,0,0),rgb(244,244,244),rgb(255,255,255));


Change CSS, improve throughput by 20%.

b39469c Merge pull request #56 from sz3/cimbar-send-refactor2 3e1e435 Update dependency list to single out GLES 2da86fe Find clang 257973d Slightly more readable, maybe? 0a6bda1 killing the opencv_highgui code path after all 50d65a7 Fix cimbar_send encode_id. 8216c3b Bugfix -- reset _frameCount when we reset the stream 0ca7dd1 Rename the fountain_encoder_stream's `reset()` to `restart()` 22b58ff Switch cimbar_send to use cimbar_js 6b7022f Add header for cimbar_js, and add extra params to configure() 71ab9ea Moving cimbar_js into src/lib/ 4415602 Merge pull request #55 from sz3/decode-more 918ffb2 Update build instructions to match reality 6376e12 indentation fc5ef1c Merge remote-tracking branch 'origin/master' into decode-more 79a91b2 Smaller image 9ffb94e Merge pull request #54 from sz3/glshake 61fff8f I don't think I mentioned compression anywhere. fc04ea6 more docs? 8e5fa90 Misc docs updates, including new performance numbers! 206c786 Add a few tests for the larger size bf2c492 Misc code cleanup f8408b3 I think document.documentElement is what I actually want 7247339 Fullscreen toggle? 48b2d1a Spending my time on important things, like gradients e340559 New numbers, and they are quick a865dd4 Simpler gradient, same effect 5b86c29 WIP: having more light (white) helps the camera 4c5f826 I like this interface slightly more. f979012 🤔 0626afc Tired of the mouse cursor getting in the way... 21e39db Update help 6933d3e fix color bug with highgui c15ac29 The motivation of the shaky_cam is to detect cross-frame overlap, 7be3d91 Do the preprocessing sharpen after the grayscale a49fe7f Only dark mode for now 87338c4 Reimplement shakycam for opencv_highgui a2ffe08 maybe this instead -- variable canvas size for CimbWriter 396939a It's something like this, but... f6b6496 I don't know about this one chief 355ef7c Is this better? d869656 use loop_iterator? cb84a59 hmm 0de861b quick WIP for GLSL transpose for the shakycam effect git-subtree-dir: app/src/cpp/libcimbar git-subtree-split: b39469c

sz3 added 26 commits February 19, 2021 20:50

quick WIP for GLSL transpose for the shakycam effect

0de861b

hmm

cb84a59

(still using the shaky_cam object, though its days are likely numbered)

use loop_iterator?

d869656

Not sure where the magical "shake" calculation should live -- it's tied to the shader, but feels weird to prop out into the window_* classes. Might pass in the dimensions to gl_2d_display(), it kinda makes sense for it to have them

Is this better?

355ef7c

I don't know about this one chief

f6b6496

It's something like this, but...

396939a

maybe this instead -- variable canvas size for CimbWriter

a2ffe08

It's slightly more data to copy to the GPU (1040x1040px vs 1024x1024), but everything else is much more sane...

Reimplement shakycam for opencv_highgui

87338c4

I kinda want to leave rotate as not implemented? Maybe I'll add a todo

Only dark mode for now

a49fe7f

The motivation of the shaky_cam is to detect cross-frame overlap,

c15ac29

... not an interpretive barcode dance. Less snarkily, I'm trying to find the right balance between "decoder can discard bad frames" and "user gets a headache looking at the barcode".

fix color bug with highgui

6933d3e

It'd be nice to have a test to catch this 🤔

Update help

21e39db

Tired of the mouse cursor getting in the way...

0626afc

🤔

f979012

I like this interface slightly more.

4c5f826

The idea here is to improve frame pacing.

WIP: having more light (white) helps the camera

5b86c29

= cleaner frames = better transfer rates. Checking in a cool gradient, because I can't resist

Simpler gradient, same effect

a865dd4

New numbers, and they are quick

e340559

8-color probably needs higher ECC... 🤔

Spending my time on important things, like gradients

48b2d1a

Fullscreen toggle?

7247339

I think document.documentElement is what I actually want

f8408b3

Misc code cleanup

bf2c492

Add a few tests for the larger size

206c786

Misc docs updates, including new performance numbers!

8e5fa90

100 KB/s is defeated

more docs?

fc04ea6

I don't think I mentioned compression anywhere.

61fff8f

sz3 commented Mar 5, 2021

View reviewed changes

sz3 changed the title ~~Encoder improvements for the web. 118 KB/s~~ Encoder improvements. 118 KB/s Mar 5, 2021

sz3 merged commit 9ffb94e into master Mar 6, 2021

sz3 deleted the glshake branch March 6, 2021 03:12

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Encoder improvements. 118 KB/s #54

Encoder improvements. 118 KB/s #54

sz3 commented Mar 5, 2021 •

edited

Loading

sz3 Mar 5, 2021

sz3 Mar 5, 2021

sz3 Mar 5, 2021

sz3 Mar 5, 2021

sz3 Mar 5, 2021

sz3 Mar 5, 2021

sz3 Mar 5, 2021

sz3 Mar 5, 2021

sz3 Mar 5, 2021

sz3 Mar 5, 2021

sz3 Mar 5, 2021


		* other notes:
		* having better lighting in the frame often leads to better results -- this is why cimbar.org has a (mostly) white background. cfc uses android's auto-exposure, auto-focus, etc (it's a very simple app). Good ambient light -- or a white background -- can lead to more consitent quality frame capture.

Encoder improvements. 118 KB/s #54

Encoder improvements. 118 KB/s #54

Conversation

sz3 commented Mar 5, 2021 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

sz3 commented Mar 5, 2021 •

edited

Loading