Squashed 'app/src/cpp/libcimbar/' changes from 9b90e78..b39469c

b39469c Merge pull request #56 from sz3/cimbar-send-refactor2
3e1e435 Update dependency list to single out GLES
2da86fe Find clang
257973d Slightly more readable, maybe?
0a6bda1 killing the opencv_highgui code path after all
50d65a7 Fix cimbar_send encode_id.
8216c3b Bugfix -- reset _frameCount when we reset the stream
0ca7dd1 Rename the fountain_encoder_stream's `reset()` to `restart()`
22b58ff Switch cimbar_send to use cimbar_js
6b7022f Add header for cimbar_js, and add extra params to configure()
71ab9ea Moving cimbar_js into src/lib/
4415602 Merge pull request #55 from sz3/decode-more
918ffb2 Update build instructions to match reality
6376e12 indentation
fc5ef1c Merge remote-tracking branch 'origin/master' into decode-more
79a91b2 Smaller image
9ffb94e Merge pull request #54 from sz3/glshake
61fff8f I don't think I mentioned compression anywhere.
fc04ea6 more docs?
8e5fa90 Misc docs updates, including new performance numbers!
206c786 Add a few tests for the larger size
bf2c492 Misc code cleanup
f8408b3 I think document.documentElement is what I actually want
7247339 Fullscreen toggle?
48b2d1a Spending my time on important things, like gradients
e340559 New numbers, and they are quick
a865dd4 Simpler gradient, same effect
5b86c29 WIP: having more light (white) helps the camera
4c5f826 I like this interface slightly more.
f979012 🤔
0626afc Tired of the mouse cursor getting in the way...
21e39db Update help
6933d3e fix color bug with highgui
c15ac29 The motivation of the shaky_cam is to detect cross-frame overlap,
7be3d91 Do the preprocessing sharpen after the grayscale
a49fe7f Only dark mode for now
87338c4 Reimplement shakycam for opencv_highgui
a2ffe08 maybe this instead -- variable canvas size for CimbWriter
396939a It's something like this, but...
f6b6496 I don't know about this one chief
355ef7c Is this better?
d869656 use loop_iterator?
cb84a59 hmm
0de861b quick WIP for GLSL transpose for the shakycam effect

git-subtree-dir: app/src/cpp/libcimbar
git-subtree-split: b39469c

.github/workflows/ci.yml

@@ -13,27 +13,26 @@ jobs:
  - name: "linux gcc7"
  os: ubuntu-18.04
  env: CXX="g++-7" CC="gcc-7"
- cmake-flags: "-DUSE_GLFW=1"
- extra-packages: "libglfw3-dev libgles2-mesa-dev"
 
  - name: "linux clang"
  os: ubuntu-18.04
- env: CXX="clang++-6.0" CC="clang-6.0"
+ env: CXX="clang++" CC="clang"
+ extra-packages: "clang"
 
  steps:
  - uses: actions/checkout@v2
  with:
  submodules: 'recursive'
 
  - name: Install dependencies
- run: sudo apt-get update && sudo apt-get install libopencv-dev ${{ matrix.config.extra-packages }}
+ run: sudo apt-get update && sudo apt-get install libopencv-dev libglfw3-dev libgles2-mesa-dev ${{ matrix.config.extra-packages }}
 
  - name: Create Build Environment
  run: cmake -E make_directory ${{runner.workspace}}/build
 
  - name: Configure CMake
  working-directory: ${{runner.workspace}}/build
- run: ${{ matrix.config.env }} cmake $GITHUB_WORKSPACE ${{ matrix.config.cmake-flags }}
+ run: ${{ matrix.config.env }} cmake $GITHUB_WORKSPACE
 
  - name: Build
  working-directory: ${{runner.workspace}}/build

CMakeLists.txt

@@ -38,17 +38,11 @@ if(NOT DEFINED CPPFILESYSTEM)
  set(CPPFILESYSTEM "stdc++fs")
 endif()
 
-if(DEFINED USE_GLFW)
- set(libcimbar_UI_LIB "glfw" "GL")
- add_definitions("-DLIBCIMBAR_USE_GLFW=${USE_GLFW}")
-else()
- set(libcimbar_UI_LIB "opencv_highgui")
-endif()
-
 set (PROJECTS
  src/lib/bit_file
  src/lib/chromatic_adaptation
  src/lib/cimb_translator
+ src/lib/cimbar_js
  src/lib/compression
  src/lib/encoder
  src/lib/fountain
@@ -78,13 +72,6 @@ set(PROJECTS
 )
 endif()
 
-if (DEFINED USE_WASM OR USE_GLFW)
-set(PROJECTS
- ${PROJECTS}
- src/wasm/cimbar_js
-)
-endif()
-
 include_directories(
  ${libcimbar_SOURCE_DIR}/src/lib
  ${libcimbar_SOURCE_DIR}/src/third_party_lib

DETAILS.md

@@ -3,7 +3,7 @@
 
 ## The premise
 
-Conceptually, cimbar is built on top of `image hashing`:
+Cimbar is a grid of colored tiles. Conceptually, it is built on the idea of `image hashing`:
 
 ![example image hash](https://github.com/sz3/cimbar-samples/blob/v0.5/docs/imagehash.png)
 
@@ -84,7 +84,7 @@ These properties may appear to be magical as you consider them more, and they do
 2. wirehair requires the file contents to be stored in RAM
  * this relates to the size limit!
 
-This constraint is less of an obstacle than it may seem -- the fountain codes are essentially being used as a wire format, and the encoder and decoder could agree on a scheme to split up, and then reassemble, larger files. Cimbar does not yet implement this, however!
+This constraint is less of an obstacle than it may seem -- the fountain codes are essentially being used as a wire format, and the encoder and decoder could agree on a scheme to split up, and then reassemble, larger files. Cimbar does not (yet?) implement this, however!
 
 ## Implementation: Decoder
 

PERFORMANCE.md

@@ -9,22 +9,32 @@
  * There are 4 or 8 possible colors, encoding an additional 2-3 bits per tile.
  * These 6-7 bits per tile work out to a maximum of 9300-10850 bytes per barcode, though in practice this number is reduced by error correction.
  * The default ecc setting is 30/155, which is how we go from 9300 -> 7500 bytes of real data for a 4-color cimbar image.
-  * Reed Solomon is not an ideal for this use case -- specifically, it corrects byte errors, and cimbar errors tend to involve 1-3 bits at a time. However, since Reed Solomon implementations are ubiquitous, I used it for this prototype.
+  * Reed Solomon is not perfect for this use case -- specifically, it corrects byte errors, and cimbar errors tend to involve 1-3 bits at a time. However, since Reed Solomon implementations are ubiquitous, it is currently in use.
 
 ## Current sustained benchmark
 
 * 4-color cimbar with ecc=30:
- * 2,980,556 bytes (after compression) in 36s -> 662 kilobits/s (~82 KB/s)
+ * 4,717,525 bytes (after compression) in 45s -> 838 kilobits/s (~104 KB/s)
 
 * 8-color cimbar with ecc=30:
- * 2,980,556 bytes in 31s -> 769 kilobits/s (~96 KB/s)
+ * 4,717,525 bytes in 40s -> 943 kilobits/s (~118 KB/s)
 
 * details:
- * these numbers are use https://github.com/sz3/cfc, running with 4 CPU threads on a Qualcomm Snapdragon 625
+ * cimbar has built-in compression using zstd. What's being measured here is bits over the wire, e.g. data after compression is applied.
+ * these numbers are using https://github.com/sz3/cfc, running with 4 CPU threads on a Qualcomm Snapdragon 625
   * perhaps I will buy a new cell phone to inflate the benchmark numbers.
- * the sender commandline is `./cimbar_send /path/to/file -s`
-  * the `shakycam` option allows cfc to quickly discard ghosted frames, and spend more time decoding real data.
+ * the sender is the cimbar.org wasm implementation. An equivalent command line is `./cimbar_send /path/to/file -s`
+  * cimbar.org uses the `shakycam` option to allow the receiver to detect/discard "in between" frames as part of the scan step. This allows it to spend more processing time decoding real data.
  * burst rate can be higher (or lower)
   * to this end, lower ecc settings *can* provide better burst rates
- * 8-color cimbar is considerably more sensitive to lighting conditions. Notably, decoding has some issues with dim screens.
+ * 4-color cimbar is currently preferred, and will give more consistent transfer speeds.
+ * 8-color cimbar should be considered a prototype within a prototype. It is considerably more sensitive to lighting conditions and color tints.
 
+* other notes:
+ * having better lighting in the frame often leads to better results -- this is why cimbar.org has a (mostly) white background. cfc uses android's auto-exposure, auto-focus, etc (it's a very simple app). Good ambient light -- or a white background -- can lead to more consitent quality frame capture.
+ * because of the lighting/exposure question, landscape *may* be better than portrait.
+ * cfc currently has a low resolution, so the cimbar frame should take up as much of the display as possible (trust the guide brackets)
+ * similarly, it's best to keep the camera angle straight-on -- instead of at an angle -- to decode the whole image successfully. Decodes should still happen at higher angles, but the "smaller" part of the image may have more errors than the ECC can deal with.
+ * other things to be wary of:
+  * glare from light sources.
+  * shaky hands.

README.md

@@ -5,7 +5,11 @@
 
 Behold: an experimental barcode format for air-gapped data transfer.
 
-It can sustain speeds of 770+ kilobits/s (~96 KB/s) using nothing but a smartphone camera!
+It can sustain speeds of 943+ kilobits/s (~118 KB/s) using just a computer monitor and a smartphone camera!
+
+<p align="center">
+<img src="https://github.com/sz3/cimbar-samples/blob/v0.5/6bit/4cecc30f.png" width="70%" title="A non-animated cimbar code" >
+</p>
 
 ## Explain?
 
@@ -21,17 +25,21 @@ No internet/bluetooth/NFC/etc is used. All data is transmitted through the camer
 
 `cimbar` is a high-density 2D barcode format. Data is stored in a grid of colored tiles -- bits are encoded based on which tile is chosen, and which color is chosen to draw the tile. Reed Solomon error correction is applied on the data, to account for the lossy nature of the video -> digital decoding. Sub-1% error rates are expected, and corrected.
 
-`libcimbar`, this optimized implementation, includes a simple protocol for file encoding based on fountain codes (`wirehair`). Files of up to 33MB can be encoded in a series of cimbar codes, which can be output as images or a live video feed. Once enough distinct image frames have been decoded successfully, the file will be reconstructed successfully. This is true even if the images are received out of order, or if some have been corrupted or are missing.
+`libcimbar`, this optimized implementation, includes a simple protocol for file encoding built on fountain codes (`wirehair`) and zstd compression. Files of up to 33MB (after compression!) are encoded in a series of cimbar codes, which can be output as images or a live video feed. Once enough distinct image frames have been decoded successfully, the file will be reconstructed and decompressed successfully. This is true even if the images are received out of order, or if some have been corrupted or are missing.
 
 ## Platforms
 
-The code is written in C++, and developed/tested on amd64+linux, arm64+android, and emscripten+wasm. It probably works, or can be made to work, on other platforms.
+The code is written in C++, and developed/tested on amd64+linux, arm64+android (decoder only), and emscripten+WASM (encoder only). It probably works, or can be made to work, on other platforms.
+
+Crucially, because the encoder compiles to asmjs and wasm, it can run on anything with a modern web browser. There are [releases](https://github.com/sz3/libcimbar/releases/latest) if you wish to run the encoder locally instead of via cimbar.org.
 
 ## Library dependencies
 
-[OpenCV](https://opencv.org/) must be installed before building. All other dependencies are included in the source tree.
+[OpenCV](https://opencv.org/) and [GLFW](https://github.com/glfw/glfw) (+ OpenGL ES headers) must be installed before building. All other dependencies are included in the source tree.
 
-* opencv - https://opencv.org/
+* opencv - https://opencv.org/ (`libopencv-dev`)
+* GLFW - https://github.com/glfw/glfw (`libglfw3-dev`)
+* GLES3/gl3.h - `libgles2-mesa-dev`
 * base - https://github.com/r-lyeh-archived/base
 * catch2 - https://github.com/catchorg/Catch2
 * concurrentqueue - https://github.com/cameron314/concurrentqueue
@@ -45,12 +53,14 @@ The code is written in C++, and developed/tested on amd64+linux, arm64+android,
 * wirehair - https://github.com/catid/wirehair
 * zstd - https://github.com/facebook/zstd
 
-Optional:
-* GLFW - https://github.com/glfw/glfw or `libglfw3-dev` (for when `opencv-highgui` is not available or unwanted)
- * the GLFW code path also needs/uses `GLES3/gl3.h` (`libgles2-mesa-dev` on ubuntu 18.04)
-
 ## Build
 
+1. install opencv and GLFW. On ubuntu/debian, this looks like:
+```
+sudo apt install libopencv-dev libglfw3-dev libgles2-mesa-dev
+```
+
+2. run the cmake + make incantation
 ```
 cmake .
 make -j7
@@ -59,7 +69,7 @@ make install
 
 By default, libcimbar will try to install build products under `./dist/bin/`.
 
-To build the emscripten+WASM encoder (what cimbar.org uses), see [WASM](WASM.md).
+To build cimbar.js (what cimbar.org uses), see [WASM](WASM.md).
 
 ## Usage
 
@@ -80,6 +90,8 @@ Encode and animate to window:
 ./cimbar_send inputfile.pdf
 ```
 
+You can also encode a file using [cimbar.org](https://cimbar.org), or the latest [release](https://github.com/sz3/libcimbar/releases/latest).
+
 ## Performance numbers
 
 [PERFORMANCE](PERFORMANCE.md)

TODO.md

@@ -8,15 +8,15 @@ libcimbar is fairly optimized, to achieve the *proof* part of proof-of-concept.
 Performance optimizations aside, there are a number of paths that might be interesting to pursue. Some I may take a look at, but most I will leave to any enterprising developer who wants to take up the cause:
 
 * proper metadata/header information?
- * would be nice to be able to determine ecc/#colors/#smybols from the cimbar image itself?
+ * would be nice to be able to determine ecc/#colors/#symbols from the cimbar image itself?
  * The bottom right corner is the obvious place to reclaim space to make this possible.
 * multi-frame decoding?
  * when decoding a static cimbar image, it would be useful to be able to use prior (unsuccessful) decode attempts to inform a future decode, and -- hopefully -- increase the probability of success. Currently, all frames are decoded independently.
   * there is already a granular confidence metric that could be reused -- the `distance` that's tracked when decoding symbol tiles...
 * optimal symbol set?
  * the 16-symbol (4 bit) set is hand-drawn. I stared with ~40 or so hand-drawn symbols, and used the 16 that performed best with each other.
  * there is surely a more optimal set -- a more rigorous approach should yield lower error rates!
- * but, more importantly, it may be possible to go up to 32 symbols, and encode 5 bits per tile?
+ * but, more importantly, it may be possible to go up to 32 symbols, and encode 5 symbol bits per tile?
 * optimal symbol size?
  * the symbols that make up each cell on the cimbar grid are 8x8 (in a 9x9 grid).
  * this is because imagehash was on 8x8 tiles!
@@ -25,16 +25,24 @@ Performance optimizations aside, there are a number of paths that might be inter
 * optimal color set?
  * the 4-color (2 bit) pallettes seem reasonable. 8-color, perhaps less so?
  * this may be a limitation of the algorithm/approach, however. Notably, since each symbol is drawn with one pallette color, all colors need sufficient contrast against the backdrop (#000 or #FFF, depending). This constrains the color space somewhat, and less distinct colors == more errors.
+ * in addition to contrast, there is interplay (that I don't currently understand) between the overall brightness of the image and the exposure time needed for high framerate capture. More clean frames == more troughput.
 * optimal grid size?
  * 1024x1024 is a remnant of the early prototyping process. There is nothing inherently special about it (except that it fits on a 1920x1080 screen, which seems good)
   * the tile grid itself is 1008x1008 (1008 == 9x112 -- there are 112 tile rows and columns)
  * a smaller grid would be less information dense, but more resilient to errors. Probably.
 * optimal grid shape?
  * it's a square because QR codes are square. That's it. Should it be?
+ * I'm strongly considering 4:3 for the next revision.
 * more efficient ECC?
- * LDPC?
- * Reed Solomon operates on bytes. Most decode errors tend to average out at 1-3 bits. It's not a total disaster, because it works. However, it would be nice to have denser error correction codes.
+ * QC-LDPC?
+ * Reed Solomon operates on bytes. Most decode errors tend to average out at 1-3 bits. (In the pathological case, a single read error will span two bytes.) It's not a total disaster -- it still works. 
+ * I expect that state of the art ECC will allow 6-15% better throughput.
+  * it's a wide range due to various unknowns (unknowns to me, anyway)
 * proper GPU support (OpenCV + openCL) on android?
+ * It *might* be useful. [CFC]((https://github.com/sz3/cfc) is the current test bed for this.
+* wasm decoder?
+ * probably needs to use Web Workers
+ * in-browser GPGPU support would be interesting (but I'm not counting on it)
 * ???
  * still reading? Of course there's more! There's always more!
 

WASM.md

@@ -22,7 +22,7 @@ mkdir build-wasm
 cd build-wasm
 source /path/to/emscripten/emsdk/emsdk_env.sh
 emcmake cmake .. -DUSE_WASM=1 -DOPENCV_DIR=/path/to/opencv
-emcmake make -j7 install
+make -j7 install
 ```
 
 (do `-DUSE_WASM=2` to use asm.js instead of wasm)

src/exe/cimbar/cimbar.cpp

@@ -74,7 +74,7 @@ int main(int argc, char** argv)
  unsigned ecc = cimbar::Config::ecc_bytes();
  options.add_options()
  ("i,in", "Encoded pngs/jpgs/etc (for decode), or file to encode", cxxopts::value<vector<string>>())
- ("o,out", "Output file or directory.", cxxopts::value<string>())
+ ("o,out", "Output file prefix (encoding) or directory (decoding).", cxxopts::value<string>())
  ("c,color-bits", "Color bits. [0-3]", cxxopts::value<int>()->default_value(turbo::str::str(colorBits)))
  ("e,ecc", "ECC level", cxxopts::value<unsigned>()->default_value(turbo::str::str(ecc)))
  ("f,fountain", "Attempt fountain encode/decode", cxxopts::value<bool>())

src/exe/cimbar_send/CMakeLists.txt

@@ -13,13 +13,7 @@ add_executable (
 
 target_link_libraries(cimbar_send
 
- cimb_translator
-
- correct_static
- wirehair
- zstd
- ${OPENCV_LIBS}
- ${libcimbar_UI_LIB}
+ cimbar_js
 )
 
 add_custom_command(