Add support for AVB FEC

This commit adds initial support for the FEC (forward error correction)
used by AVB hashtree descriptors. AVB FEC uses standard Reed-Solomon in
the RS(255, 253) + GF(256) + 0x11d configuration, but does not read data
sequentially. Instead, it uses an interleaving pattern where each RS
block covers bytes that span nearly the entire file.

Due to the interleaving file access pattern, we're forced to use mmap or
read the entire file into memory. Using seek = read when dealing with a
large input file results in a ~500x increase in processing time. Reading
one byte at a time non-sequentially is about the worst possible access
pattern for seek + read.

All of the processing is done multithreaded. Although the AOSP libfec
implementation is also multithreaded, this implementation runs at ~5x
the speed of AOSP libfec. Unfortunately, while the implementation is
extremely fast, it required using `unsafe` blocks. Multiple threads need
to write to disjoint, non-contiguous indices in a buffer, but there's no
way to prove to the compiler that this is safe, so we're stuck with
using raw pointers.

Issue: #145

Signed-off-by: Andrew Gunnerson <accounts+github@chiller3.com>

README.extra.md

@@ -58,6 +58,51 @@ avbroot boot info -i <input boot image>
 
 All of the `boot` subcommands show the boot image information. This specific subcommand just does it without performing any other operation. To show avbroot's internal representation of the information, pass in `-d`.
 
+## `avbroot fec`
+
+This set of commands is for working with the FEC (forward error correction) data used by AVB hashtree descriptors. These are primarily used for correcting errors (eg. bit flips) in dm-verity partition images. Although the FEC data is primarily used in AVB, these `avbroot fec` commands operate on FEC data stored in AOSP's standalone format, not on AVB descriptors. The actual FEC data is identical in both formats, only the headers are different. These standalone FEC files are compatible with AOSP's `fec` tool.
+
+AOSP's FEC data is unique in that it's not generated from a sequential read of the file. Instead, it uses an interleaved access pattern. If an input file is visualized as a 2D table:
+
+```
+| 0    1    2    3    ... 4095 |
+| 4096 4097 4098 4099 ... 8191 |
+| 8192 8193 8194 8195 ... 8196 |
+| .... .... .... .... ... .... |
+```
+
+then the file access pattern can be thought of as being column-by-column instead of row-by-row.
+
+Data correction happens at the codeword level. A Reed-Solomon codeword is 255 bytes where some portion is file data and the rest is parity data. AOSP and avbroot both default to 253 bytes of data and 2 bytes of parity information. Each column in the table represents the 253-byte data portion of the codeword. A contiguous sequence of corrupted data will span multiple columns and since error correction happens at the column level, this interleaved design increases the chances of recovery.
+
+However, this columnar file access pattern looks very much like random reads to the operating system. When going down each column, 1 byte is being read at a time with gaps in the middle. The data is never read in sequential chunks. This would normally cause a massive performance hit. To work around this, `avbroot fec` commands use `mmap` to map the file contents into memory. If this isn't supported (eg. some virtual machine shared folders), avbroot will have to read the entire file contents into memory.
+
+### Generating FEC data
+
+```bash
+avbroot fec generate -i <input data file> -f <output FEC file>
+```
+
+The default behavior is to use 2 bytes of parity information per 253 bytes of input data. Within each 253-byte column described above, this is sufficient for correcting a single corrupted byte in the column (`parity / 2` bytes in general).
+
+avbroot currently only supports 2 bytes of parity (which is the default for FEC data generated by AOSP tools).
+
+### Verifying a file
+
+```bash
+avbroot fec verify -i <input data file> -f <input FEC file>
+```
+
+This will check if the input file has any corrupted bytes. This command runs significantly faster than `avbroot fec repair` and is useful if only detection of corrupted data is needed.
+
+### Repairing a file
+
+```bash
+avbroot fec repair -i <input/output data file> -f <input FEC file>
+```
+
+This will repair the file in place. As described above, in each column, up to `parity / 2` bytes can be corrected.
+
 ## `avbroot ramdisk`
 
 ### Dumping a cpio archive

avbroot/Cargo.toml

@@ -18,9 +18,11 @@ cms = { version = "0.2.2", features = ["std"] }
 const-oid = "0.9.5"
 ctrlc = "3.4.0"
 flate2 = "1.0.27"
+gf256 = { version = "0.3.0", features = ["rs"] }
 hex = "0.4.3"
 lz4_flex = "0.11.1"
 memchr = "2.6.0"
+memmap2 = "0.7.1"
 num-bigint-dig = "0.8.4"
 num-traits = "0.2.16"
 phf = { version = "0.11.2", features = ["macros"] }

avbroot/src/cli/args.rs

@@ -8,14 +8,15 @@ use std::sync::{atomic::AtomicBool, Arc};
 use anyhow::Result;
 use clap::{Parser, Subcommand};
 
-use crate::cli::{avb, boot, completion, key, ota, ramdisk};
+use crate::cli::{avb, boot, completion, fec, key, ota, ramdisk};
 
 #[allow(clippy::large_enum_variant)]
 #[derive(Debug, Subcommand)]
 pub enum Command {
     Avb(avb::AvbCli),
     Boot(boot::BootCli),
     Completion(completion::CompletionCli),
+    Fec(fec::FecCli),
     Key(key::KeyCli),
     Ota(ota::OtaCli),
     Ramdisk(ramdisk::RamdiskCli),
@@ -41,6 +42,7 @@ pub fn main(cancel_signal: &Arc<AtomicBool>) -> Result<()> {
         Command::Avb(c) => avb::avb_main(&c, cancel_signal),
         Command::Boot(c) => boot::boot_main(&c),
         Command::Completion(c) => completion::completion_main(&c),
+        Command::Fec(c) => fec::fec_main(&c, cancel_signal),
         Command::Key(c) => key::key_main(&c),
         Command::Ota(c) => ota::ota_main(&c, cancel_signal),
         Command::Ramdisk(c) => ramdisk::ramdisk_main(&c),