Collabs Many-User Rich-Text Editing Benchmarks

Many-user rich-text editing benchmarks for Collabs, Yjs, Automerge, ShareDB, and Google Docs. As described in the Collabs paper's Section 7.1.

Usage

Dependencies:

• Linux
• Node v18+

Install:

npm ci --legacy-peer-deps

Build:

npm run build

(This just runs npm run build in the server/ folder.)

See README.md in each folder for specific usage instructions.

Organization

• server/: Web server that serves each framework's rich-text editor and orchestrates the experiment.
• client/: Node.js program that uses Puppeteer to run a client, i.e., open the server's page in Chromium and simulate user activity. Also contains script to run a full experiment locally.
• analysis/: Analyzes the output of clients and the server to generates metrics for the paper.
• aws/: Helpers in case you choose to run the experiment clients on AWS ECS.
• metadata_overhead/: Microbenchmarks to measure metadata overhead, as described in the paper's Section 7.2.

Data

• analysis/charts/ contains LibreOffice Calc spreadsheets with the summary data used in the paper.
• Analysis script output (including CPU profiles and saved states - needed to run metadata_overhead benchmarks) and raw data can be downloaded separately from https://cmu.box.com/s/ina8bc35c2wyu22q0tydfyaeb2c4744c.

Acknowledgments

Extensively modified from the OWebSync paper's eDesigners benchmark's code, provided by Kristof Jannes.

Simulated user activity is based on Martin Kleppmann's automerge-perf typing trace.