Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Is Corrosion a solution to replicate SQLite across multiple application servers with eventual consistency? #284

Open
Sleepful opened this issue Jan 28, 2025 · 1 comment

Comments

@Sleepful
Copy link

Sleepful commented Jan 28, 2025

Is Corrosion a solution to replicate SQLite across multiple application servers with eventual consistency?

I notice LiteFS is marketed a way to have an application-level SQLite with single-writer with multiple read-replicas.

I am wondering if Corrosion could fit a similar role, however, with multiple write-replicas that converge on eventual consistency. Similar to this sqlite marmot project.

Or... whether I am misunderstanding Corrosion, assuming that it shouldn't be used for application data, and maybe meant more as a consul-alternative only.

EDIT: I suppose the one piece that Corrosion is missing to meet this use-case is the ability to communicate with a SQLite driver through a VFS or similar, like LiteFS does.

@jeromegn
Copy link
Member

Corrosion exposes a replicated, multi-writer, SQLite.

That said, it only works if your data fits. There are significant constraints:

  • No unique indexes allowed (except for the default primary key unique index that does not need to be created)
  • The primary key must be non nullable
  • Non-nullable columns require a default value

Every write needs to go through Corrosion, but reads can SQLite, from disk, normally. Corrosion is most suitable for read-heavy workloads where read performance matters a lot.

I haven't used Marmot, but looking at its internals doc, I see these differences / similarities:

  • Marmot adds TRIGGERs to capture changes to tables. Corrosion as well (via cr-sqlite)
  • Marmot watches for changes (either periodically or by some other mechanism looking at the WAL?). Corrosion intercepts changes at write time and propagates them immediately
  • Marmot uses NATS JetStream which has strong delivery guarantees via Raft. Corrosion uses various heuristics to fanout changes semi-randomly between nodes and periodically, constantly, synchronize between random nodes to make sure changes are not missed
  • Corrosion uses CRDTs to ensure strong, eventual, consistency. Marmot relies on JetStream's Raft and deterministic ordering to resolve race conditions and conflicts.

I don't think either is at an advantage or disadvantage, but does it seem easier to not have to write to Corrosion and simpler write to SQLite like you would normally do.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

No branches or pull requests

2 participants