ETS backend write-ahead log #554
Merged
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
|
Implements #538 |
zachallaun
approved these changes
Jan 17, 2024
apps/remote_control/lib/lexical/remote_control/search/store/backends/ets/wal.ex
Outdated
Show resolved
Hide resolved
apps/remote_control/lib/lexical/remote_control/search/store/backends/ets/schema.ex
Outdated
Show resolved
Hide resolved
apps/remote_control/lib/lexical/remote_control/search/store/backends/ets/wal.ex
Show resolved
Hide resolved
apps/remote_control/lib/lexical/remote_control/search/store/backends/ets/wal.ex
Show resolved
Hide resolved
apps/remote_control/lib/lexical/remote_control/search/store/backends/ets/wal.ex
Show resolved
Hide resolved
apps/remote_control/lib/lexical/remote_control/search/store/backends/ets/schema.ex
Outdated
Show resolved
Hide resolved
apps/remote_control/lib/lexical/remote_control/search/store/backends/ets/state.ex
Outdated
Show resolved
Hide resolved
apps/remote_control/lib/lexical/remote_control/search/store/backends/ets/wal.ex
Outdated
Show resolved
Hide resolved
apps/remote_control/lib/lexical/remote_control/search/store/backends/ets/wal.ex
Outdated
Show resolved
Hide resolved
apps/remote_control/test/lexical/remote_control/search/store/backends/ets/wal_test.exs
Outdated
Show resolved
Hide resolved
zachallaun
approved these changes
Jan 17, 2024
scohen
force-pushed
the
disk_log
branch
7 times, most recently
from
4a5c5fe to
f90d2d5
Compare
Closed
Our store was using :ets.tab2file to write its entries to the filesystem, which would require writing the entire index every time it was called. This is problematic, as updates to the index come in whenever code is added or removed. This would also cause a lot of contention during startup because of the amount of data being read off the filesystem. This branch introduces a write ahead log, using erlang's built in `disk_logger` module. The idea is that the WAL writes operations to disk before they're applied to ETS, so that when we start up, we first read a checkpoint, then apply the operations one-by-one to get to the prior state. Note that this approach is in contrast to using Khepri. Khepri is very nice, but it's significantly slower than this approach, and it uses a lot more memory. It's also worth noting that the WAL based approach can have its memory efficiency improved by removing the need to shuffle all of the entries around between processes, but that's future work.
We were calling start_supervised, which meant the store wasn't around to do cleanup. Instead, I added a destroy_all function which blows away all of a backend's indexes, but doesn't rely on the store or backend being up. This seems to help a bit.
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Our store was using
:ets.tab2fileto write its entries to the filesystem, which would require writing the entire index every time it was called. This is problematic, as updates to the index come in whenever code is added or removed. This would also cause a lot of contention during startup because of the amount of data being read off the filesystem.This branch introduces a write ahead log, using erlang's built in
disk_loggermodule. The idea is that the WAL writes operations to disk before they're applied to ETS, so that when we start up, we first read a checkpoint, then apply the operations one-by-one to get to the prior state.Note that this approach is in contrast to using Khepri. Khepri is very nice, but it's significantly slower than this approach, and it uses a lot more memory. It's also worth noting that the WAL based approach can have its memory efficiency improved by removing the need to shuffle all of the entries around between processes, but that's future work.