feat(dbless) use LMDB as DB-less backend #8224
Merged
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
dndx
force-pushed
the
feat/lmdb
branch
4 times, most recently
from
48079be
to
4f63811
Compare
dndx
force-pushed
the
feat/lmdb
branch
2 times, most recently
from
c1a1d8d
to
38068cf
Compare
dndx
commented
Dec 23, 2021
fffonion
reviewed
Dec 23, 2021
dndx
force-pushed
the
feat/lmdb
branch
5 times, most recently
from
c038130
to
2516980
Compare
dndx
force-pushed
the
feat/lmdb
branch
2 times, most recently
from
a1e58fa
to
2516980
Compare
kikito
previously requested changes
Jan 6, 2022
bungle
reviewed
Jan 7, 2022
dndx
force-pushed
the
feat/lmdb
branch
4 times, most recently
from
cb69028
to
8d79fbb
Compare
dndx
force-pushed
the
feat/lmdb
branch
3 times, most recently
from
303af09
to
0c1187b
Compare
flrgh
reviewed
Mar 24, 2022
| -- If it takes more than 60s it is very likely to be an internal error. | ||
| -- However it will be reported as: "failed to broadcast reconfigure event: recursive". | ||
| -- Let's paste the error message here in case someday we try to search it. | ||
| -- Should we handle this case specially? |
There was a problem hiding this comment.
As somebody who has seen this specific error message on a production system, big big +1 to anything that we can do to make the error more meaningful and explanatory.
There was a problem hiding this comment.
I am not a 100% sure about the triggering condition of this, and I hesitate to make a change without understanding why the "recursive" error occurs. Will try to syncup with @suika-kong more offline on this error handling after it is merged (the current PR is getting hard to manage).
flrgh
reviewed
Mar 24, 2022
flrgh
reviewed
Mar 24, 2022
dndx
force-pushed
the
feat/lmdb
branch
3 times, most recently
from
fe85dae
to
5dccba6
Compare
aboudreault
approved these changes
Mar 30, 2022
This PR adds LMDB (Lightning Memory-Mapped Database) support for DB-less. At the same time, the shdict based DB-less storage backend has been retired and removed from the codebase. LMDB has better concurrency characters and is generally much more stable than shdict for storing config data. Because it can be natively accessed from different processes, we also removed the hack that sends the full DB-less config to the stream subsystem, which should improve the stability of DB-less reload at runtime as well. New config options `lmdb_environment_path` and `lmdb_map_size` has been added. Co-authored-by: Suika <xumin.zhou@konghq.com>
bungle
added a commit
that referenced
this pull request
Jun 14, 2022
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
This PR adds LMDB (Lightning Memory-Mapped Database) support for
DB-less. At the same time, the shdict based DB-less storage backend has
been retired and removed from the codebase.
LMDB has better concurrency characters and is generally much more stable
than shdict for storing config data. Because it can be natively accessed
from different processes, we also removed the hack that sends the full
DB-less config to the stream subsystem, which should improve the
stability of DB-less reload at runtime as well.
New config options
lmdb_environment_pathandlmdb_map_sizehas beenadded.
TODO: LMDB is fully capable of persisting the config between restarts, currently this is not enabled so it behaves like a in-memory storage (content is always wiped at startup). This will be enabled once we have more understanding of the impact.