Introduce skeleton of memlog store #19261
Merged
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This changes introduces the skeleton and documentation of the memlog store. The addition of the statestore package is split up into multiple changeset to ease review. The final version of the package can be found [here](https://github.com/urso/beats/tree/fb-input-v2-combined/libbeat/statestore). Once finalized, the libbeat/statestore package contains: - The statestore frontend and interface for use within Beats - Interfaces for the store backend - A common set of tests store backends need to support - a storetest package for testing new features that require a store. The testing helpers use map[string]interface{} that can be initialized or queried after the test run for validation purposes. - The default memlog backend + tests This change introduces the skeleton and internal documentation of the memlog store for review and discussion, but not the full implementation yet to ease review. The final implementation and unit tests will be added later. The file doc.go documents how the final implementation works.
urso
added
review
Filebeat
|
Pinging @elastic/integrations-services (Team:Services) |
botelastic
bot
added
needs_team
💔 Tests Failed
Expand to view the summary
Build stats
Test stats 🧪
Test errorsExpand to view the tests failures
--------------------- >> end captured stdout << ---------------------- Steps errorsExpand to view the steps failures
Log outputExpand to view the last 100 lines of log output
|
andresrc
added
[zube]: In Review
Team:Platforms
|
Pinging @elastic/integrations-platforms (Team:Platforms) |
andrewkroh
reviewed
Jun 23, 2020
Co-authored-by: Andrew Kroh <andrew.kroh@elastic.co>
faec
approved these changes
Jun 25, 2020
| // specific language governing permissions and limitations | ||
| // under the License. | ||
|
|
||
| // Package memlog implements the memlog statestore backend. |
There was a problem hiding this comment.
Would it make sense to put this content in a README.md instead? That would have the additional benefit that it would be automatically rendered when browsing this package directory on github, which helps discoverability.
There was a problem hiding this comment.
Hm, interesting point. I have put it here, because I use go docs :)
| // The data files filenames do include the change sequence number. Which allows | ||
| // us to sort them by name. The checkpoint operation of memlog, writes the full state | ||
| // into a new data file, that consists of an JSON array with all known key-value pairs. | ||
| // All fields of an entry are store top-level in the object. The `_key` field |
There was a problem hiding this comment.
I'm not sure what this sentence ("All fields...") means: it sounds like it's saying the object has a flat structure, but the description above says that map[string]interface{} can be included, so I think I'm misunderstanding what "entry" is referring to here, could that be clarified?
There was a problem hiding this comment.
Will try. It's indeed an object that supports nesting (map[string]interface{})
kvch
approved these changes
Jun 25, 2020
urso
pushed a commit
to urso/beats
that referenced
this pull request
Jul 6, 2020
Introduce skeleton of memlog store This changes introduces the skeleton and documentation of the memlog store. The addition of the statestore package is split up into multiple changeset to ease review. The final version of the package can be found [here](https://github.com/urso/beats/tree/fb-input-v2-combined/libbeat/statestore). Once finalized, the libbeat/statestore package contains: - The statestore frontend and interface for use within Beats - Interfaces for the store backend - A common set of tests store backends need to support - a storetest package for testing new features that require a store. The testing helpers use map[string]interface{} that can be initialized or queried after the test run for validation purposes. - The default memlog backend + tests This change introduces the skeleton and internal documentation of the memlog store for review and discussion, but not the full implementation yet to ease review. The final implementation and unit tests will be added later. The file doc.go documents how the final implementation works. (cherry picked from commit c0f1b75)
2 tasks
urso
pushed a commit
that referenced
this pull request
Jul 7, 2020
Introduce skeleton of memlog store This changes introduces the skeleton and documentation of the memlog store. The addition of the statestore package is split up into multiple changeset to ease review. The final version of the package can be found [here](https://github.com/urso/beats/tree/fb-input-v2-combined/libbeat/statestore). Once finalized, the libbeat/statestore package contains: - The statestore frontend and interface for use within Beats - Interfaces for the store backend - A common set of tests store backends need to support - a storetest package for testing new features that require a store. The testing helpers use map[string]interface{} that can be initialized or queried after the test run for validation purposes. - The default memlog backend + tests This change introduces the skeleton and internal documentation of the memlog store for review and discussion, but not the full implementation yet to ease review. The final implementation and unit tests will be added later. The file doc.go documents how the final implementation works. (cherry picked from commit c0f1b75)
melchiormoulin
pushed a commit
to melchiormoulin/beats
that referenced
this pull request
Oct 14, 2020
Introduce skeleton of memlog store This changes introduces the skeleton and documentation of the memlog store. The addition of the statestore package is split up into multiple changeset to ease review. The final version of the package can be found [here](https://github.com/urso/beats/tree/fb-input-v2-combined/libbeat/statestore). Once finalized, the libbeat/statestore package contains: - The statestore frontend and interface for use within Beats - Interfaces for the store backend - A common set of tests store backends need to support - a storetest package for testing new features that require a store. The testing helpers use map[string]interface{} that can be initialized or queried after the test run for validation purposes. - The default memlog backend + tests This change introduces the skeleton and internal documentation of the memlog store for review and discussion, but not the full implementation yet to ease review. The final implementation and unit tests will be added later. The file doc.go documents how the final implementation works.
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
What does this PR do?
This changes introduces the skeleton and documentation of the memlog
store.
The addition of the statestore package is split up into multiple
changeset to ease review. The final version of the package can be found
here.
Once finalized, the libbeat/statestore package contains:
testing helpers use map[string]interface{} that can be initialized or
queried after the test run for validation purposes.
This change introduces the skeleton and internal documentation of the
memlog store for review and discussion, but not the full implementation
yet to ease review. The final implementation and unit tests will be
added later.
The file doc.go documents how the final implementation works. When reviewing please have a look at this file first to get an idea how the structures fit together.
Why is it important?
The statestore introduces a simple key-value store to Beats. The statestore will be used to replace the registry in filebeat in the future.
Checklist
- [ ] I have made corresponding changes to the documentation- [ ] I have made corresponding change to the default configuration files- [ ] I have added tests that prove my fix is effective or that my feature works- [ ] I have added an entry inCHANGELOG.next.asciidocorCHANGELOG-developer.next.asciidoc.Author's Checklist
Related issues