[7.x] [SIEM][Detections] Adds large list support using REST endpoints (#62552) #64710
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
* Adds large list support using REST endpoints. Status: --- * Currently ready to be merged behind the feature flag of it being disabled with ongoing work happening after it is merged. * REST Endpoints shouldn't have large refactoring at this point * Team meeting occurred where the pieces were discussed in person. What is left? --- - [ ] Add other data types. At the moment `ip` and `keyword` are the two types of lists. See: https://www.elastic.co/guide/en/elasticsearch/reference/current/mapping-types.html - [x] Unit tests - [x] Lots of misc TODO's in the code base still - [ ] Import loads everything into memory first when it should attempt streaming - [ ] Add end to end backend tests - [x] Add transform and io-ts validation for returns Testing --- Ensure you set this in your ENV before starting Kibana: ```ts export ELASTIC_XPACK_SIEM_LISTS_FEATURE=true ``` Download or create a large list file such as this one filled with IP's: https://cinsscore.com/list/ci-badguys.txt Go to your REST endpoint folder of scripts: ```ts cd kibana/x-pack/plugins/lists/server/scripts ``` Do a hard reset: ```ts ./hard_reset ``` Then import it as either a data type of `ip`: ```ts ./import_list_items_by_filename.sh ip ~/Downloads/ci-badguys-smaller.txt ``` Or as a `keyword` ```ts ./import_list_items_by_filename.sh keyword ~/Downloads/ci-badguys-smaller.txt ``` Then you can export it through: ```ts ./export_list_items.sh ci-badgusy-smaller.txt ``` For all the other endpoints and testing of the CRUD operations you have access to: ```ts delete_all_lists.sh delete_list.sh delete_list_index.sh delete_list_item.sh delete_list_item_by_id.sh delete_list_item_by_value.sh export_list_items.sh export_list_items_to_file.sh get_list.sh get_list_item_by_id.sh get_list_item_by_value.sh import_list_items.sh import_list_items_by_filename.sh lists_index_exists.sh patch_list.sh patch_list_item.sh post_list.sh post_list_index.sh post_list_item.sh ``` Delete any items that are not applicable to this PR. - [ ] [Unit or functional tests](https://github.com/elastic/kibana/blob/master/CONTRIBUTING.md#cross-browser-compatibility) were updated or added to match the most common scenarios - [ ] This was checked for breaking API changes and was [labeled appropriately](https://github.com/elastic/kibana/blob/master/CONTRIBUTING.md#release-notes-process)
Code owners should not be backported
|
@elasticmachine merge upstream |
💚 Build SucceededTo update your PR or re-run it, just comment with: |
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Backports the following commits to 7.x: