[esArchiver] combine elasticDump and ScenarioManager #10359
Merged
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
spalger
force-pushed
the
implement/es-archiver
branch
from
e429ff1
to
2017ff8
Compare
spalger
force-pushed
the
implement/es-archiver
branch
2 times, most recently
from
197e4cd
to
09fe2c8
Compare
spalger
force-pushed
the
implement/es-archiver
branch
from
09fe2c8
to
0d869b9
Compare
… these plugins a way to capture and reload es indexes without needing to write a bunch of custom code. It works similarly to the elasticDump and ScenarioManager tools that it replaces. Differences: - Streaming implementation allows for much larger archives - CLI for creating and using archives - Configurable archive location - Stores the data in gzipped files (better for source control, searching, large archives) - Automatically identifies and upgrades Kibana config documents Methods: - `#load(name)`: import an archive - `#loadIfNeeded(name)`: import an archive, but skip the documents what belong to any existing index - `#unload(name)`: delete the indexes stored in an archive CLI operations: - `./bin/es_archiver save <name> [index patterns...]`: save the mapping and documents in one or more indexes that match the wild-card patterns into an the `<name>` archive - `./bin/es_archiver load <name>`: load the mapping and documents from the `<name>` archive
spalger
force-pushed
the
implement/es-archiver
branch
from
0d869b9
to
70fc947
Compare
|
To review this I think we need to zoom. I'm mostly busy tomorrow, but maybe the day after? Just sync up so you can show me the important stuff, then I can take a deeper look. |
spalger
force-pushed
the
implement/es-archiver
branch
from
52ecc50
to
3b6676c
Compare
|
jenkins, test this |
- remove auto-upgrading config doc logic (until we have better access to kibana version info) - export unload command - remove preemptive checks in favor of reacting to errors - use type "doc" vs "hit" for doc records (consistency) - wrote a bunch of pending tests to think though and plan
|
@epixa @kjbekkelund Tests are passing and I'm working on writing more tests for the added functionality but it's ready to look at I think |
spalger
force-pushed
the
implement/es-archiver
branch
from
7561500
to
ca6ddc3
Compare
spalger
force-pushed
the
implement/es-archiver
branch
from
ca6ddc3
to
79add82
Compare
spalger
force-pushed
the
implement/es-archiver
branch
from
79add82
to
edfe4b3
Compare
|
@epixa @kjbekkelund pushed updates to address feedback if you don't mind taking a look. |
|
The tests for esArchiver are failing on jenkins |
|
I don't immediately see any big problems with the changes, so LGTM. (CI is struggling, though) |
|
@epixa the CI gods have blessed this build, hurry, look now! |
epixa
approved these changes
Mar 25, 2017
|
Before merging, please remove both load_dump_data files (the script wrapper and the actual file), and please update the PR description to accurately reflect the changes to the implementation from the PR review. |
|
I'm going to merge this because it'll unblock CI builds that are currently failing due to elasticdump (which this removes entirely). |
simianhacker
pushed a commit
to simianhacker/kibana
that referenced
this pull request
Mar 27, 2017
* As a part of bringing functional testing to plugins, esArchiver gives these plugins a way to capture and reload es indexes without needing to write a bunch of custom code. It works similarly to the elasticDump and ScenarioManager tools that it replaces. Differences: - Streaming implementation allows for much larger archives - CLI for creating and using archives - Configurable archive location - Stores the data in gzipped files (better for source control, searching, large archives) - Automatically identifies and upgrades Kibana config documents Methods: - `#load(name)`: import an archive - `#loadIfNeeded(name)`: import an archive, but skip the documents what belong to any existing index - `#unload(name)`: delete the indexes stored in an archive CLI operations: - `./bin/es_archiver save <name> [index patterns...]`: save the mapping and documents in one or more indexes that match the wild-card patterns into an the `<name>` archive - `./bin/es_archiver load <name>`: load the mapping and documents from the `<name>` archive * [functional_tests/common/nagivate] check for statusPage * [es_archiver] move bins into new scripts dir * [functional_tests/apps/context] use esArchiver * [esArchiver] general improvements after showing to a few folks - remove auto-upgrading config doc logic (until we have better access to kibana version info) - export unload command - remove preemptive checks in favor of reacting to errors - use type "doc" vs "hit" for doc records (consistency) - wrote a bunch of pending tests to think though and plan * [esArchiver] make log a stream that writes to itself * [esArchiver] fill in stats and archive format tests * [esArchiver] splitup action logic * [esArchiver/cli] fix cli --help output and comment * [esArchiver] remove type-based param coercion * [esArchiver/log] use strings for log levels * [esArchvier] remove unused var * [esArchiver/indexDocRecordsStream] add tests * [esArchive] fill in remaining tests * [esArchiver] fix dem tests * [eslint] remove unused vars * [esArchiver/loadIfNeeded] fix call to load() * [esArchiver] remove loadDumpData helpers
simianhacker
added a commit
that referenced
this pull request
Mar 28, 2017
…#10890) * Removing derivative behavior... adding better tooltips to add delete * Re-organizing metrics; Adding headers; Adding default behavoir * Adding help text * Additional fixes - Fix for deactivated still being able to be added - Added Experimental Feature to description * [elasticsearch/healthCheck] ensure that multi.allow_explicit_index=true (#10855) * [elasticsearch/healthCheck] ensure that multi.allow_explicit_index=true * [elasticsearch/healthCheck] fix tests * [eslint/console] enable no-undef rule (#10881) * [eslint/console] enable no-undef rule * [eslint/console] fix no-undef violations * Navbarextensions improvements (#9871) * - added controller option - added $location service to determine on which the user is at currently * added test for the hideButton method that is dependent on the location path * removed argument * use locals object to pass variables to the template's scope * use map object * fixing percentage mode extents (#10843) * fixes axis title for new axis (#10866) * fixes metrics options matching (#10865) * Add kuiLocalTab-isDisabled state. (#10830) * Add kuiLocalTab-isDisabled state. * Update Management header to use kuiLocalTab-isDisabled class. * test: send content-type with proxy POST tests (#10903) Elasticsearch now requires that content-type be sent on all requests with payloads, so our tests should be sending it with requests as well. * [esArchiver] combine elasticDump and ScenarioManager (#10359) * As a part of bringing functional testing to plugins, esArchiver gives these plugins a way to capture and reload es indexes without needing to write a bunch of custom code. It works similarly to the elasticDump and ScenarioManager tools that it replaces. Differences: - Streaming implementation allows for much larger archives - CLI for creating and using archives - Configurable archive location - Stores the data in gzipped files (better for source control, searching, large archives) - Automatically identifies and upgrades Kibana config documents Methods: - `#load(name)`: import an archive - `#loadIfNeeded(name)`: import an archive, but skip the documents what belong to any existing index - `#unload(name)`: delete the indexes stored in an archive CLI operations: - `./bin/es_archiver save <name> [index patterns...]`: save the mapping and documents in one or more indexes that match the wild-card patterns into an the `<name>` archive - `./bin/es_archiver load <name>`: load the mapping and documents from the `<name>` archive * [functional_tests/common/nagivate] check for statusPage * [es_archiver] move bins into new scripts dir * [functional_tests/apps/context] use esArchiver * [esArchiver] general improvements after showing to a few folks - remove auto-upgrading config doc logic (until we have better access to kibana version info) - export unload command - remove preemptive checks in favor of reacting to errors - use type "doc" vs "hit" for doc records (consistency) - wrote a bunch of pending tests to think though and plan * [esArchiver] make log a stream that writes to itself * [esArchiver] fill in stats and archive format tests * [esArchiver] splitup action logic * [esArchiver/cli] fix cli --help output and comment * [esArchiver] remove type-based param coercion * [esArchiver/log] use strings for log levels * [esArchvier] remove unused var * [esArchiver/indexDocRecordsStream] add tests * [esArchive] fill in remaining tests * [esArchiver] fix dem tests * [eslint] remove unused vars * [esArchiver/loadIfNeeded] fix call to load() * [esArchiver] remove loadDumpData helpers
simianhacker
added a commit
that referenced
this pull request
Mar 28, 2017
…#10890) * Removing derivative behavior... adding better tooltips to add delete * Re-organizing metrics; Adding headers; Adding default behavoir * Adding help text * Additional fixes - Fix for deactivated still being able to be added - Added Experimental Feature to description * [elasticsearch/healthCheck] ensure that multi.allow_explicit_index=true (#10855) * [elasticsearch/healthCheck] ensure that multi.allow_explicit_index=true * [elasticsearch/healthCheck] fix tests * [eslint/console] enable no-undef rule (#10881) * [eslint/console] enable no-undef rule * [eslint/console] fix no-undef violations * Navbarextensions improvements (#9871) * - added controller option - added $location service to determine on which the user is at currently * added test for the hideButton method that is dependent on the location path * removed argument * use locals object to pass variables to the template's scope * use map object * fixing percentage mode extents (#10843) * fixes axis title for new axis (#10866) * fixes metrics options matching (#10865) * Add kuiLocalTab-isDisabled state. (#10830) * Add kuiLocalTab-isDisabled state. * Update Management header to use kuiLocalTab-isDisabled class. * test: send content-type with proxy POST tests (#10903) Elasticsearch now requires that content-type be sent on all requests with payloads, so our tests should be sending it with requests as well. * [esArchiver] combine elasticDump and ScenarioManager (#10359) * As a part of bringing functional testing to plugins, esArchiver gives these plugins a way to capture and reload es indexes without needing to write a bunch of custom code. It works similarly to the elasticDump and ScenarioManager tools that it replaces. Differences: - Streaming implementation allows for much larger archives - CLI for creating and using archives - Configurable archive location - Stores the data in gzipped files (better for source control, searching, large archives) - Automatically identifies and upgrades Kibana config documents Methods: - `#load(name)`: import an archive - `#loadIfNeeded(name)`: import an archive, but skip the documents what belong to any existing index - `#unload(name)`: delete the indexes stored in an archive CLI operations: - `./bin/es_archiver save <name> [index patterns...]`: save the mapping and documents in one or more indexes that match the wild-card patterns into an the `<name>` archive - `./bin/es_archiver load <name>`: load the mapping and documents from the `<name>` archive * [functional_tests/common/nagivate] check for statusPage * [es_archiver] move bins into new scripts dir * [functional_tests/apps/context] use esArchiver * [esArchiver] general improvements after showing to a few folks - remove auto-upgrading config doc logic (until we have better access to kibana version info) - export unload command - remove preemptive checks in favor of reacting to errors - use type "doc" vs "hit" for doc records (consistency) - wrote a bunch of pending tests to think though and plan * [esArchiver] make log a stream that writes to itself * [esArchiver] fill in stats and archive format tests * [esArchiver] splitup action logic * [esArchiver/cli] fix cli --help output and comment * [esArchiver] remove type-based param coercion * [esArchiver/log] use strings for log levels * [esArchvier] remove unused var * [esArchiver/indexDocRecordsStream] add tests * [esArchive] fill in remaining tests * [esArchiver] fix dem tests * [eslint] remove unused vars * [esArchiver/loadIfNeeded] fix call to load() * [esArchiver] remove loadDumpData helpers
2 tasks
spalger
added a commit
to spalger/kibana
that referenced
this pull request
Apr 11, 2017
* As a part of bringing functional testing to plugins, esArchiver gives these plugins a way to capture and reload es indexes without needing to write a bunch of custom code. It works similarly to the elasticDump and ScenarioManager tools that it replaces. Differences: - Streaming implementation allows for much larger archives - CLI for creating and using archives - Configurable archive location - Stores the data in gzipped files (better for source control, searching, large archives) - Automatically identifies and upgrades Kibana config documents Methods: - `#load(name)`: import an archive - `#loadIfNeeded(name)`: import an archive, but skip the documents what belong to any existing index - `#unload(name)`: delete the indexes stored in an archive CLI operations: - `./bin/es_archiver save <name> [index patterns...]`: save the mapping and documents in one or more indexes that match the wild-card patterns into an the `<name>` archive - `./bin/es_archiver load <name>`: load the mapping and documents from the `<name>` archive * [functional_tests/common/nagivate] check for statusPage * [es_archiver] move bins into new scripts dir * [functional_tests/apps/context] use esArchiver * [esArchiver] general improvements after showing to a few folks - remove auto-upgrading config doc logic (until we have better access to kibana version info) - export unload command - remove preemptive checks in favor of reacting to errors - use type "doc" vs "hit" for doc records (consistency) - wrote a bunch of pending tests to think though and plan * [esArchiver] make log a stream that writes to itself * [esArchiver] fill in stats and archive format tests * [esArchiver] splitup action logic * [esArchiver/cli] fix cli --help output and comment * [esArchiver] remove type-based param coercion * [esArchiver/log] use strings for log levels * [esArchvier] remove unused var * [esArchiver/indexDocRecordsStream] add tests * [esArchive] fill in remaining tests * [esArchiver] fix dem tests * [eslint] remove unused vars * [esArchiver/loadIfNeeded] fix call to load() * [esArchiver] remove loadDumpData helpers
epixa
pushed a commit
that referenced
this pull request
Apr 12, 2017
* [esArchiver] combine elasticDump and ScenarioManager (#10359) * As a part of bringing functional testing to plugins, esArchiver gives these plugins a way to capture and reload es indexes without needing to write a bunch of custom code. It works similarly to the elasticDump and ScenarioManager tools that it replaces. Differences: - Streaming implementation allows for much larger archives - CLI for creating and using archives - Configurable archive location - Stores the data in gzipped files (better for source control, searching, large archives) - Automatically identifies and upgrades Kibana config documents Methods: - `#load(name)`: import an archive - `#loadIfNeeded(name)`: import an archive, but skip the documents what belong to any existing index - `#unload(name)`: delete the indexes stored in an archive CLI operations: - `./bin/es_archiver save <name> [index patterns...]`: save the mapping and documents in one or more indexes that match the wild-card patterns into an the `<name>` archive - `./bin/es_archiver load <name>`: load the mapping and documents from the `<name>` archive * [functional_tests/common/nagivate] check for statusPage * [es_archiver] move bins into new scripts dir * [functional_tests/apps/context] use esArchiver * [esArchiver] general improvements after showing to a few folks - remove auto-upgrading config doc logic (until we have better access to kibana version info) - export unload command - remove preemptive checks in favor of reacting to errors - use type "doc" vs "hit" for doc records (consistency) - wrote a bunch of pending tests to think though and plan * [esArchiver] make log a stream that writes to itself * [esArchiver] fill in stats and archive format tests * [esArchiver] splitup action logic * [esArchiver/cli] fix cli --help output and comment * [esArchiver] remove type-based param coercion * [esArchiver/log] use strings for log levels * [esArchvier] remove unused var * [esArchiver/indexDocRecordsStream] add tests * [esArchive] fill in remaining tests * [esArchiver] fix dem tests * [eslint] remove unused vars * [esArchiver/loadIfNeeded] fix call to load() * [esArchiver] remove loadDumpData helpers * [esArchiver] update archives for 5.x * [functionalTestRunner] replace intern (#10910) * [functional_test_runner] replace functional testing tools with custom/pluggable solution * [functional_test_runner] Convert unit tests to commonjs format * [functional_test_runner] Fix dashboard test in wrong mode * [functional_test_runner] Add dashboardLandingPage test subject * [functional_test_runner] Get Visualize page object * [functional_test_runner] Fix outdated references * [functional_test_runner] Fix more outdated refs * [functional_test_runner] Remove duplicate tests * [functional_test_runner] Improve test readability * [functional_test_runner] 😞 So many duplicate methods * [functional_test_runner] Move mgmt `before` outside toplevel describe * [functional_test_runner] Settings page obj missing methods * [functional_test_runner] Add improvements from @gammon * [functional_test_runner] Fix return statements in async funcs * [functional_test_runner] Move before() to correct scope * [functional_test_runner] Add after() hooks to remove index patterns * [functional_test_runner] Attempt to fix vertical bar chart tests * [functional_test_runner] Clean up * [functional_test_runner] Reinstate unit tests * [functional_test_runner] Set default loglevel back to info * [functional_test_runner] Replace `context`s with `describe`s * [functional_test_runner] Better error handling * [functional_test_runner] Add in new Tile Map tests * Incorporate changes from master * [functional_test_runner] validate that every test file has a single top-level suite * Update contributing doc with link to full doc * [docs] Spelling and grammar fixes * docs: writing and running functional tests * [docs] Move plugin doc to plugin area * [docs] Housekeeping. Doc in wrong place * [docs] Remove dup doc file * [grunt] Only run mocha_setup when running tests, not every grunt task * [eslint] remove use of context()
12 tasks
1 task
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Builds on top of #10187
A dependency of #9853
As a part of bringing functional testing to plugins, esArchiver gives these plugins a way to capture and reload es indexes without needing to write a bunch of custom code. It works similarly to the elasticDump and ScenarioManager tools that it replaces.
Differences:
Methods:
#load(name): import an archive#loadIfNeeded(name): import an archive, but skip the documents what belong to any existing index#unload(name): delete the indexes stored in an archive#save(name, indices): programmatically save indices to an archive (using the CLI is preferred)CLI operations:
node scripts/es_archiver save <name> [index patterns...]<name>archivenode scripts/es_archiver load <name><name>archivenode scripts/es_archiver --help