add a test that creates a zar space under brim #858
Merged
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
mikesbrown
commented
Jun 4, 2020
|
This is ready again. The failure is brimdata/super#613 . |
jameskerr
reviewed
Jun 15, 2020
jameskerr
approved these changes
Jun 15, 2020
Brim isn't fully ready to handle zar spaces yet, but this test establishes a temporary workflow for getting zar data into a space and making sure Brim starts. As Brim's functionality to read a Zar space improves, this test can be modified. The purpose of this is to put something in place that changes alongside the functionality as opposed to having nothing until all the functionality is ready. As such, there's not as much abstraction as there otherwise would be: much of this should get thrown away once Brim supports Zar. The test uses Brim as the driver for starting zqd so that the test don't have to do zqd orchestration itself. It ingests sample.tsv as usual. Then it uses zealot to create a second, empty space called sample.zar and also get space metadata from space sample.tsv.brim. The test then imports samle.tsv.brim's all.zng into sample.zar, with a small enough chop size to ensure multiple files. The test creates a number of indexes. Finally, it restarts Brim and ensures Brim restarts.
|
👍 |
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Brim isn't fully ready to handle zar spaces yet, but this test
establishes a temporary workflow for getting zar data into a space and
making sure Brim starts. As Brim's functionality to read a Zar space
improves, this test can be modified. The purpose of this is to put
something in place that changes alongside the functionality as opposed
to having nothing until all the functionality is ready. As such, there's
not as much abstraction as there otherwise would be: much of this should
get thrown away once Brim supports Zar.
The test uses Brim as the driver for starting zqd so that the test don't
have to do zqd orchestration itself. It ingests sample.tsv as usual.
Then it uses zealot to create a second, empty space called sample.zar
and also get space metadata from space sample.tsv.brim. The test then
imports samle.tsv.brim's all.zng into sample.zar, with a small enough
chop size to ensure multiple files. The test creates a number of
indexes. Finally, it reloads Brim and ensures Brim sees the second space.