Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Windsock: redis benches - generate topology.yaml #1332

Merged
merged 2 commits into from
Sep 22, 2023

Conversation

rukai
Copy link
Member

@rukai rukai commented Sep 12, 2023

blocked on:
#1334
#1335
#1336

This PR begins a refactor for windsock to define its topology.yaml files using the config structs from shotover and serializing them to yaml.
This allows us to programmatically create the configuration we need for windsock.
The advantages of this are:

  • avoids an exponential blow up in unique topology.yaml files. I wanted to add more cases to windsock but found I would have to double the number of unque topology.yaml files for kafka from 4 to 8.
  • Allows us to reuse more logic between local and cloud runs, particularly important because we currently do not run cloud benches in CI.

However this comes at the cost of introducing a lot of complexity where previously it was simple but just repetitive.

I currently think that we should never use this topology.yaml generation for our integration tests because having hardcoded yaml files is easier to understand, and allows us to properly integration test shotover as average users will have to write these files by hand.

@rukai rukai force-pushed the windsock_generate_topology_yaml branch 6 times, most recently from 07e3aa3 to 0ce6882 Compare September 13, 2023 05:23
@shotover shotover deleted a comment from github-actions bot Sep 13, 2023
@rukai rukai force-pushed the windsock_generate_topology_yaml branch 2 times, most recently from fd14f34 to 59247ab Compare September 13, 2023 23:20
@rukai rukai force-pushed the windsock_generate_topology_yaml branch from 59247ab to d5140ac Compare September 18, 2023 21:05
@rukai rukai force-pushed the windsock_generate_topology_yaml branch from d5140ac to ccf0586 Compare September 21, 2023 01:28
@rukai rukai changed the title Windsock: generate topology.yaml Windsock: redis cloud benches - generate topology.yaml Sep 21, 2023
@rukai rukai changed the title Windsock: redis cloud benches - generate topology.yaml Windsock: redis benches - generate topology.yaml Sep 21, 2023
@rukai rukai marked this pull request as ready for review September 21, 2023 02:24
@rukai rukai enabled auto-merge (squash) September 22, 2023 09:56
@rukai rukai merged commit c3d0615 into shotover:main Sep 22, 2023
38 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

3 participants