Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

feat: ingestion logs and migrations #167

Merged
merged 15 commits into from
Sep 12, 2024
Merged

feat: ingestion logs and migrations #167

merged 15 commits into from
Sep 12, 2024

Conversation

mfiedorowicz
Copy link
Member

@mfiedorowicz mfiedorowicz commented Sep 12, 2024 •
edited
Loading

  • add migrations
  • store and retrieve ingestions logs (using protojson for consistent bi-directional marshalling/unmarshalling from/to proto)
  • improve apply change set error handling

@mfiedorowicz
remove diode-redis-cli service
a3f728a 
to be replaced with migrations

Signed-off-by: Michal Fiedorowicz <mfiedorowicz@netboxlabs.com>
@mfiedorowicz
allow to disable migration with environment variable
435bee8 
Signed-off-by: Michal Fiedorowicz <mfiedorowicz@netboxlabs.com>
@mfiedorowicz
add function to apply migrations
f488863 
Signed-off-by: Michal Fiedorowicz <mfiedorowicz@netboxlabs.com>
@mfiedorowicz
marshal ingestion log using protojson
5145bf5 
this is to allow us correctly marshal/unmarshal protobuf messages into json and back into protobuf when retrieving ingestion logs

Signed-off-by: Michal Fiedorowicz <mfiedorowicz@netboxlabs.com>
@mfiedorowicz
unmarshal ingestion log from redis JSON back to protobuf
bf563e2 
Signed-off-by: Michal Fiedorowicz <mfiedorowicz@netboxlabs.com>
@mfiedorowicz
set defaults for virtualization object types
0f45156 
Signed-off-by: Michal Fiedorowicz <mfiedorowicz@netboxlabs.com>
@mfiedorowicz
set default status for dcim device and site if an empty string
97dd4d3 
Signed-off-by: Michal Fiedorowicz <mfiedorowicz@netboxlabs.com>
@mfiedorowicz
update codeowners
11da139 
Signed-off-by: Michal Fiedorowicz <mfiedorowicz@netboxlabs.com>
@mfiedorowicz
add tests for migrations
1d64d9e 
Signed-off-by: Michal Fiedorowicz <mfiedorowicz@netboxlabs.com>
@mfiedorowicz
remove noisy log
a08fe6b 
Signed-off-by: Michal Fiedorowicz <mfiedorowicz@netboxlabs.com>
@mfiedorowicz
rename ChangeSetError to IngestionError
c91e4ad 
Signed-off-by: Michal Fiedorowicz <mfiedorowicz@netboxlabs.com>
@mfiedorowicz
improve apply change set error handling
7ab68c3 
Signed-off-by: Michal Fiedorowicz <mfiedorowicz@netboxlabs.com>
@mfiedorowicz mfiedorowicz self-assigned this Sep 12, 2024
@github-actions GitHub Actions
Copy link

github-actions bot commented Sep 12, 2024 •
edited
Loading

Go test coverage

STATUS ELAPSED PACKAGE COVER PASS FAIL SKIP
🟢 PASS 1.22s github.com/netboxlabs/diode/diode-server/ingester 4.6% 9 0 0
🟢 PASS 0.01s github.com/netboxlabs/diode/diode-server/netbox 0.0% 0 0 0
🟢 PASS 1.57s github.com/netboxlabs/diode/diode-server/netboxdiodeplugin 3.4% 40 0 0
🟢 PASS 1.44s github.com/netboxlabs/diode/diode-server/reconciler 11.2% 50 0 0
🟢 PASS 1.46s github.com/netboxlabs/diode/diode-server/reconciler/changeset 32.7% 81 0 0
🟢 PASS 1.02s github.com/netboxlabs/diode/diode-server/server 85.7% 14 0 0
🟢 PASS 1.01s github.com/netboxlabs/diode/diode-server/version 100.0% 1 0 0

Total coverage: 80.0%

@mfiedorowicz
tidy up
042446c 
Signed-off-by: Michal Fiedorowicz <mfiedorowicz@netboxlabs.com>
@leoparente
Copy link
Contributor

Entity still not being parsed, right?

@mfiedorowicz
document MIGRATION_ENABLED env var
1a902a4 
Signed-off-by: Michal Fiedorowicz <mfiedorowicz@netboxlabs.com>
@leoparente leoparente self-requested a review September 12, 2024 19:04
leoparente
leoparente approved these changes Sep 12, 2024
View reviewed changes
Copy link
Contributor

@leoparente leoparente left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Overall, LGTM. Great work with migrations!

@mfiedorowicz
assert retrieved ingested logs
504e0c3 
Signed-off-by: Michal Fiedorowicz <mfiedorowicz@netboxlabs.com>
@mfiedorowicz
Copy link
Member Author

Entity still not being parsed, right?

Added appropriate assertions in tests 🔬

@mfiedorowicz mfiedorowicz merged commit 3171be5 into develop Sep 12, 2024
6 checks passed
@mfiedorowicz mfiedorowicz deleted the ingestion-logs-protojson-migrations branch September 12, 2024 19:29
@orb-ci
Copy link

orb-ci commented Sep 26, 2024

🎉 This PR is included in version 0.6.0 🎉

The release is available on GitHub release

Your semantic-release bot 📦🚀

@orb-ci
Copy link

orb-ci commented Sep 26, 2024

🎉 This PR is included in version 0.6.0 🎉

The release is available on GitHub release

Your semantic-release bot 📦🚀

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Reviewers

@leoparente leoparente leoparente approved these changes

@natm natm Awaiting requested review from natm

Assignees

@mfiedorowicz mfiedorowicz

Labels
diode-proto diode-reconciler diode-server go markdown released
Projects
None yet
Milestone
No milestone
Development

Successfully merging this pull request may close these issues.
3 participants
@mfiedorowicz @leoparente @orb-ci