-
Notifications
You must be signed in to change notification settings - Fork 324
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
delete: add endpoint for deleting namespaces #2244
Conversation
Codecov Report
@@ Coverage Diff @@
## main #2244 +/- ##
=========================================
Coverage 76.72% 76.72%
Complexity 1147 1147
=========================================
Files 219 219
Lines 5318 5318
Branches 423 423
=========================================
Hits 4080 4080
Misses 763 763
Partials 475 475 📣 We’re building smart automated test selection to slash your CI/CD build times. Learn more |
3d324c7
to
603c147
Compare
|
||
jobs = client.listJobs(namespaceName); | ||
assertThat(jobs).hasSize(1); | ||
} |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Could we add some undelete
scenario?
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
The test does check the number of datasets after the namespace is undeleted. But, looks like that's a new dataset added to the namespace and not the original datasets?
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Yes, the idea here is not to automatically undelete everything in the namespace.
api/src/test/java/marquez/db/migrations/V44_2__BackfillAirflowParentRunsTest.java
Show resolved
Hide resolved
72b4ec7
to
cbcec83
Compare
.findBy(name.getValue()) | ||
.orElseThrow(() -> new NamespaceNotFoundException(name)); | ||
datasetService.deleteByNamespaceName(namespace.getName().getValue()); | ||
jobService.deleteByNamespaceName(namespace.getName().getValue()); |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
What if we soft delete the dataset and job, but an error happens right before we soft delete the namespace?
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Only thing would be that visually empty namespace would appear on the UI, as this is filtered on the frontend.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Do you think we should have more resiliency
about those type of failures? I think this operation is idempotent, as long as there are no followup events for the same namespace.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
I think it's fine as is. Perhaps I would have had NamespaceService.delete()
soft delete the dataset and job within a transaction (in the DAO layer).
status: | ||
patch: off | ||
|
||
ignore: | ||
- "api/src/main/java/marquez/db/migrations/V44_1__UpdateRunsWithJobUUID.java" |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
OOh, cool! Had no idea this was even an option 💯
8cc6eb7
to
342a1c6
Compare
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
A much much requested feature! 💯 🥇
Signed-off-by: Maciej Obuchowski <obuchowski.maciej@gmail.com>
342a1c6
to
b882daa
Compare
Signed-off-by: Maciej Obuchowski <obuchowski.maciej@gmail.com>
b882daa
to
74abc92
Compare
Add option to hide/soft delete namespace.
When namespace is hidden, all datasets and jobs contained by this namespace are also hidden. Namespace stops being shown on UI - frontend filters it by isHidden attribute, instead of filtering on backend. Reason for doing it this way is huge potential complexity of changing namespace contract - for example, a lot of methods check whether namespace exists before performing.
Deleted namespace is being undeleted when relevant OpenLineage event is received. This does not automatically undelete all the datasets and jobs in the namespace, only those that are received in this event.
Also, fixes bug where deleted child job (from an event with
ParentRunFacet
) wasn't getting deleted due to internal job name handling.Closes: #2095
Signed-off-by: Maciej Obuchowski obuchowski.maciej@gmail.com