feat(ui) Retrieve last ingested timestamp and display on frontend #5600
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
jjoyce0510
reviewed
Aug 11, 2022
| @@ -62,6 +63,9 @@ public Chart apply(@Nonnull final EntityResponse entityResponse) { | |||
| result.setUrn(entityResponse.getUrn().toString()); | |||
| result.setType(EntityType.CHART); | |||
| EnvelopedAspectMap aspectMap = entityResponse.getAspects(); | |||
| Long lastIngested = SystemMetadataUtils.getLastIngested(aspectMap); | |||
jjoyce0510
reviewed
Aug 11, 2022
| SystemMetadata systemMetadata = aspectMap.get(aspect).getSystemMetadata(); | ||
| if (systemMetadata.hasRunId() && !systemMetadata.getRunId().equals(DEFAULT_RUN_ID) && systemMetadata.hasLastObserved()) { | ||
| Long lastObserved = systemMetadata.getLastObserved(); | ||
| if (lastIngested == null || lastObserved > lastIngested) { |
jjoyce0510
reviewed
Aug 11, 2022
| @@ -39,6 +40,8 @@ public static Container map(final EntityResponse entityResponse) { | |||
| final Container result = new Container(); | |||
| final Urn entityUrn = entityResponse.getUrn(); | |||
| final EnvelopedAspectMap aspects = entityResponse.getAspects(); | |||
| Long lastIngested = SystemMetadataUtils.getLastIngested(aspects); | |||
There was a problem hiding this comment.
One thing to keep in mind -- There is a possibility that in the future we become smarter about which aspects we fetch here. At that time we'll need to make some decisions...
There was a problem hiding this comment.
totally - always good to keep that in mind!
jjoyce0510
reviewed
Aug 11, 2022
| entityRegistry: EntityRegistry, | ||
| entityType: EntityType, | ||
| ) { | ||
| const entityTypeCased = |
jjoyce0510
approved these changes
Aug 12, 2022
|
Sorry for reviving this merged PR, @chriscollins3456 , but can I confirm if the logic considers TimeSeriesAspects as well? that is, if i programmatically ONLY form a time series aspect containing datasetStats and include a runId and lastObserved in the MCP, the last synchronized timestamp will also be updated? |
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
This PR gets the timestamp for when a dataset entity was last ingested by checking all of its aspects for the most recent
lastObservedin itssystemMetadatawhere therunIdexists and is not the default (no-run-id-provided) which means that ingestion is what last touched this aspect.Here's what it's looking like on the frontend:
within 1 week of ingestion, green
on hover
hovering over the help icon
Orange when it's more than a week but less than a month
Red when it's more than a month since
Checklist