Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

fix: retain null fields in sync node merger #201

Merged
merged 8 commits into from
Feb 1, 2024

Conversation

sameerzuberi
Copy link
Member

@sameerzuberi sameerzuberi commented Jan 23, 2024

Issue #, if available:

Description of changes:

  • Previously when a full sync was performed, SyncNodeMerger would remove any fields that were set to null from the shadow document. Then, once this document was merged during the local update, those removed fields were reintroduced since the merge was unaware they had already been set to null before. With this fix we now retain null fields inside of SyncNodeMerger so that they can be handled and deleted from the local shadow document during the following merges.
  • Updated the integration test framework to allow for setting an initial local shadow state before starting the Nucleus. This allows us to simulate receiving cloud updates while the device is not yet running.

Why is this change necessary:
Fixes a bug where customers could delete fields in their cloud shadow while the device was offline, but after starting the device would not remove these fields from the local shadow.

How was this change tested:
Added an integration test which fails without the change and passes with it.

Any additional information or context required to review the change:
FullShadowSyncRequestTest.java will need to be refactored in a future PR. With this change, any item removed from a shadow document is set to null in SyncNodeMerger and then handled accordingly in UpdateThingShadowRequestHandler. In the current unit tests, UpdateThingShadowRequestHandler has mocked behavior which causes the null values to not be handled as they would be in practice, so the null value has been added into the expected result for now, pending a refactor of the tests.

Checklist:

  • Updated the README if applicable
  • Updated or added new unit tests
  • Updated or added new integration tests
  • If your code makes a remote network call, it was tested with a proxy

By submitting this pull request, I confirm that you can use, modify, copy, and redistribute this contribution, under the terms of your choice.

@sameerzuberi sameerzuberi changed the title fix: remove local shadow fields deleted in the cloud fix: retain null fields in sync node merger Feb 1, 2024
Copy link

github-actions bot commented Feb 1, 2024

Unit Tests Coverage Report

File Coverage Lines Branches
All files 83% 89% 78%
com.aws.greengrass.shadowmanager.sync.model.MergedFullShadowSyncRequest 21% 26% 15%
com.aws.greengrass.shadowmanager.sync.model.LocalUpdateSyncRequest 79% 85% 73%
com.aws.greengrass.shadowmanager.sync.model.Direction 100% 100% 100%
com.aws.greengrass.shadowmanager.sync.model.LocalDeleteSyncRequest 99% 98% 100%
com.aws.greengrass.shadowmanager.sync.model.FullShadowSyncRequest 84% 97% 71%
com.aws.greengrass.shadowmanager.sync.model.CloudDeleteSyncRequest 80% 78% 83%
com.aws.greengrass.shadowmanager.sync.model.DirectionWrapper 100% 100% 0%
com.aws.greengrass.shadowmanager.sync.model.OverwriteCloudShadowRequest 98% 95% 100%
com.aws.greengrass.shadowmanager.sync.model.BaseSyncRequest 81% 93% 69%
com.aws.greengrass.shadowmanager.sync.model.OverwriteLocalShadowRequest 98% 95% 100%
com.aws.greengrass.shadowmanager.sync.model.CloudUpdateSyncRequest 74% 76% 71%
com.aws.greengrass.shadowmanager.ShadowManagerDatabase 2% 2% 0%
com.aws.greengrass.shadowmanager.ShadowManager 76% 79% 74%
com.aws.greengrass.shadowmanager.ShadowManagerDAOImpl 96% 97% 96%
com.aws.greengrass.shadowmanager.AuthorizationHandlerWrapper 100% 100% 100%
com.aws.greengrass.shadowmanager.PubSubIntegrator 89% 93% 86%
com.aws.greengrass.shadowmanager.ShadowManager$1 43% 43% 0%
software.amazon.awssdk.aws.greengrass.GeneratedAbstractDeleteThingShadowOperationHandler 67% 67% 0%
software.amazon.awssdk.aws.greengrass.GeneratedAbstractListNamedShadowsForThingOperationHandler 67% 67% 0%
software.amazon.awssdk.aws.greengrass.GeneratedAbstractUpdateThingShadowOperationHandler 67% 67% 0%
software.amazon.awssdk.aws.greengrass.GeneratedAbstractGetThingShadowOperationHandler 67% 67% 0%
com.aws.greengrass.shadowmanager.util.JsonMerger 94% 98% 89%
com.aws.greengrass.shadowmanager.util.SyncNodeMerger 89% 96% 82%
com.aws.greengrass.shadowmanager.util.DataOwner 100% 100% 0%
com.aws.greengrass.shadowmanager.util.ShadowWriteSynchronizeHelper 100% 100% 0%
com.aws.greengrass.shadowmanager.util.Validator 100% 100% 100%
com.aws.greengrass.shadowmanager.util.JsonUtil 94% 99% 89%
com.aws.greengrass.shadowmanager.ipc.BaseRequestHandler 77% 77% 0%
com.aws.greengrass.shadowmanager.ipc.GetThingShadowRequestHandler 88% 100% 75%
com.aws.greengrass.shadowmanager.ipc.IpcRateLimiter 100% 100% 100%
com.aws.greengrass.shadowmanager.ipc.PubSubClientWrapper 100% 100% 0%
com.aws.greengrass.shadowmanager.ipc.UpdateThingShadowIPCHandler 100% 100% 0%
com.aws.greengrass.shadowmanager.ipc.InboundRateLimiter 94% 89% 100%
com.aws.greengrass.shadowmanager.ipc.InboundRateLimiter$1 100% 100% 100%
com.aws.greengrass.shadowmanager.ipc.DeleteThingShadowRequestHandler 99% 99% 100%
com.aws.greengrass.shadowmanager.ipc.UpdateThingShadowRequestHandler 91% 95% 88%
com.aws.greengrass.shadowmanager.ipc.DeleteThingShadowIPCHandler 100% 100% 0%
com.aws.greengrass.shadowmanager.ipc.GetThingShadowIPCHandler 100% 100% 0%
com.aws.greengrass.shadowmanager.ipc.ListNamedShadowsForThingIPCHandler 100% 100% 100%
com.aws.greengrass.shadowmanager.ipc.model.Operation 100% 100% 0%
com.aws.greengrass.shadowmanager.ipc.model.PubSubRequest 100% 100% 0%
com.aws.greengrass.shadowmanager.sync.strategy.PeriodicSyncStrategy 63% 77% 50%
com.aws.greengrass.shadowmanager.sync.strategy.SyncStrategyFactory 100% 100% 100%
com.aws.greengrass.shadowmanager.sync.strategy.BaseSyncStrategy 93% 97% 89%
com.aws.greengrass.shadowmanager.sync.strategy.RealTimeSyncStrategy 76% 76% 75%
com.aws.greengrass.shadowmanager.sync.IotDataPlaneClientFactory 55% 85% 25%
com.aws.greengrass.shadowmanager.sync.RequestBlockingQueue 96% 99% 93%
com.aws.greengrass.shadowmanager.sync.SyncHandler 71% 88% 53%
com.aws.greengrass.shadowmanager.sync.IotDataPlaneClientWrapper 91% 91% 0%
com.aws.greengrass.shadowmanager.sync.RequestMerger 79% 77% 81%
com.aws.greengrass.shadowmanager.sync.CloudDataClient 74% 72% 77%
com.aws.greengrass.shadowmanager.model.configuration.ShadowSyncConfiguration 87% 93% 82%
com.aws.greengrass.shadowmanager.model.configuration.ThingShadowSyncConfiguration 76% 78% 75%
com.aws.greengrass.shadowmanager.configuration.ShadowDocSizeConfiguration 100% 100% 100%
com.aws.greengrass.shadowmanager.configuration.RateLimitsConfiguration 100% 100% 0%
com.aws.greengrass.shadowmanager.configuration.ComponentConfiguration 100% 100% 0%
com.aws.greengrass.shadowmanager.sync.strategy.model.StrategyType 86% 92% 80%
com.aws.greengrass.shadowmanager.sync.strategy.model.Strategy 90% 100% 80%
com.aws.greengrass.shadowmanager.model.ResponseMessageBuilder 100% 100% 0%
com.aws.greengrass.shadowmanager.model.ShadowDocument 85% 90% 80%
com.aws.greengrass.shadowmanager.model.ShadowStateMetadata 93% 98% 89%
com.aws.greengrass.shadowmanager.model.ErrorMessage 75% 100% 50%
com.aws.greengrass.shadowmanager.model.ShadowRequest 100% 100% 100%
com.aws.greengrass.shadowmanager.model.ShadowState 78% 81% 75%
com.aws.greengrass.shadowmanager.model.LogEvents 100% 100% 0%

Minimum allowed coverage is 65%

Generated by 🐒 cobertura-action against 6859870

Copy link

github-actions bot commented Feb 1, 2024

Integration Tests Coverage Report

File Coverage Lines Branches
All files 72% 76% 68%
com.aws.greengrass.shadowmanager.sync.model.MergedFullShadowSyncRequest 67% 74% 60%
com.aws.greengrass.shadowmanager.sync.model.LocalUpdateSyncRequest 78% 83% 73%
com.aws.greengrass.shadowmanager.sync.model.Direction 83% 91% 75%
com.aws.greengrass.shadowmanager.sync.model.LocalDeleteSyncRequest 54% 58% 50%
com.aws.greengrass.shadowmanager.sync.model.FullShadowSyncRequest 71% 79% 62%
com.aws.greengrass.shadowmanager.sync.model.CloudDeleteSyncRequest 52% 54% 50%
com.aws.greengrass.shadowmanager.sync.model.DirectionWrapper 100% 100% 0%
com.aws.greengrass.shadowmanager.sync.model.OverwriteCloudShadowRequest 34% 43% 25%
com.aws.greengrass.shadowmanager.sync.model.BaseSyncRequest 52% 57% 46%
com.aws.greengrass.shadowmanager.sync.model.OverwriteLocalShadowRequest 29% 29% 0%
com.aws.greengrass.shadowmanager.sync.model.CloudUpdateSyncRequest 62% 60% 64%
com.aws.greengrass.shadowmanager.ShadowManagerDatabase 60% 60% 60%
com.aws.greengrass.shadowmanager.ShadowManager 90% 91% 88%
com.aws.greengrass.shadowmanager.ShadowManagerDAOImpl 81% 87% 75%
com.aws.greengrass.shadowmanager.AuthorizationHandlerWrapper 100% 100% 100%
com.aws.greengrass.shadowmanager.PubSubIntegrator 25% 22% 29%
com.aws.greengrass.shadowmanager.ShadowManager$1 14% 14% 0%
software.amazon.awssdk.aws.greengrass.GeneratedAbstractDeleteThingShadowOperationHandler 100% 100% 0%
software.amazon.awssdk.aws.greengrass.GeneratedAbstractListNamedShadowsForThingOperationHandler 100% 100% 0%
software.amazon.awssdk.aws.greengrass.GeneratedAbstractUpdateThingShadowOperationHandler 100% 100% 0%
software.amazon.awssdk.aws.greengrass.GeneratedAbstractGetThingShadowOperationHandler 100% 100% 0%
com.aws.greengrass.shadowmanager.util.JsonMerger 68% 75% 61%
com.aws.greengrass.shadowmanager.util.SyncNodeMerger 75% 86% 64%
com.aws.greengrass.shadowmanager.util.DataOwner 100% 100% 0%
com.aws.greengrass.shadowmanager.util.ShadowWriteSynchronizeHelper 100% 100% 0%
com.aws.greengrass.shadowmanager.util.Validator 67% 68% 65%
com.aws.greengrass.shadowmanager.util.JsonUtil 83% 90% 77%
com.aws.greengrass.shadowmanager.ipc.BaseRequestHandler 77% 77% 0%
com.aws.greengrass.shadowmanager.ipc.GetThingShadowRequestHandler 75% 74% 75%
com.aws.greengrass.shadowmanager.ipc.IpcRateLimiter 100% 100% 100%
com.aws.greengrass.shadowmanager.ipc.PubSubClientWrapper 82% 82% 0%
com.aws.greengrass.shadowmanager.ipc.UpdateThingShadowIPCHandler 55% 55% 0%
com.aws.greengrass.shadowmanager.ipc.InboundRateLimiter 94% 100% 88%
com.aws.greengrass.shadowmanager.ipc.InboundRateLimiter$1 75% 100% 50%
com.aws.greengrass.shadowmanager.ipc.DeleteThingShadowRequestHandler 86% 72% 100%
com.aws.greengrass.shadowmanager.ipc.UpdateThingShadowRequestHandler 74% 77% 71%
com.aws.greengrass.shadowmanager.ipc.DeleteThingShadowIPCHandler 53% 53% 0%
com.aws.greengrass.shadowmanager.ipc.GetThingShadowIPCHandler 95% 95% 0%
com.aws.greengrass.shadowmanager.ipc.ListNamedShadowsForThingIPCHandler 61% 55% 67%
com.aws.greengrass.shadowmanager.ipc.model.Operation 100% 100% 0%
com.aws.greengrass.shadowmanager.ipc.model.PubSubRequest 100% 100% 0%
com.aws.greengrass.shadowmanager.sync.strategy.PeriodicSyncStrategy 71% 92% 50%
com.aws.greengrass.shadowmanager.sync.strategy.SyncStrategyFactory 100% 100% 100%
com.aws.greengrass.shadowmanager.sync.strategy.BaseSyncStrategy 78% 87% 70%
com.aws.greengrass.shadowmanager.sync.strategy.RealTimeSyncStrategy 83% 90% 75%
com.aws.greengrass.shadowmanager.sync.IotDataPlaneClientFactory 77% 93% 61%
com.aws.greengrass.shadowmanager.sync.RequestBlockingQueue 61% 70% 52%
com.aws.greengrass.shadowmanager.sync.SyncHandler 82% 93% 71%
com.aws.greengrass.shadowmanager.sync.IotDataPlaneClientWrapper 100% 100% 0%
com.aws.greengrass.shadowmanager.sync.RequestMerger 58% 65% 51%
com.aws.greengrass.shadowmanager.sync.CloudDataClient 54% 51% 57%
com.aws.greengrass.shadowmanager.model.configuration.ShadowSyncConfiguration 71% 81% 61%
com.aws.greengrass.shadowmanager.model.configuration.ThingShadowSyncConfiguration 76% 78% 75%
com.aws.greengrass.shadowmanager.configuration.ShadowDocSizeConfiguration 61% 73% 50%
com.aws.greengrass.shadowmanager.configuration.RateLimitsConfiguration 100% 100% 0%
com.aws.greengrass.shadowmanager.configuration.ComponentConfiguration 100% 100% 0%
com.aws.greengrass.shadowmanager.sync.strategy.model.StrategyType 55% 69% 40%
com.aws.greengrass.shadowmanager.sync.strategy.model.Strategy 90% 100% 80%
com.aws.greengrass.shadowmanager.model.ResponseMessageBuilder 91% 91% 0%
com.aws.greengrass.shadowmanager.model.ShadowDocument 89% 91% 87%
com.aws.greengrass.shadowmanager.model.ShadowStateMetadata 84% 89% 79%
com.aws.greengrass.shadowmanager.model.ErrorMessage 97% 94% 100%
com.aws.greengrass.shadowmanager.model.ShadowRequest 88% 77% 100%
com.aws.greengrass.shadowmanager.model.ShadowState 96% 98% 94%
com.aws.greengrass.shadowmanager.model.LogEvents 100% 100% 0%

Minimum allowed coverage is 45%

Generated by 🐒 cobertura-action against 6859870

@sameerzuberi sameerzuberi merged commit ef1ceeb into main Feb 1, 2024
2 checks passed
@sameerzuberi sameerzuberi deleted the remove-deleted-fields branch February 1, 2024 21:37
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

4 participants