Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

fix: just synchronize all DAO operations to avoid H2 threading problems #202

Merged
merged 4 commits into from
Feb 27, 2024
Merged

fix: just synchronize all DAO operations to avoid H2 threading problems #202

merged 4 commits into from
Feb 27, 2024

Conversation

MikeDombo
Copy link
Member

@MikeDombo MikeDombo commented Feb 22, 2024 •
edited
Loading

Issue #, if available:

Description of changes:
Added a stress test that updates shadow over IPC 1,000 times as fast as possible, causing shadow manager to need to sync to cloud quickly as well. This replicated a customer issue where they saw a nullpointerexception coming from H2. The fix is to simply synchronize all DB access.

I auto-formatted src/integrationtests/java/com/aws/greengrass/integrationtests/ipc/ShadowIPCTest.java, so forgive the large change there. The meaningful change is the addition of test 0 which is the stress test.

Why is this change necessary:

How was this change tested:

Any additional information or context required to review the change:

Checklist:

  • Updated the README if applicable
  • Updated or added new unit tests
  • Updated or added new integration tests
  • If your code makes a remote network call, it was tested with a proxy

By submitting this pull request, I confirm that you can use, modify, copy, and redistribute this contribution, under the terms of your choice.

@github-actions GitHub Actions
Copy link

github-actions bot commented Feb 22, 2024 •
edited
Loading

Unit Tests Coverage Report

File Coverage Lines Branches
All files 83% 89% 78% ✅
com.aws.greengrass.shadowmanager.sync.model.MergedFullShadowSyncRequest 21% 26% 15% ❌
com.aws.greengrass.shadowmanager.sync.model.LocalUpdateSyncRequest 79% 85% 73% ✅
com.aws.greengrass.shadowmanager.sync.model.Direction 100% 100% 100% ✅
com.aws.greengrass.shadowmanager.sync.model.LocalDeleteSyncRequest 99% 98% 100% ✅
com.aws.greengrass.shadowmanager.sync.model.FullShadowSyncRequest 84% 97% 71% ✅
com.aws.greengrass.shadowmanager.sync.model.CloudDeleteSyncRequest 80% 78% 83% ✅
com.aws.greengrass.shadowmanager.sync.model.DirectionWrapper 100% 100% 0% ✅
com.aws.greengrass.shadowmanager.sync.model.OverwriteCloudShadowRequest 98% 95% 100% ✅
com.aws.greengrass.shadowmanager.sync.model.BaseSyncRequest 80% 93% 68% ✅
com.aws.greengrass.shadowmanager.sync.model.OverwriteLocalShadowRequest 98% 95% 100% ✅
com.aws.greengrass.shadowmanager.sync.model.CloudUpdateSyncRequest 74% 76% 71% ✅
com.aws.greengrass.shadowmanager.ShadowManagerDatabase 2% 2% 0% ❌
com.aws.greengrass.shadowmanager.ShadowManager 75% 79% 72% ✅
com.aws.greengrass.shadowmanager.ShadowManagerDAOImpl 98% 99% 96% ✅
com.aws.greengrass.shadowmanager.AuthorizationHandlerWrapper 100% 100% 100% ✅
com.aws.greengrass.shadowmanager.PubSubIntegrator 89% 93% 86% ✅
com.aws.greengrass.shadowmanager.ShadowManager$1 43% 43% 0% ❌
software.amazon.awssdk.aws.greengrass.GeneratedAbstractDeleteThingShadowOperationHandler 67% 67% 0% ✅
software.amazon.awssdk.aws.greengrass.GeneratedAbstractListNamedShadowsForThingOperationHandler 67% 67% 0% ✅
software.amazon.awssdk.aws.greengrass.GeneratedAbstractUpdateThingShadowOperationHandler 67% 67% 0% ✅
software.amazon.awssdk.aws.greengrass.GeneratedAbstractGetThingShadowOperationHandler 67% 67% 0% ✅
com.aws.greengrass.shadowmanager.util.JsonMerger 94% 98% 89% ✅
com.aws.greengrass.shadowmanager.util.SyncNodeMerger 89% 96% 82% ✅
com.aws.greengrass.shadowmanager.util.DataOwner 100% 100% 0% ✅
com.aws.greengrass.shadowmanager.util.ShadowWriteSynchronizeHelper 100% 100% 0% ✅
com.aws.greengrass.shadowmanager.util.Validator 100% 100% 100% ✅
com.aws.greengrass.shadowmanager.util.JsonUtil 94% 99% 89% ✅
com.aws.greengrass.shadowmanager.ipc.BaseRequestHandler 77% 77% 0% ✅
com.aws.greengrass.shadowmanager.ipc.GetThingShadowRequestHandler 88% 100% 75% ✅
com.aws.greengrass.shadowmanager.ipc.PubSubClientWrapper 100% 100% 0% ✅
com.aws.greengrass.shadowmanager.ipc.IpcRateLimiter 100% 100% 100% ✅
com.aws.greengrass.shadowmanager.ipc.UpdateThingShadowIPCHandler 100% 100% 0% ✅
com.aws.greengrass.shadowmanager.ipc.InboundRateLimiter 94% 89% 100% ✅
com.aws.greengrass.shadowmanager.ipc.DeleteThingShadowRequestHandler 99% 99% 100% ✅
com.aws.greengrass.shadowmanager.ipc.InboundRateLimiter$1 100% 100% 100% ✅
com.aws.greengrass.shadowmanager.ipc.UpdateThingShadowRequestHandler 91% 95% 88% ✅
com.aws.greengrass.shadowmanager.ipc.DeleteThingShadowIPCHandler 100% 100% 0% ✅
com.aws.greengrass.shadowmanager.ipc.GetThingShadowIPCHandler 100% 100% 0% ✅
com.aws.greengrass.shadowmanager.ipc.ListNamedShadowsForThingIPCHandler 100% 100% 100% ✅
com.aws.greengrass.shadowmanager.ipc.model.Operation 100% 100% 0% ✅
com.aws.greengrass.shadowmanager.ipc.model.PubSubRequest 100% 100% 0% ✅
com.aws.greengrass.shadowmanager.sync.strategy.PeriodicSyncStrategy 63% 77% 50% ❌
com.aws.greengrass.shadowmanager.sync.strategy.SyncStrategyFactory 100% 100% 100% ✅
com.aws.greengrass.shadowmanager.sync.strategy.BaseSyncStrategy 93% 97% 89% ✅
com.aws.greengrass.shadowmanager.sync.strategy.RealTimeSyncStrategy 76% 76% 75% ✅
com.aws.greengrass.shadowmanager.sync.IotDataPlaneClientFactory 55% 85% 25% ❌
com.aws.greengrass.shadowmanager.sync.RequestBlockingQueue 96% 99% 93% ✅
com.aws.greengrass.shadowmanager.sync.SyncHandler 71% 88% 53% ✅
com.aws.greengrass.shadowmanager.sync.IotDataPlaneClientWrapper 91% 91% 0% ✅
com.aws.greengrass.shadowmanager.sync.RequestMerger 79% 77% 81% ✅
com.aws.greengrass.shadowmanager.sync.CloudDataClient 74% 72% 77% ✅
com.aws.greengrass.shadowmanager.model.configuration.ShadowSyncConfiguration 87% 93% 82% ✅
com.aws.greengrass.shadowmanager.model.configuration.ThingShadowSyncConfiguration 76% 78% 75% ✅
com.aws.greengrass.shadowmanager.configuration.ShadowDocSizeConfiguration 88% 92% 83% ✅
com.aws.greengrass.shadowmanager.configuration.RateLimitsConfiguration 100% 100% 0% ✅
com.aws.greengrass.shadowmanager.configuration.ComponentConfiguration 100% 100% 0% ✅
com.aws.greengrass.shadowmanager.sync.strategy.model.StrategyType 86% 92% 80% ✅
com.aws.greengrass.shadowmanager.sync.strategy.model.Strategy 90% 100% 80% ✅
com.aws.greengrass.shadowmanager.model.ResponseMessageBuilder 100% 100% 0% ✅
com.aws.greengrass.shadowmanager.model.ShadowDocument 85% 90% 80% ✅
com.aws.greengrass.shadowmanager.model.ShadowStateMetadata 93% 98% 89% ✅
com.aws.greengrass.shadowmanager.model.ErrorMessage 75% 100% 50% ✅
com.aws.greengrass.shadowmanager.model.ShadowRequest 100% 100% 100% ✅
com.aws.greengrass.shadowmanager.model.ShadowState 78% 81% 75% ✅
com.aws.greengrass.shadowmanager.model.LogEvents 100% 100% 0% ✅

Minimum allowed coverage is 65%

Generated by 🐒 cobertura-action against 82e9b49

@github-actions GitHub Actions
Copy link

github-actions bot commented Feb 22, 2024 •
edited
Loading

Integration Tests Coverage Report

File Coverage Lines Branches
All files 72% 76% 69% ✅
com.aws.greengrass.shadowmanager.sync.model.MergedFullShadowSyncRequest 72% 78% 65% ✅
com.aws.greengrass.shadowmanager.sync.model.LocalUpdateSyncRequest 78% 83% 73% ✅
com.aws.greengrass.shadowmanager.sync.model.Direction 83% 91% 75% ✅
com.aws.greengrass.shadowmanager.sync.model.LocalDeleteSyncRequest 54% 58% 50% ✅
com.aws.greengrass.shadowmanager.sync.model.FullShadowSyncRequest 71% 79% 62% ✅
com.aws.greengrass.shadowmanager.sync.model.CloudDeleteSyncRequest 52% 54% 50% ✅
com.aws.greengrass.shadowmanager.sync.model.DirectionWrapper 100% 100% 0% ✅
com.aws.greengrass.shadowmanager.sync.model.OverwriteCloudShadowRequest 34% 43% 25% ❌
com.aws.greengrass.shadowmanager.sync.model.BaseSyncRequest 54% 57% 50% ✅
com.aws.greengrass.shadowmanager.sync.model.OverwriteLocalShadowRequest 29% 29% 0% ❌
com.aws.greengrass.shadowmanager.sync.model.CloudUpdateSyncRequest 62% 60% 64% ✅
com.aws.greengrass.shadowmanager.ShadowManagerDatabase 53% 57% 50% ✅
com.aws.greengrass.shadowmanager.ShadowManager 89% 91% 88% ✅
com.aws.greengrass.shadowmanager.ShadowManagerDAOImpl 82% 90% 75% ✅
com.aws.greengrass.shadowmanager.AuthorizationHandlerWrapper 100% 100% 100% ✅
com.aws.greengrass.shadowmanager.PubSubIntegrator 25% 22% 29% ❌
com.aws.greengrass.shadowmanager.ShadowManager$1 14% 14% 0% ❌
software.amazon.awssdk.aws.greengrass.GeneratedAbstractDeleteThingShadowOperationHandler 100% 100% 0% ✅
software.amazon.awssdk.aws.greengrass.GeneratedAbstractListNamedShadowsForThingOperationHandler 100% 100% 0% ✅
software.amazon.awssdk.aws.greengrass.GeneratedAbstractUpdateThingShadowOperationHandler 100% 100% 0% ✅
software.amazon.awssdk.aws.greengrass.GeneratedAbstractGetThingShadowOperationHandler 100% 100% 0% ✅
com.aws.greengrass.shadowmanager.util.JsonMerger 73% 81% 66% ✅
com.aws.greengrass.shadowmanager.util.SyncNodeMerger 75% 86% 64% ✅
com.aws.greengrass.shadowmanager.util.DataOwner 100% 100% 0% ✅
com.aws.greengrass.shadowmanager.util.ShadowWriteSynchronizeHelper 100% 100% 0% ✅
com.aws.greengrass.shadowmanager.util.Validator 67% 68% 65% ✅
com.aws.greengrass.shadowmanager.util.JsonUtil 83% 90% 77% ✅
com.aws.greengrass.shadowmanager.ipc.BaseRequestHandler 77% 77% 0% ✅
com.aws.greengrass.shadowmanager.ipc.GetThingShadowRequestHandler 75% 74% 75% ✅
com.aws.greengrass.shadowmanager.ipc.PubSubClientWrapper 82% 82% 0% ✅
com.aws.greengrass.shadowmanager.ipc.IpcRateLimiter 100% 100% 100% ✅
com.aws.greengrass.shadowmanager.ipc.UpdateThingShadowIPCHandler 55% 55% 0% ✅
com.aws.greengrass.shadowmanager.ipc.InboundRateLimiter 94% 100% 88% ✅
com.aws.greengrass.shadowmanager.ipc.DeleteThingShadowRequestHandler 86% 72% 100% ✅
com.aws.greengrass.shadowmanager.ipc.InboundRateLimiter$1 75% 100% 50% ✅
com.aws.greengrass.shadowmanager.ipc.UpdateThingShadowRequestHandler 74% 77% 71% ✅
com.aws.greengrass.shadowmanager.ipc.DeleteThingShadowIPCHandler 53% 53% 0% ✅
com.aws.greengrass.shadowmanager.ipc.GetThingShadowIPCHandler 95% 95% 0% ✅
com.aws.greengrass.shadowmanager.ipc.ListNamedShadowsForThingIPCHandler 61% 55% 67% ✅
com.aws.greengrass.shadowmanager.ipc.model.Operation 100% 100% 0% ✅
com.aws.greengrass.shadowmanager.ipc.model.PubSubRequest 100% 100% 0% ✅
com.aws.greengrass.shadowmanager.sync.strategy.PeriodicSyncStrategy 71% 92% 50% ✅
com.aws.greengrass.shadowmanager.sync.strategy.SyncStrategyFactory 100% 100% 100% ✅
com.aws.greengrass.shadowmanager.sync.strategy.BaseSyncStrategy 75% 83% 67% ✅
com.aws.greengrass.shadowmanager.sync.strategy.RealTimeSyncStrategy 83% 90% 75% ✅
com.aws.greengrass.shadowmanager.sync.IotDataPlaneClientFactory 77% 93% 61% ✅
com.aws.greengrass.shadowmanager.sync.RequestBlockingQueue 61% 70% 52% ✅
com.aws.greengrass.shadowmanager.sync.SyncHandler 82% 93% 71% ✅
com.aws.greengrass.shadowmanager.sync.IotDataPlaneClientWrapper 100% 100% 0% ✅
com.aws.greengrass.shadowmanager.sync.RequestMerger 60% 67% 52% ✅
com.aws.greengrass.shadowmanager.sync.CloudDataClient 51% 46% 57% ✅
com.aws.greengrass.shadowmanager.model.configuration.ShadowSyncConfiguration 71% 81% 61% ✅
com.aws.greengrass.shadowmanager.model.configuration.ThingShadowSyncConfiguration 76% 78% 75% ✅
com.aws.greengrass.shadowmanager.configuration.ShadowDocSizeConfiguration 60% 69% 50% ✅
com.aws.greengrass.shadowmanager.configuration.RateLimitsConfiguration 100% 100% 0% ✅
com.aws.greengrass.shadowmanager.configuration.ComponentConfiguration 100% 100% 0% ✅
com.aws.greengrass.shadowmanager.sync.strategy.model.StrategyType 68% 77% 60% ✅
com.aws.greengrass.shadowmanager.sync.strategy.model.Strategy 90% 100% 80% ✅
com.aws.greengrass.shadowmanager.model.ResponseMessageBuilder 91% 91% 0% ✅
com.aws.greengrass.shadowmanager.model.ShadowDocument 92% 95% 90% ✅
com.aws.greengrass.shadowmanager.model.ShadowStateMetadata 91% 95% 87% ✅
com.aws.greengrass.shadowmanager.model.ErrorMessage 97% 94% 100% ✅
com.aws.greengrass.shadowmanager.model.ShadowRequest 88% 77% 100% ✅
com.aws.greengrass.shadowmanager.model.ShadowState 96% 98% 94% ✅
com.aws.greengrass.shadowmanager.model.LogEvents 100% 100% 0% ✅

Minimum allowed coverage is 45%

Generated by 🐒 cobertura-action against 82e9b49

saranyailla
saranyailla previously approved these changes Feb 22, 2024
View reviewed changes
jcosentino11
jcosentino11 previously approved these changes Feb 23, 2024
View reviewed changes
@MikeDombo MikeDombo dismissed stale reviews from jcosentino11 and saranyailla via 40ecf4c February 27, 2024 17:12
@MikeDombo MikeDombo marked this pull request as ready for review February 27, 2024 17:21
jcosentino11
jcosentino11 previously approved these changes Feb 27, 2024
View reviewed changes
tiancishen
tiancishen previously approved these changes Feb 27, 2024
View reviewed changes
saranyailla
saranyailla previously approved these changes Feb 27, 2024
View reviewed changes
@MikeDombo MikeDombo merged commit 0199edd into main Feb 27, 2024
4 checks passed
@MikeDombo MikeDombo deleted the thread branch February 27, 2024 20:59
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Reviewers

@saranyailla saranyailla saranyailla approved these changes

@jcosentino11 jcosentino11 jcosentino11 approved these changes

@tiancishen tiancishen Awaiting requested review from tiancishen

Assignees
No one assigned
Labels
None yet
Projects
None yet
Milestone
No milestone
Development

Successfully merging this pull request may close these issues.
4 participants
@MikeDombo @jcosentino11 @saranyailla @tiancishen