Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

UI deadlock issue during SSO #2224

Closed
kiiadi opened this issue Nov 21, 2020 · 4 comments
Closed

UI deadlock issue during SSO #2224

kiiadi opened this issue Nov 21, 2020 · 4 comments
Labels
bug We can reproduce the issue and confirmed it is a bug.

Comments

@kiiadi
Copy link
Contributor

kiiadi commented Nov 21, 2020

Describe the bug

UI thread-lock completely locks IDE UI thread during SSO authentication flow.

To reproduce

  1. Create SSO credential
  2. Select SSO credential
  3. When prompted hit "Open in Browser"
  4. Close project (leave IDE running)
  5. Re-open project
  6. Hit "Start SSO Sign-In" action in AWS Explorer

UI Hangs

Thread dump shows UI hang is occurring here :
https://github.com/aws/aws-toolkit-jetbrains/blob/master/jetbrains-core/src/software/aws/toolkits/jetbrains/core/credentials/RefreshConnectionAction.kt#L26

Relevant excerpt from stack-dump:

"AWT-EventQueue-0" prio=0 tid=0x0 nid=0x0 blocked
     java.lang.Thread.State: BLOCKED
 on java.util.concurrent.ConcurrentHashMap$ReservationNode@6c11a232 owned by "ApplicationImpl pooled thread 5" Id=24
	at java.base@11.0.7/java.util.concurrent.ConcurrentHashMap.clear(ConcurrentHashMap.java:1201)
	at software.aws.toolkits.jetbrains.core.DefaultAwsResourceCache.clear(AwsResourceCache.kt:345)
	at software.aws.toolkits.jetbrains.core.credentials.RefreshConnectionAction.actionPerformed(RefreshConnectionAction.kt:30)
	at software.aws.toolkits.jetbrains.core.explorer.ExplorerToolWindow$createActionLabel$1.hyperlinkActivated(ExplorerToolWindow.kt:141)
	at com.intellij.ui.HyperlinkAdapter.hyperlinkUpdate(HyperlinkAdapter.java:11)
	at com.intellij.ui.HyperlinkLabel.fireHyperlinkEvent(HyperlinkLabel.java:205)
	at com.intellij.ui.HyperlinkLabel.processMouseEvent(HyperlinkLabel.java:128)
	at java.desktop@11.0.7/java.awt.Component.processEvent(Component.java:6415)
	at java.desktop@11.0.7/java.awt.Container.processEvent(Container.java:2263)
	at java.desktop@11.0.7/java.awt.Component.dispatchEventImpl(Component.java:5025)
	at java.desktop@11.0.7/java.awt.Container.dispatchEventImpl(Container.java:2321)
	at java.desktop@11.0.7/java.awt.Component.dispatchEvent(Component.java:4857)
	at java.desktop@11.0.7/java.awt.LightweightDispatcher.retargetMouseEvent(Container.java:4918)
	at java.desktop@11.0.7/java.awt.LightweightDispatcher.processMouseEvent(Container.java:4547)
	at java.desktop@11.0.7/java.awt.LightweightDispatcher.dispatchEvent(Container.java:4488)
	at java.desktop@11.0.7/java.awt.Container.dispatchEventImpl(Container.java:2307)
	at java.desktop@11.0.7/java.awt.Window.dispatchEventImpl(Window.java:2773)
	at java.desktop@11.0.7/java.awt.Component.dispatchEvent(Component.java:4857)
	at java.desktop@11.0.7/java.awt.EventQueue.dispatchEventImpl(EventQueue.java:778)
	at java.desktop@11.0.7/java.awt.EventQueue$4.run(EventQueue.java:727)
	at java.desktop@11.0.7/java.awt.EventQueue$4.run(EventQueue.java:721)
	at java.base@11.0.7/java.security.AccessController.doPrivileged(Native Method)
	at java.base@11.0.7/java.security.ProtectionDomain$JavaSecurityAccessImpl.doIntersectionPrivilege(ProtectionDomain.java:85)
	at java.base@11.0.7/java.security.ProtectionDomain$JavaSecurityAccessImpl.doIntersectionPrivilege(ProtectionDomain.java:95)
	at java.desktop@11.0.7/java.awt.EventQueue$5.run(EventQueue.java:751)
	at java.desktop@11.0.7/java.awt.EventQueue$5.run(EventQueue.java:749)
	at java.base@11.0.7/java.security.AccessController.doPrivileged(Native Method)
	at java.base@11.0.7/java.security.ProtectionDomain$JavaSecurityAccessImpl.doIntersectionPrivilege(ProtectionDomain.java:85)
	at java.desktop@11.0.7/java.awt.EventQueue.dispatchEvent(EventQueue.java:748)
	at com.intellij.ide.IdeEventQueue.defaultDispatchEvent(IdeEventQueue.java:967)
	at com.intellij.ide.IdeEventQueue.dispatchMouseEvent(IdeEventQueue.java:904)
	at com.intellij.ide.IdeEventQueue._dispatchEvent(IdeEventQueue.java:836)
	at com.intellij.ide.IdeEventQueue.lambda$dispatchEvent$8(IdeEventQueue.java:450)
	at com.intellij.ide.IdeEventQueue$$Lambda$574/0x00000008005f8440.compute(Unknown Source)
	at com.intellij.openapi.progress.impl.CoreProgressManager.computePrioritized(CoreProgressManager.java:744)
	at com.intellij.ide.IdeEventQueue.lambda$dispatchEvent$9(IdeEventQueue.java:449)
	at com.intellij.ide.IdeEventQueue$$Lambda$571/0x00000008005c3c40.run(Unknown Source)
	at com.intellij.openapi.application.impl.ApplicationImpl.runIntendedWriteActionOnCurrentThread(ApplicationImpl.java:802)
	at com.intellij.ide.IdeEventQueue.dispatchEvent(IdeEventQueue.java:503)
	at java.desktop@11.0.7/java.awt.EventDispatchThread.pumpOneEventForFilters(EventDispatchThread.java:203)
	at java.desktop@11.0.7/java.awt.EventDispatchThread.pumpEventsForFilter(EventDispatchThread.java:124)
	at java.desktop@11.0.7/java.awt.EventDispatchThread.pumpEventsForHierarchy(EventDispatchThread.java:113)
	at java.desktop@11.0.7/java.awt.EventDispatchThread.pumpEvents(EventDispatchThread.java:109)
	at java.desktop@11.0.7/java.awt.EventDispatchThread.pumpEvents(EventDispatchThread.java:101)
	at java.desktop@11.0.7/java.awt.EventDispatchThread.run(EventDispatchThread.java:90)


"ApplicationImpl pooled thread 5" prio=0 tid=0x0 nid=0x0 waiting on condition
     java.lang.Thread.State: TIMED_WAITING
 on kotlinx.coroutines.BlockingCoroutine@735d43f8
	at java.base@11.0.7/jdk.internal.misc.Unsafe.park(Native Method)
	at java.base@11.0.7/java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:234)
	at kotlinx.coroutines.BlockingCoroutine.joinBlocking(Builders.kt:82)
	at kotlinx.coroutines.BuildersKt__BuildersKt.runBlocking(Builders.kt:54)
	at kotlinx.coroutines.BuildersKt.runBlocking(Unknown Source)
	at software.aws.toolkits.core.credentials.sso.SsoCredentialProvider.refreshCredentials(SsoCredentialProvider.kt:35)
	at software.aws.toolkits.core.credentials.sso.SsoCredentialProvider.access$refreshCredentials(SsoCredentialProvider.kt:23)
	at software.aws.toolkits.core.credentials.sso.SsoCredentialProvider$sessionCache$1.invoke(SsoCredentialProvider.kt:29)
	at software.aws.toolkits.core.credentials.sso.SsoCredentialProvider$sessionCache$1.invoke(SsoCredentialProvider.kt:23)
	at software.aws.toolkits.core.credentials.sso.SsoCredentialProvider$sam$java_util_function_Supplier$0.get(SsoCredentialProvider.kt)
	at software.amazon.awssdk.utils.cache.CachedSupplier.refreshCache(CachedSupplier.java:132)
	at software.amazon.awssdk.utils.cache.CachedSupplier.get(CachedSupplier.java:89)
	at software.aws.toolkits.core.credentials.sso.SsoCredentialProvider.resolveCredentials(SsoCredentialProvider.kt:31)
	at software.amazon.awssdk.awscore.client.handler.AwsClientHandlerUtils.createExecutionContext(AwsClientHandlerUtils.java:76)
	at software.amazon.awssdk.awscore.client.handler.AwsSyncClientHandler.createExecutionContext(AwsSyncClientHandler.java:68)
	at software.amazon.awssdk.core.internal.handler.BaseSyncClientHandler.lambda$execute$1(BaseSyncClientHandler.java:97)
	at software.amazon.awssdk.core.internal.handler.BaseSyncClientHandler$$Lambda$2167/0x0000000801788040.get(Unknown Source)
	at software.amazon.awssdk.core.internal.handler.BaseSyncClientHandler.measureApiCallSuccess(BaseSyncClientHandler.java:167)
	at software.amazon.awssdk.core.internal.handler.BaseSyncClientHandler.execute(BaseSyncClientHandler.java:94)
	at software.amazon.awssdk.core.client.handler.SdkSyncClientHandler.execute(SdkSyncClientHandler.java:45)
	at software.amazon.awssdk.awscore.client.handler.AwsSyncClientHandler.execute(AwsSyncClientHandler.java:55)
	at software.amazon.awssdk.services.sts.DefaultStsClient.assumeRole(DefaultStsClient.java:288)
	at software.amazon.awssdk.services.sts.auth.StsAssumeRoleCredentialsProvider.getUpdatedCredentials(StsAssumeRoleCredentialsProvider.java:67)
	at software.amazon.awssdk.services.sts.auth.StsCredentialsProvider.updateSessionCredentials(StsCredentialsProvider.java:80)
	at software.amazon.awssdk.services.sts.auth.StsCredentialsProvider$$Lambda$2194/0x000000080178a040.get(Unknown Source)
	at software.amazon.awssdk.utils.cache.CachedSupplier.refreshCache(CachedSupplier.java:132)
	at software.amazon.awssdk.utils.cache.CachedSupplier.get(CachedSupplier.java:89)
	at software.amazon.awssdk.services.sts.auth.StsCredentialsProvider.resolveCredentials(StsCredentialsProvider.java:91)
	at software.amazon.awssdk.services.sts.auth.StsAssumeRoleCredentialsProvider.resolveCredentials(StsAssumeRoleCredentialsProvider.java:41)
	at software.aws.toolkits.jetbrains.core.credentials.CorrectThreadCredentialsProvider.resolveCredentials(CorrectThreadCredentialsProvider.kt:27)
	at software.aws.toolkits.jetbrains.core.credentials.CredentialManager$AwsCredentialProviderProxy.resolveCredentials(CredentialManager.kt:76)
	at software.aws.toolkits.core.credentials.ToolkitCredentialsProvider.resolveCredentials(ToolkitCredentialsProvider.kt)
	at software.amazon.awssdk.awscore.client.handler.AwsClientHandlerUtils.createExecutionContext(AwsClientHandlerUtils.java:76)
	at software.amazon.awssdk.awscore.client.handler.AwsSyncClientHandler.createExecutionContext(AwsSyncClientHandler.java:68)
	at software.amazon.awssdk.core.internal.handler.BaseSyncClientHandler.lambda$execute$1(BaseSyncClientHandler.java:97)
	at software.amazon.awssdk.core.internal.handler.BaseSyncClientHandler$$Lambda$2167/0x0000000801788040.get(Unknown Source)
	at software.amazon.awssdk.core.internal.handler.BaseSyncClientHandler.measureApiCallSuccess(BaseSyncClientHandler.java:167)
	at software.amazon.awssdk.core.internal.handler.BaseSyncClientHandler.execute(BaseSyncClientHandler.java:94)
	at software.amazon.awssdk.core.client.handler.SdkSyncClientHandler.execute(SdkSyncClientHandler.java:45)
	at software.amazon.awssdk.awscore.client.handler.AwsSyncClientHandler.execute(AwsSyncClientHandler.java:55)
	at software.amazon.awssdk.services.sts.DefaultStsClient.getCallerIdentity(DefaultStsClient.java:982)
	at software.amazon.awssdk.services.sts.StsClient.getCallerIdentity(StsClient.java:1611)
	at software.aws.toolkits.jetbrains.services.sts.StsResources$ACCOUNT$1.invoke(StsResources.kt:12)
	at software.aws.toolkits.jetbrains.services.sts.StsResources$ACCOUNT$1.invoke(StsResources.kt:10)
	at software.aws.toolkits.jetbrains.core.ClientBackedCachedResource.fetch(AwsResourceCache.kt:224)
	at software.aws.toolkits.jetbrains.core.DefaultAwsResourceCache.fetch(AwsResourceCache.kt:379)
	at software.aws.toolkits.jetbrains.core.DefaultAwsResourceCache.fetchIfNeeded(AwsResourceCache.kt:365)
	at software.aws.toolkits.jetbrains.core.DefaultAwsResourceCache.access$fetchIfNeeded(AwsResourceCache.kt:260)
	at software.aws.toolkits.jetbrains.core.DefaultAwsResourceCache$getCachedResource$1$result$1.apply(AwsResourceCache.kt:293)
	at software.aws.toolkits.jetbrains.core.DefaultAwsResourceCache$getCachedResource$1$result$1.apply(AwsResourceCache.kt:260)
	at java.base@11.0.7/java.util.concurrent.ConcurrentHashMap.compute(ConcurrentHashMap.java:1908)
	at software.aws.toolkits.jetbrains.core.DefaultAwsResourceCache$getCachedResource$1.run(AwsResourceCache.kt:292)
	at com.intellij.util.RunnableCallable.call(RunnableCallable.java:20)
	at com.intellij.util.RunnableCallable.call(RunnableCallable.java:11)
	at com.intellij.openapi.application.impl.ApplicationImpl$1.call(ApplicationImpl.java:268)
	at java.base@11.0.7/java.util.concurrent.FutureTask.run(FutureTask.java:264)
	at java.base@11.0.7/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128)
	at java.base@11.0.7/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628)
	at java.base@11.0.7/java.util.concurrent.Executors$PrivilegedThreadFactory$1$1.run(Executors.java:668)
	at java.base@11.0.7/java.util.concurrent.Executors$PrivilegedThreadFactory$1$1.run(Executors.java:665)
	at java.base@11.0.7/java.security.AccessController.doPrivileged(Native Method)
	at java.base@11.0.7/java.util.concurrent.Executors$PrivilegedThreadFactory$1.run(Executors.java:665)
	at java.base@11.0.7/java.lang.Thread.run(Thread.java:834)
@kiiadi kiiadi added the bug We can reproduce the issue and confirmed it is a bug. label Nov 21, 2020
@kiiadi
Copy link
Contributor Author

kiiadi commented Nov 21, 2020

Problem seems to occur because AwsConnectionManager.validate calls into the ResourceCache - and takes a lock while the SSO update is occurring.

The "sign into SSO" flow attempts to clear the resource cache - but can't get a lock.

@kiiadi
Copy link
Contributor Author

kiiadi commented Nov 21, 2020

As discussed with @abrooksv AwsResourceCache's underlying cache implementation should hold CompleteableFutures instead of actual values - that way inflight requests do not block other requests to the cache.

@abrooksv
Copy link
Contributor

abrooksv commented Nov 23, 2020 •
edited
Loading

Our CachingAsyncEvaluator is similar in this regard (stores Promise / AsyncPromise from JetBrains SDK vs CompletableFuture). May be able to reuse that internally.

If we do reuse CachingAsyncEvaluator, we should update its locking to use a ReadWrite lock instead of a synchronized(lock) block, or switch to ConcurrentHashMap.

@kiiadi
Copy link
Contributor Author

kiiadi commented Nov 30, 2020

I think we can follow a similar pattern - but we should use Deferred instead so that we're in better shape when we push this whole thing to be co-routine based.

@rli rli mentioned this issue Dec 17, 2020
Merged
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees
No one assigned
Labels
bug We can reproduce the issue and confirmed it is a bug.
Projects
None yet
Milestone
No milestone
Development

No branches or pull requests
2 participants
@kiiadi @abrooksv