Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Fix deadlock when starting #12659

Merged
merged 1 commit into from
Jul 14, 2023
Merged

Conversation

liufeiyu1002
Copy link
Contributor

What is the purpose of the change

fix #12620 线程死锁问题
dubbo-deatlock drawio

  1. 在spring容器中启动时,如果开启了 @EnableScheduled 且在 @Scheduled 注解修饰的方法中调用了 dubbo 接口,
    如图中spring容器正常启动以后 发布了 ContextRefreshedEvent 即会 先调用 scheduleProcessor 再调用 dubboDeployApplicationListener (事件listener如果没有配置单独的线程池异步执行(大部分情况应该都没配置),则是同步顺序执行的) ,就可能会出现 异步任务调用 reference.get 获取了spring 单例锁(3.1.5 修改引入),dubboDeployApplicationListener 调用 deploy.start 获取了 dubbo 发布器锁, 此时如果 dubbo 存在 引用 懒加载的 spring bean 则会导致 死锁
    解决方式
    1.1 提升 dubboDeployApplicationListener 优先级先于 scheduleProcessor 处理。 (如果是listener 是配置了异步执行则无效)
    1.2 在 dubboDeployApplicationListener 调用 deploy 之前 先加 spring 单例锁 保证锁的获取顺序
  2. 容器启动过程中 在init 中调用了 dubbo 接口。
    2.1 同步调用 此时可能存在 部分服务发布、订阅非预期 ,
    2.2 异步调用 dubbo 启动过程中 可能存在部分服务发布、订阅非预期 或者同 @Scheduled 调度执行一样 造成死锁 可以通过deploy 加锁 进行排序加锁
  3. 其他情况的异步线程调用 dubbo 接口,调用初始化入口为 reference.get 和1 2 异步执行一样
  4. 主要解决思路就是 将 加锁 顺序话 先加 spring 单例锁 再加 dubbo deploy 锁,包装 dubbo start 过程中能够拿到 spring 锁避免造成死锁 。 图中 黄色 1 在 3.1.5版本中 加了单例锁,所以修复 在 黄色 2 处也加单例锁

Brief changelog

Verifying this change

Checklist

  • Make sure there is a GitHub_issue field for the change (usually before you start working on it). Trivial changes like typos do not require a GitHub issue. Your pull request should address just this issue, without pulling in other changes - one PR resolves one issue.
  • Each commit in the pull request should have a meaningful subject line and body.
  • Write a pull request description that is detailed enough to understand what the pull request does, how, and why.
  • Check if is necessary to patch to Dubbo 3 if you are work on Dubbo 2.7
  • Write necessary unit-test to verify your logic correction, more mock a little better when cross module dependency exist. If the new feature or significant change is committed, please remember to add sample in dubbo samples project.
  • Add some description to dubbo-website project if you are requesting to add a feature.
  • GitHub Actions works fine on your own branch.
  • If this contribution is large, please follow the Software Donation Guide.

@codecov-commenter
Copy link

Codecov Report

Merging #12659 (0995ac3) into 3.2 (dce7b87) will decrease coverage by 2.81%.
The diff coverage is n/a.

@@             Coverage Diff              @@
##                3.2   #12659      +/-   ##
============================================
- Coverage     69.70%   66.89%   -2.81%     
+ Complexity      318        2     -316     
============================================
  Files          3481     1763    -1718     
  Lines        165010    70859   -94151     
  Branches      27726    10158   -17568     
============================================
- Hits         115015    47403   -67612     
+ Misses        40010    18881   -21129     
+ Partials       9985     4575    -5410     

see 2060 files with indirect coverage changes

📣 We’re building smart automated test selection to slash your CI/CD build times. Learn more

@AlbumenJ
Copy link
Member

AlbumenJ commented Jul 3, 2023

还有一种场景是在ApplicationListener 的 init 中调用 Dubbo,这个时候会报 ReferenceBean not ready

@liufeiyu1002
Copy link
Contributor Author

@AlbumenJ 这个考虑合并不~

@@ -177,7 +182,7 @@ private void onContextClosedEvent(ContextClosedEvent event) {

@Override
public int getOrder() {
return LOWEST_PRECEDENCE;
return LOWEST_PRECEDENCE - 100;
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

How about change to HIGHEST_PRECEDENCE?

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

提高到最高优先级也是可以的

Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Go ahead.
BTW, pls add some test cases to verify this PR.

@liufeiyu1002 liufeiyu1002 force-pushed the fix/20230703_deadlock branch from c495c8d to fd7d7e0 Compare July 13, 2023 12:29
@AlbumenJ AlbumenJ changed the title fix #12620 修复spring dubbo 启动过程中死锁问题 Fix deadlock when starting Jul 13, 2023
@sonarqubecloud
Copy link

Kudos, SonarCloud Quality Gate passed!    Quality Gate passed

Bug A 0 Bugs
Vulnerability A 0 Vulnerabilities
Security Hotspot A 0 Security Hotspots
Code Smell A 1 Code Smell

100.0% 100.0% Coverage
0.0% 0.0% Duplication

@AlbumenJ AlbumenJ merged commit 1c52650 into apache:3.2 Jul 14, 2023
@wuwen5
Copy link
Contributor

wuwen5 commented Jul 25, 2023

@AlbumenJ 这个是否可以cherry-pick合入到3.1中 ?

@AlbumenJ
Copy link
Member

3.1.x 的版本目前不会发布 bugfix 版本了,只有 security fix 了

wuwen5 pushed a commit to wuwen5/dubbo that referenced this pull request Aug 8, 2023
wuwen5 added a commit to wuwen5/dubbo that referenced this pull request Aug 8, 2023
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

dubbo3.1.5版本BUG!!!,启动是出现线程死锁导致启动服务失败
4 participants