Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

[Bug] The imbalance of the rate in XSOSI #1329

Closed
bifenglin opened this issue Jul 3, 2023 · 7 comments · Fixed by #1333
Closed

[Bug] The imbalance of the rate in XSOSI #1329

bifenglin opened this issue Jul 3, 2023 · 7 comments · Fixed by #1333
Labels
kind/bug Category issues or prs related to bug. waiting for author need issue author's feedback

Comments

@bifenglin
Copy link
Contributor

Current Behavior

As the #1327 shows that the rate of TieWay59 is 26.08% which is an imbalance mentioned in https://github.com/orgs/X-lab2017/discussions/284.
Do you know how to fix it?

Expected Behavior

Can you please provide me with the appropriate XSOXI rate for May?

Any Additional Comments?

No response

@open-digger-bot open-digger-bot bot added the kind/bug Category issues or prs related to bug. label Jul 3, 2023
@github-actions github-actions bot added the waiting for repliers need other's feedback label Jul 3, 2023
@frank-zsy
Copy link
Contributor

This is not a bug of OpenRank, actually we need macro-control to adjust the over high OpenRank of OSS101 and TieWay59. So you may just ask @will-ww to give you a proper result from management perspective.

@github-actions github-actions bot added waiting for author need issue author's feedback and removed waiting for repliers need other's feedback labels Jul 4, 2023
@will-ww
Copy link
Contributor

will-ww commented Jul 4, 2023

Yes. The simplest way to handle the OpenRank of TieWay59 is to remove him from XSOSI as TieWay59 is no longer active in any repos in X-lab, and we can pay for him alone for his contribution as the teaching assistant.

@will-ww
Copy link
Contributor

will-ww commented Jul 4, 2023

这个事情引发了一个问题。就是一个开发者(例如 TieWay59 )因为某项特殊的工作(例如助教批改作业)带来的 OpenRank 值提升。根据实际情况,我又不想把这个内容放到 XSOSI 激励计划中(因为助教补贴学校已经发过了),可以如何做呢?

而更复杂的情况是,这个高 OpenRank 还会一直影响到后续的几个月,如果 TieWay59 还会继续活跃在其他 X-lab 仓库下,能否手动的移除几个特定 PR 上的贡献带来的 OpenRank 值呢?

就是能否手工去掉下面三个协作单元带来的 OpenRank 值的提升:

上述三个 Issue 带来一些列的 PR 工作:https://github.com/X-lab2017/oss101/pulls?q=is%3Apr+is%3Aclosed

能够做到吗?:-) @frank-zsy

@frank-zsy
Copy link
Contributor

frank-zsy commented Jul 4, 2023

针对本次问题的说明:

可以在数据上做一些处理,但这个问题的本质在于 OSS101 仓库本身的全域 OpenRank 值过高,5 月 TieWay 在 OSS101 的贡献占比只有 22% 左右,但由于仓库本身的值过高导致了他的分成比例较高。

以上面三个 Issue 为例,第四讲和第七讲作业两个 Issue 的活跃都发生在 4 月,事实上 4 月 TieWay 在该仓库的分配比例并不高,只有 13% 左右,加上 4 月时 OSS101 仓库的全域 OpenRank 只有 30 左右,与 OpenDigger + Hypercrx 持平,所以总体符合预期。

而 5 月中,第九讲作业 Issue 本身的价值并不高,因为参与人数并不是非常多,所以即使剔除也不会有很大影响。

5 月 TieWay 贡献比例 22%,个人感觉相较而言比较符合预期,其中大部分是由于同学们修改一个 action 的 PR 所导致了( https://github.com/X-lab2017/oss101/blob/main/.github/workflows/add-reaction.yaml ),这是第九次作业中的一部分,但由于同学们并没有关联 Issue 和 PR 的习惯,这个的区分需要通过 PR 的描述或修改文件来人工核对。

因此我个人感觉的结论是:

  • 如果激励不发放到课程项目中,则可以直接去掉课程相关的数据即可。
  • TieWay 在其他项目中的活跃与 OSS101 并无直接关联,在其他项目的贡献度与 OSS101 的活跃完全无关。唯一的影响是其个人的全域 OpenRank 会偏高,目前为 4 左右,是相对可以接受的。

针对底层问题:

这个的本质问题与 2022.4 OpenLeaderboard 全球排行中 OpenRank Top2 的项目均为区块链空投项目是非常类似的,是由于短期内大量开发者同时参与同一个项目导致该项目的 OpenRank 值过高,这里面有两个因素导致了这个结果:1、OpenRank 算法本身带来的马太效应,度较高的节点结果会偏高。2、新人开发者的初始价值过高,因为所有开发者的初始值为 1,这导致新账号带来的波动会较大。

针对上述两个问题,可以的解决方案是:

  • 修改全域 OpenRank 的计算方法,将线性规范化方法改为 BiRank 的 L2 规范化,可以降低马太效应。
  • 降低新人的初始值,最极端的方案是所有开发者初始值均为 0,项目初始值为 1,所有的开发者价值均通过其参与的项目分得。

@will-ww
Copy link
Contributor

will-ww commented Jul 4, 2023

理解了,正好看到最新的 OpenLeaderboard(七月)榜单,里面的一个飙升的项目 OpenMMLabCamp 也是一个课程类项目,学生在仓库中交作业~

image

@bifenglin
Copy link
Contributor Author

那么有没有一个方法,可以看到去除OS101课程openrank值。

@github-actions github-actions bot added waiting for repliers need other's feedback and removed waiting for author need issue author's feedback labels Jul 5, 2023
@frank-zsy
Copy link
Contributor

image

image

上面是 5、6 月份数据去除 OSS101 仓库后的结果

@github-actions github-actions bot added waiting for author need issue author's feedback and removed waiting for repliers need other's feedback labels Jul 10, 2023
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
kind/bug Category issues or prs related to bug. waiting for author need issue author's feedback
Projects
None yet
Development

Successfully merging a pull request may close this issue.

3 participants