feat: impl fast semver search #495

killagu · 2023-06-06T04:54:37Z

Including SQL change.

close #134

app/core/entity/PackageVersion.ts

app/core/entity/PaddingSemVer.ts

app/core/entity/SqlRange.ts

app/core/event/BugVersionFixHandler.ts

socket-security · 2023-06-06T05:05:49Z

New and updated dependency changes detected. Learn more about Socket for GitHub ↗︎

Packages		Version	New capabilities	Transitives¹	Size	Publisher
npm-package-arg	🆕	10.1.0	environment	`+4`	63.1 kB	gar

https://docs.socket.dev ↩

killagu · 2023-06-06T05:06:45Z

app/core/service/BugVersionService.ts

+  private readonly bugVersionStore: BugVersionStore;
+
+  async getBugVersion(): Promise<BugVersion | undefined> {
+    // TODO performance problem, cache bugVersion and update with schedule


这里可能是一个热点，如果上线后实际存在需要额外增加缓存。

先加一个进程 LRU？缓存一下1分钟，性能应该就不是问题了。

killagu · 2023-06-06T05:07:26Z

app/core/service/FixNoPaddingVersionService.ts

+  @Inject()
+  private readonly logger: EggLogger;
+
+  async fixPaddingVersion(id?: number): Promise<void> {


需要在业务低峰阶段调用此方法来订正数据，此方法会造成单进程 CPU 持续打满。

app/core/service/PackageVersionService.ts

killagu · 2023-06-06T05:09:22Z

app/core/service/PackageVersionService.ts

+    }
+    let manifest;
+    if (isFullManifests) {
+      manifest = await this.distRepository.findPackageVersionManifest(pkgId, version);


存在一次额外的数据库访问，可以直接用过 fullname + version 的方式拼接出 nfs key。

app/core/entity/PaddingSemVer.ts

app/port/controller/admin/PaddingVersionController.ts

app/core/entity/PackageVersion.ts

app/core/entity/PaddingSemVer.ts

app/core/entity/SqlRange.ts

app/core/event/BugVersionFixHandler.ts

app/core/service/PackageVersionService.ts

killagu · 2023-06-06T05:17:23Z

#134
#452

codecov · 2023-06-06T05:17:28Z

Codecov Report

Merging #495 (26d78ae) into master (6624a36) will decrease coverage by 0.27%.
The diff coverage is 90.11%.

❗ Current head 26d78ae differs from pull request most recent head 4d0815b. Consider uploading reports for the commit 4d0815b to get more accurate results

@@            Coverage Diff             @@
##           master     #495      +/-   ##
==========================================
- Coverage   97.45%   97.18%   -0.27%     
==========================================
  Files         168      174       +6     
  Lines       15691    16182     +491     
  Branches     2018     2078      +60     
==========================================
+ Hits        15291    15727     +436     
- Misses        400      455      +55

Impacted Files	Coverage Δ
.../port/controller/admin/PaddingVersionController.ts	`48.83% <48.83%> (ø)`
app/core/service/FixNoPaddingVersionService.ts	`54.54% <54.54%> (ø)`
app/repository/PackageVersionRepository.ts	`87.95% <87.95%> (ø)`
app/core/service/BugVersionService.ts	`91.08% <91.66%> (+0.06%)`	⬆️
app/core/service/PackageVersionService.ts	`94.95% <94.95%> (ø)`
app/core/entity/PaddingSemVer.ts	`95.55% <95.55%> (ø)`
app/core/entity/SqlRange.ts	`97.53% <97.53%> (ø)`
app/core/service/PackageManagerService.ts	`98.93% <97.77%> (-0.02%)`	⬇️
app/core/entity/PackageVersion.ts	`100.00% <100.00%> (ø)`
app/core/entity/Token.ts	`100.00% <100.00%> (ø)`
... and 15 more

... and 1 file with indirect coverage changes

app/port/controller/package/ShowPackageVersionController.ts

app/core/entity/PackageVersion.ts

app/core/entity/PaddingSemVer.ts

app/core/entity/SqlRange.ts

app/core/event/BugVersionFixHandler.ts

sql/3.24.0.sql

sql/3.28.0.sql

app/core/entity/PaddingSemVer.ts

fengmk2 · 2023-06-06T09:33:29Z

app/core/entity/SqlRange.ts

+        $and: [
+          {
+            isPreRelease: {
+              $lte: 0,


$eq 0 性能会不会更好些？

fengmk2 · 2023-06-06T09:52:10Z

app/repository/model/PackageVersion.ts

+  @Attribute(DataTypes.BOOLEAN)
+  isPreRelease: boolean;
+
+  static beforeCreate(instance) {


学习了。

fengmk2 · 2023-06-06T09:53:35Z

sql/3.28.0.sql

+  ADD
+    COLUMN `is_pre_release` tinyint(4) DEFAULT NULL COMMENT '是否是先行版本',
+  ADD
+    KEY `idx_pkg_id_is_pre_release_padding_version` (`package_id`, `padding_version`, `is_pre_release`, `version`);


索引为什么会加上 version？

这样可以直接利用索引数据，避免回表，通过 explain sql 可以看到 using index 是最佳的效果。

fengmk2 · 2023-06-06T10:00:11Z

app/core/service/PackageManagerService.ts

    }
-    return { manifest, blockReason, pkg };
+    const fullname = getFullname(scope, name);
+    const result = npa(`${fullname}@${spec}`);


希望未来 npa 不会成为性能瓶颈

这个开销比较大的，好在前面有 cdn 缓存，不会高频使用。

fengmk2 · 2023-06-06T10:03:15Z

app/port/controller/PackageVersionFileController.ts

@@ -107,7 +107,7 @@ export class PackageVersionFileController extends AbstractController {
      }
      return files;
    }
-    const { manifest } = await this.packageManagerService.showPackageVersionManifest(scope, name, versionOrTag);
+    const { manifest } = await this.packageManagerService.showPackageVersionManifest(scope, name, versionOrTag, false, true);


加一下 range 的测试用例？

fengmk2

补充一下子 range 相关测试用例，其他没问题了。

fengmk2 · 2023-06-06T10:04:28Z

cc @hax

fengmk2 · 2023-06-06T10:11:31Z

@elrrrrrrr 也帮忙看看，今晚合并，然后在 r.cnpmjs.org 跑一下补数据，看看跑完历史数据要多久。

elrrrrrrr

+1

elrrrrrrr

+1

elrrrrrrr

+1

[skip ci] ## [3.28.0](v3.27.0...v3.28.0) (2023-06-06) ### Features * impl fast semver search ([#495](#495)) ([a7fd3a8](a7fd3a8))

fengmk2 · 2023-06-06T13:12:38Z

大表重建索引要很长时间

cc @elrrrrrrr 参考。

fengmk2 · 2023-06-06T13:22:17Z

大表重建索引要很长时间

cc @elrrrrrrr 参考。

[Success] ，耗时：950,047(ms)

> follow #495 after supporting spec, adjust the parameter validation rules 1. 🆕 Add `Spec` validation rule, validating the spec by npa 2. 🛠️ Upgrade versionOrTag to versionSpec to support semver expressions, such as `^2.x || > 3.x` --------- > follow #495 支持 spec 后，调整参数校验规则 1. 🆕 新增 `Sepc` 校验规则，使用 npa 拼接包名进行验证 2. 🛠️ versionOrTag 升级为 versionSpec 支持 semver 表达式，例如 `^2.x || > 3.x`

github-actions bot reviewed Jun 6, 2023

View reviewed changes

app/core/entity/PackageVersion.ts Show resolved Hide resolved

github-actions bot reviewed Jun 6, 2023

View reviewed changes

app/core/entity/PaddingSemVer.ts Show resolved Hide resolved

github-actions bot reviewed Jun 6, 2023

View reviewed changes

app/core/entity/SqlRange.ts Show resolved Hide resolved

github-actions bot reviewed Jun 6, 2023

View reviewed changes

app/core/event/BugVersionFixHandler.ts Show resolved Hide resolved

killagu force-pushed the feat/fast_semver branch from 9143e25 to 91ee7af Compare June 6, 2023 05:03

killagu commented Jun 6, 2023

View reviewed changes

app/core/service/PackageVersionService.ts Show resolved Hide resolved

killagu commented Jun 6, 2023

View reviewed changes

app/core/service/PackageVersionService.ts Show resolved Hide resolved

killagu commented Jun 6, 2023

View reviewed changes

elrrrrrrr reviewed Jun 6, 2023

View reviewed changes

app/core/entity/PaddingSemVer.ts Outdated Show resolved Hide resolved

github-actions bot reviewed Jun 6, 2023

View reviewed changes

app/port/controller/admin/PaddingVersionController.ts Show resolved Hide resolved

killagu force-pushed the feat/fast_semver branch from 26d78ae to 62cf2ca Compare June 6, 2023 05:13

github-actions bot reviewed Jun 6, 2023

View reviewed changes

app/core/entity/PackageVersion.ts Show resolved Hide resolved

github-actions bot reviewed Jun 6, 2023

View reviewed changes

app/core/entity/PaddingSemVer.ts Show resolved Hide resolved

github-actions bot reviewed Jun 6, 2023

View reviewed changes

app/core/entity/SqlRange.ts Show resolved Hide resolved

github-actions bot reviewed Jun 6, 2023

View reviewed changes

app/core/event/BugVersionFixHandler.ts Show resolved Hide resolved

elrrrrrrr reviewed Jun 6, 2023

View reviewed changes

app/core/service/PackageVersionService.ts Outdated Show resolved Hide resolved

elrrrrrrr reviewed Jun 6, 2023

View reviewed changes

app/port/controller/package/ShowPackageVersionController.ts Show resolved Hide resolved

fengmk2 added the enhancement New feature or request label Jun 6, 2023

This was linked to issues Jun 6, 2023

支持 range 接口查询版本 #134

Closed

Support list and view raw files #452

Closed

feat: impl fast semver search

bd51659

killagu force-pushed the feat/fast_semver branch from 62cf2ca to bd51659 Compare June 6, 2023 05:44

github-actions bot reviewed Jun 6, 2023

View reviewed changes

app/core/entity/PackageVersion.ts Show resolved Hide resolved

github-actions bot reviewed Jun 6, 2023

View reviewed changes

app/core/entity/PaddingSemVer.ts Show resolved Hide resolved

github-actions bot reviewed Jun 6, 2023

View reviewed changes

app/core/entity/SqlRange.ts Show resolved Hide resolved

github-actions bot reviewed Jun 6, 2023

View reviewed changes

app/core/event/BugVersionFixHandler.ts Show resolved Hide resolved

fengmk2 reviewed Jun 6, 2023

View reviewed changes

sql/3.24.0.sql Outdated Show resolved Hide resolved

fengmk2 added the sql include sql change label Jun 6, 2023

f

4d0815b

github-actions bot reviewed Jun 6, 2023

View reviewed changes

sql/3.28.0.sql Show resolved Hide resolved

fengmk2 reviewed Jun 6, 2023

View reviewed changes

app/core/entity/PaddingSemVer.ts Show resolved Hide resolved

fengmk2 reviewed Jun 6, 2023

View reviewed changes

app/core/entity/SqlRange.ts

$and: [

{

isPreRelease: {

$lte: 0,

Copy link

Member

fengmk2 Jun 6, 2023

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

$eq 0 性能会不会更好些？

fengmk2 reviewed Jun 6, 2023

View reviewed changes

fengmk2 approved these changes Jun 6, 2023

View reviewed changes

elrrrrrrr approved these changes Jun 6, 2023

View reviewed changes

fengmk2 merged commit a7fd3a8 into master Jun 6, 2023

fengmk2 deleted the feat/fast_semver branch June 6, 2023 12:56

fengmk2 pushed a commit that referenced this pull request Jun 6, 2023

Release 3.28.0

df8e114

[skip ci] ## [3.28.0](v3.27.0...v3.28.0) (2023-06-06) ### Features * impl fast semver search ([#495](#495)) ([a7fd3a8](a7fd3a8))

elrrrrrrr mentioned this pull request Jun 7, 2023

refactor: validate spec #500

Merged

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

feat: impl fast semver search #495

feat: impl fast semver search #495

killagu commented Jun 6, 2023 •

edited by atian25

Loading

socket-security bot commented Jun 6, 2023

killagu Jun 6, 2023

fengmk2 Jun 6, 2023 •

edited

Loading

killagu Jun 6, 2023

killagu Jun 6, 2023

killagu commented Jun 6, 2023

codecov bot commented Jun 6, 2023 •

edited

Loading

fengmk2 Jun 6, 2023

fengmk2 Jun 6, 2023

fengmk2 Jun 6, 2023

killagu Jun 16, 2023

fengmk2 Jun 6, 2023

killagu Jun 16, 2023

fengmk2 Jun 6, 2023

fengmk2 left a comment

fengmk2 commented Jun 6, 2023

fengmk2 commented Jun 6, 2023

elrrrrrrr left a comment

elrrrrrrr left a comment

elrrrrrrr left a comment

fengmk2 commented Jun 6, 2023

fengmk2 commented Jun 6, 2023

feat: impl fast semver search #495

feat: impl fast semver search #495

Conversation

killagu commented Jun 6, 2023 • edited by atian25 Loading

socket-security bot commented Jun 6, 2023

Footnotes

Choose a reason for hiding this comment

fengmk2 Jun 6, 2023 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

killagu commented Jun 6, 2023

codecov bot commented Jun 6, 2023 • edited Loading

Codecov Report

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

fengmk2 left a comment

Choose a reason for hiding this comment

fengmk2 commented Jun 6, 2023

fengmk2 commented Jun 6, 2023

elrrrrrrr left a comment

Choose a reason for hiding this comment

elrrrrrrr left a comment

Choose a reason for hiding this comment

elrrrrrrr left a comment

Choose a reason for hiding this comment

fengmk2 commented Jun 6, 2023

fengmk2 commented Jun 6, 2023

killagu commented Jun 6, 2023 •

edited by atian25

Loading

fengmk2 Jun 6, 2023 •

edited

Loading

codecov bot commented Jun 6, 2023 •

edited

Loading