fix: Nested validations optimization #1048

aaleksandrov · 2021-05-01T09:40:40Z

Description

This optimizes performance of all validations and closes #343.

For example in my company we have 50+ complex models so MetadatStorage.validationMetadatas is array of ~5000 elements, MetadatStorage.constraintMetadatas is an array of 2500 elements. Validator loops over these arrays multiple times, so it takes 5+ seconds to validate a batch of 550 objects.

Checklist

the pull request title describes what this PR does (not a vague title like Update index.md)
the pull request targets the default branch of the repository (develop)
the code follows the established code style of the repository
- npm run prettier:check passes
- npm run lint:check passes
tests are added for the changes I made (if any source code was modified)
documentation added or updated
I have run the project locally and verified that there are no errors

Didn't add any tests because my PR is pure refactoring of MetadatStorage, functionality is covered by tests already

Fixes

fixes #343

Running tests in my company's project normally

  V1BulkProductsSpec
    ✓ should ingest 550 products as bulk (5938ms)

  1 passing (6s)

Running tests in my company's project with this PR (42% performance improvement)

   V1BulkProductsSpec
    ✓ should ingest 550 products as bulk (3451ms)
    
  1 passing (3.5s)

)

)

….1 (typestack#991)

….0 (typestack#1618)

b26 · 2022-07-13T02:21:08Z

Bump

braaar · 2022-10-03T10:27:15Z

@aaleksandrov, could you rebase/merge this and get rid of the irrelevant changes in this PR?

aaleksandrov · 2022-10-22T21:50:29Z

Hi @braaar
I merged develop into my branch and resolved conflicts. However as you can see meanwhile develop was force-pushed so my PR still contains random commits. Do you want to me fix this (maybe creating a new PR would be the simplest way) or it's ok since changes are correct?

braaar · 2022-10-24T08:32:32Z

Hi @braaar I merged develop into my branch and resolved conflicts. However as you can see meanwhile develop was force-pushed so my PR still contains random commits. Do you want to me fix this (maybe creating a new PR would be the simplest way) or it's ok since changes are correct?

In my opinion, the diff is all that matters and squashing and merging is the better way to deal with PRs, but I'm not sure if that is the opinion of the maintainers.

src/metadata/MetadataStorage.ts

Clashsoft · 2022-12-02T17:52:07Z

src/metadata/MetadataStorage.ts

+    const filteredForInheritedMetadatasSearch = [];
+    for (const [key, value] of this.validationMetadatas.entries()) {
+      if (targetConstructor.prototype instanceof key) {
+        filteredForInheritedMetadatasSearch.push(...value);
+      }
+    }


Isn't it faster to loop over the prototype chain and look up the contructors in the map?

I think you are right, but this I did not fix in the new PR. I just merged it in its current state for the performance gains. If you are up for the task, please implement it in a new PR.

NoNameProvided · 2022-12-03T12:22:33Z

You have merged back the diverged branch in your PR, as a result, I cannot clean this, I have cherry-picked your commit and opened a new PR.

Superset by #1825

github-actions · 2023-01-03T00:12:14Z

This pull request has been automatically locked since there has not been any recent activity after it was closed. Please open a new issue for related bugs.

dependabot bot added 30 commits March 23, 2021 17:40

build(deps-dev): bump ts-jest from 26.5.3 to 26.5.4 (typestack#985)

26e1525

build(deps): bump libphonenumber-js from 1.9.12 to 1.9.14 (typestack#984

8cc1bd7

)

build(deps-dev): bump @typescript-eslint/eslint-plugin (typestack#981)

1505416

build(deps-dev): bump @types/jest from 26.0.20 to 26.0.21 (typestack#983

5d21e42

)

build(deps-dev): bump eslint from 7.21.0 to 7.22.0 (typestack#986)

c611de9

build(deps-dev): bump @typescript-eslint/parser from 4.16.1 to 4.19.0 (…

82e9e67

…typestack#982)

build(deps-dev): bump rollup from 2.42.3 to 2.42.4 (typestack#987)

f87ba6c

build(deps-dev): bump @types/jest from 26.0.21 to 26.0.22 (typestack#989

5b2910f

)

build(deps-dev): bump @types/node from 14.14.35 to 14.14.36 (typestac…

5d05e67

…k#990)

build(deps-dev): bump @rollup/plugin-node-resolve from 11.2.0 to 11.2…

25be325

….1 (typestack#991)

build(deps-dev): bump @types/node from 14.14.36 to 14.14.37 (typestac…

0dcb0f2

…k#992)

build(deps-dev): bump eslint from 7.22.0 to 7.23.0 (typestack#995)

2b5f354

build(deps-dev): bump rollup from 2.42.4 to 2.43.1 (typestack#994)

70fa7d2

build(deps-dev): bump @rollup/plugin-commonjs from 17.1.0 to 18.0.0 (t…

55ae9be

…ypestack#993)

build(deps-dev): bump rollup from 2.43.1 to 2.44.0 (typestack#998)

1bd7b0d

build(deps-dev): bump @typescript-eslint/parser from 4.19.0 to 4.20.0 (…

c1b5094

…typestack#999)

build(deps-dev): bump @typescript-eslint/eslint-plugin (typestack#1000)

dccbde8

build(deps): bump libphonenumber-js from 1.9.14 to 1.9.15 (typestack#…

615931e

…1001)

build(deps-dev): bump eslint-plugin-jest from 24.3.2 to 24.3.4 (types…

0ecbd0f

…tack#1004)

build(deps): bump libphonenumber-js from 1.9.15 to 1.9.16 (typestack#…

de0c673

…1006)

build(deps-dev): bump @typescript-eslint/parser from 4.20.0 to 4.21.0 (…

dac243b

…typestack#1007)

build(deps-dev): bump @typescript-eslint/eslint-plugin (typestack#1008)

1f612b5

build(deps-dev): bump typescript from 4.2.3 to 4.2.4 (typestack#1012)

3d668a6

build(deps-dev): bump rollup from 2.44.0 to 2.45.0 (typestack#1015)

88ae8d9

build(deps-dev): bump eslint-plugin-jest from 24.3.4 to 24.3.5 (types…

7a8b192

…tack#1016)

build(deps-dev): bump eslint from 7.23.0 to 7.24.0 (typestack#1017)

3f5b7e2

build(deps-dev): bump rollup from 2.45.0 to 2.45.1 (typestack#1018)

75f6757

build(deps-dev): bump @typescript-eslint/parser from 4.21.0 to 4.22.0 (…

42842d8

…typestack#1019)

build(deps-dev): bump rollup from 2.45.1 to 2.45.2 (typestack#1021)

29abff6

build(deps-dev): bump @typescript-eslint/eslint-plugin (typestack#1020)

4251428

dependabot bot added 17 commits March 16, 2022 09:08

build(deps-dev): bump prettier from 2.5.1 to 2.6.0 (typestack#1585)

7d8bd5f

build(deps): bump libphonenumber-js from 1.9.49 to 1.9.50 (typestack#…

303edfd

…1589)

build(deps-dev): bump lint-staged from 12.3.6 to 12.3.7 (typestack#1590)

fd67e22

build(deps-dev): bump eslint-plugin-jest from 26.1.1 to 26.1.2 (types…

397473b

…tack#1591)

build(deps-dev): bump @types/node from 17.0.21 to 17.0.22 (typestack#…

363a314

…1592)

build(deps-dev): bump @types/node from 17.0.22 to 17.0.23 (typestack#…

4b0114d

…1594)

build(deps): bump minimist from 1.2.5 to 1.2.6 (typestack#1595)

dc20354

build(deps-dev): bump typescript from 4.6.2 to 4.6.3 (typestack#1598)

0262804

build(deps-dev): bump prettier from 2.6.0 to 2.6.1 (typestack#1599)

e993a33

build(deps-dev): bump eslint-plugin-jest from 26.1.2 to 26.1.3 (types…

7098bf0

…tack#1600)

build(deps-dev): bump @types/validator from 13.7.1 to 13.7.2 (typesta…

8927baa

…ck#1602)

build(deps-dev): bump @rollup/plugin-commonjs from 21.0.2 to 21.0.3 (t…

b00838c

…ypestack#1604)

build(deps-dev): bump prettier from 2.6.1 to 2.6.2 (typestack#1607)

ba43ec3

build(deps-dev): bump eslint-plugin-jest from 26.1.3 to 26.1.4 (types…

7befb18

…tack#1613)

build(deps): bump libphonenumber-js from 1.9.50 to 1.9.51 (typestack#…

10eb3b2

…1615)

build(deps-dev): bump @rollup/plugin-node-resolve from 13.1.3 to 13.2…

1dd1012

….0 (typestack#1618)

build(deps-dev): bump @types/node from 17.0.23 to 17.0.24 (typestack#…

2ef8ff0

…1619)

braaar mentioned this pull request Oct 3, 2022

epic: overview of current fixes #1751

Closed

Merge branch 'develop' into nested-validations-optimization

7c23218

NoNameProvided force-pushed the develop branch from 2ef8ff0 to 0da8f89 Compare November 13, 2022 11:46

Clashsoft reviewed Dec 2, 2022

View reviewed changes

NoNameProvided added the status: superset by another Issue or task being tracked/handled in a different issue. label Dec 3, 2022

NoNameProvided closed this Dec 3, 2022

github-actions bot locked as resolved and limited conversation to collaborators Jan 3, 2023

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

fix: Nested validations optimization #1048

fix: Nested validations optimization #1048

aaleksandrov commented May 1, 2021 •

edited

Loading

b26 commented Jul 13, 2022

braaar commented Oct 3, 2022

aaleksandrov commented Oct 22, 2022 •

edited

Loading

braaar commented Oct 24, 2022

Clashsoft Dec 2, 2022

NoNameProvided Dec 3, 2022

NoNameProvided commented Dec 3, 2022 •

edited

Loading

github-actions bot commented Jan 3, 2023

fix: Nested validations optimization #1048

fix: Nested validations optimization #1048

Conversation

aaleksandrov commented May 1, 2021 • edited Loading

Description

Checklist

Fixes

b26 commented Jul 13, 2022

braaar commented Oct 3, 2022

aaleksandrov commented Oct 22, 2022 • edited Loading

braaar commented Oct 24, 2022

Clashsoft Dec 2, 2022

Choose a reason for hiding this comment

NoNameProvided Dec 3, 2022

Choose a reason for hiding this comment

NoNameProvided commented Dec 3, 2022 • edited Loading

github-actions bot commented Jan 3, 2023

aaleksandrov commented May 1, 2021 •

edited

Loading

aaleksandrov commented Oct 22, 2022 •

edited

Loading

NoNameProvided commented Dec 3, 2022 •

edited

Loading