Skip to content
This repository has been archived by the owner on Nov 29, 2023. It is now read-only.

feat: Improve Performance at Migration #1274

Closed
wants to merge 43 commits into from
Closed

Conversation

Aclrian
Copy link
Contributor

@Aclrian Aclrian commented Mar 14, 2023

Closes Safe-DS/Library-Analyzer#25.

Summary of Changes

Improved Performance of the migrate command:

  • add UnchangedDiffer that maps unchanged api-elements to its newer/older version.
  • rewritten weighted levenshtein distance
  • improve performance of StrictDiffer and its way to check if the parents of two api-elements are mapped to each other.

and some smaller changes:

  • remove documentation before calculating the similarity of two codes
  • move annotations that are migrated by ManyToManyMappings to the unsure annotation store

Testing Instructions

migrate a package with a lot of api-elements and measure the time of the migration.

@lars-reimann
Copy link
Member

lars-reimann commented Mar 14, 2023

🦙 MegaLinter status: ✅ SUCCESS

Descriptor Linter Files Fixed Errors Elapsed time
✅ JSON eslint-plugin-jsonc 3 0 0 16.84s
✅ JSON jsonlint 3 0 0.73s
✅ JSON npm-package-json-lint yes no 0.46s
✅ JSON prettier 3 0 0 14.99s
✅ JSON v8r 3 0 2.15s
✅ PYTHON black 19 0 0 2.55s
✅ PYTHON flake8 19 0 1.07s
✅ PYTHON isort 19 0 0 1.32s
✅ PYTHON mypy 19 0 3.36s
✅ PYTHON pylint 19 0 9.52s
✅ REPOSITORY git_diff yes no 1.18s

See detailed report in MegaLinter reports
Set VALIDATE_ALL_CODEBASE: true in mega-linter.yml to validate all sources, not only the diff

MegaLinter is graciously provided by OX Security

@codecov-commenter
Copy link

codecov-commenter commented Mar 14, 2023

Codecov Report

Merging #1274 (ef30197) into main (288d505) will increase coverage by 0.58%.
The diff coverage is 90.00%.

❗ Current head ef30197 differs from pull request most recent head f684705. Consider uploading reports for the commit f684705 to get more accurate results

@@            Coverage Diff             @@
##             main    #1274      +/-   ##
==========================================
+ Coverage   66.54%   67.12%   +0.58%     
==========================================
  Files         167      169       +2     
  Lines        7183     7310     +127     
  Branches      594      594              
==========================================
+ Hits         4780     4907     +127     
  Misses       2134     2134              
  Partials      269      269              
Components Coverage Δ
backend 81.66% <ø> (ø)
desktop 2.67% <ø> (ø)
gui 43.03% <ø> (ø)
package-parser 76.57% <90.00%> (+0.79%) ⬆️
Impacted Files Coverage Δ
...arser/processing/annotations/model/_annotations.py 97.66% <0.00%> (-0.05%) ⬇️
package-parser/package_parser/cli/_run_migrate.py 22.58% <16.66%> (-2.42%) ⬇️
...rser/package_parser/processing/api/model/_types.py 84.87% <69.23%> (-1.04%) ⬇️
...arser/processing/migration/model/_strict_differ.py 85.71% <78.57%> (-6.18%) ⬇️
...er/processing/migration/model/_unchanged_differ.py 86.53% <86.53%> (ø)
...ckage_parser/processing/migration/model/_differ.py 91.33% <93.44%> (-0.15%) ⬇️
...parser/package_parser/processing/api/model/_api.py 90.26% <100.00%> (+4.11%) ⬆️
...kage_parser/processing/api/model/_documentation.py 100.00% <100.00%> (+2.85%) ⬆️
...package_parser/processing/api/model/_parameters.py 94.28% <100.00%> (+2.85%) ⬆️
...er/package_parser/processing/migration/_migrate.py 65.70% <100.00%> (+0.33%) ⬆️
... and 3 more

... and 1 file with indirect coverage changes

@Aclrian Aclrian force-pushed the skip-unchanged-api-elements branch from b04c6f5 to cead560 Compare March 14, 2023 12:59
@Aclrian Aclrian changed the title feat: skip unchanged api elements feat: Improve Performance at Migration Mar 14, 2023
@Aclrian Aclrian force-pushed the skip-unchanged-api-elements branch from a7e0c5a to b51dd1f Compare March 14, 2023 20:48
@Aclrian
Copy link
Contributor Author

Aclrian commented Mar 15, 2023

grafik

@lars-reimann
Copy link
Member

lars-reimann commented Mar 19, 2023

@Aclrian Since this repo is about to get more busy very soon, I could not wait to finish #1258 any longer. The former package-parser is now called library-analyzer and resides in its own repo: https://github.com/Safe-DS/Library-Analyzer. Once you have the time, kindly

  1. Clone https://github.com/Safe-DS/Library-Analyzer
  2. Create a new branch there
  3. Copy the files you changed here over there
  • src contains the sources
  • tests contains the tests
  • package_parser is library_analyzer
  • Otherwise, the structure is identical so far
  1. Create a PR (with the same description as here)

Alternative approach, which might be a little faster: https://stackoverflow.com/a/2227571/21371004

Sorry for the disruption and inconvenience.

@Masara, @lukarade, @nvollroth Please do the same if you already created branches with changes to the former package-parser in this repo.

@Aclrian Aclrian force-pushed the skip-unchanged-api-elements branch from ef30197 to f684705 Compare March 19, 2023 17:25
@Aclrian
Copy link
Contributor Author

Aclrian commented Mar 19, 2023

will be completed in Safe-DS/Library-Analyzer#62.

@Aclrian Aclrian closed this Mar 19, 2023
@Aclrian Aclrian deleted the skip-unchanged-api-elements branch March 19, 2023 17:28
Sign up for free to subscribe to this conversation on GitHub. Already have an account? Sign in.
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

Migration is too slow
3 participants