Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

fix: refactor build tree algorithm #59

Merged
merged 1 commit into from
Aug 5, 2019
Merged

Conversation

talik077
Copy link
Contributor

@talik077 talik077 commented Aug 4, 2019

  • Ready for review
  • Follows CONTRIBUTING rules
  • Reviewed by Snyk internal team

What does this PR do?

running snyk test command in order to scan local NuGet dependencies, took approx ~2.5 minutes in the best case for a user. after digging a bit, It was easy to notice that the dep tree parsing algorithm was the root cause for this issue since it used recursive function inside a loop, which has high time complexity. this PR introduces a different approach for constructing a dep tree using the BFS algorithm. with this technique, processing time decreased by x25.

How should this be manually tested?

snyk-dev test --file="some.nugetdeps.json"

Any background context you want to provide?

zendesk user complain

Screenshots

Before (139120ms):
image
After (5478ms):
image

@CLAassistant
Copy link

CLAassistant commented Aug 4, 2019

CLA assistant check
All committers have signed the CLA.

@adrukh
Copy link
Contributor

adrukh commented Aug 5, 2019

:wow: 👏

Copy link
Contributor

@orsagie orsagie left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

A few comments. Thank you for doing this it's awesome!

lib/nuget-parser/dotnet-core-parser.ts Outdated Show resolved Hide resolved
lib/nuget-parser/dotnet-core-parser.ts Outdated Show resolved Hide resolved
lib/nuget-parser/dotnet-core-parser.ts Outdated Show resolved Hide resolved
lib/nuget-parser/dotnet-core-parser.ts Outdated Show resolved Hide resolved
lib/nuget-parser/dotnet-core-parser.ts Outdated Show resolved Hide resolved
lib/nuget-parser/dotnet-core-parser.ts Outdated Show resolved Hide resolved
@orsagie
Copy link
Contributor

orsagie commented Aug 5, 2019

can you also rename to commit to fix to trigger a new release 🙏

@talik077 talik077 force-pushed the chore/refactor-tree-build branch 2 times, most recently from a0b33bf to e325685 Compare August 5, 2019 10:22
@talik077 talik077 marked this pull request as ready for review August 5, 2019 10:22
@talik077 talik077 requested a review from a team as a code owner August 5, 2019 10:22
@ghost ghost requested a review from dkontorovskyy August 5, 2019 10:22
@talik077 talik077 changed the title chore: refactor build tree algo (POC) fix: refactor build tree algorithm Aug 5, 2019
let depResolvedName = '';
let originalDepKey = '';
function isFreqDep(packageName: string): boolean {
return packageName in freqDeps;
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This syntax actually works?! TIL

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@orsagie this works :) but you need to be really careful :) cos in freqDeps is an object this will also return true:
'toString' in freqDeps
'__proto__' in freqDeps
'constructor' in freqDeps etc

executing snyk test took very long time due to the parsing algorithm. this pr is a poc that introduce a different approach for constructing
dep tree using bfs algorithm. with this technique processing time decreased by x25
@orsagie orsagie merged commit d8046df into master Aug 5, 2019
@talik077 talik077 deleted the chore/refactor-tree-build branch August 5, 2019 12:25
@snyksec
Copy link

snyksec commented Aug 5, 2019

🎉 This PR is included in version 1.11.3 🎉

The release is available on:

Your semantic-release bot 📦🚀

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
Projects
None yet
Development

Successfully merging this pull request may close these issues.

6 participants