Skip to content

feat: add dependency resolution for Python #748

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom
Merged
Parent: chore: merge changes to main
merged 8 commits into from
Jun 18, 2024
Merged

feat: add dependency resolution for Python #748

merged 8 commits into from
Jun 18, 2024

Conversation

@behnazh-w
Copy link
Member

@behnazh-w behnazh-w commented May 25, 2024 •
edited
Loading

This PR adds a new feature to automatically analyze the dependencies of a Python package by providing the path to environment where the package is installed:

run_macaron.sh analyze -purl pkg:pypi/django@5.0.6 --python-venv <path-to-the-virtual-environment>
  • To implement this feature, it was necessary to detect the target software component in an SBOM to find its dependencies. So, the metadata section of an SBOM is only required for multi-module Java projects and for other projects we should be able to resolve the dependencies even if metadata is not present in the SBOM.
  • To enhance the processing of the SBOMs, two new dependencies are added:
  • cyclonedx-bom
  • cyclonedx-python-lib

cyclonedx-bom is used to automatically generate the SBOM from a virtual environment and cyclonedx-python-lib is used to improve deserialization of SBOMs and processing the components.

  • Because the the DependencyAnalyzer class has CycloneDX-specific implementations, I have moved it to the cyclonedx.py module to avoid confusions.
  • The dependency analyzer of each build tool needs to implement a new function get_purl_from_cdx_component to customize how a PURL should be constructed.
  • The private Apache Maven integration test has been fixed to correctly match with the target software component.
  • I have added the dependency analysis tutorials on the website in the integration tests.

Closes #429

@oracle-contributor-agreement oracle-contributor-agreement bot added the OCA Verified All contributors have signed the Oracle Contributor Agreement. label May 25, 2024
@behnazh-w behnazh-w force-pushed the behnazh/python-sbom branch 6 times, most recently from cb230a1 to 9dd8f1a Compare May 29, 2024 21:07
@behnazh-w behnazh-w force-pushed the behnazh/python-sbom branch from 3e6e840 to 4e2c3ab Compare June 3, 2024 21:35
@behnazh-w behnazh-w marked this pull request as ready for review June 3, 2024 22:04
@behnazh-w behnazh-w requested a review from tromai as a code owner June 3, 2024 22:04
@behnazh-w behnazh-w requested a review from benmss June 3, 2024 22:04
tromai
tromai reviewed Jun 4, 2024
View reviewed changes
src/macaron/dependency_analyzer/cyclonedx_python.py
)
)

def remove_sboms(self, dir_path: str) -> bool:
Copy link
Contributor

@tromai tromai Jun 4, 2024 •
edited
Loading

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

It might not be relevant to this PR, but it looks to me that the return value of this method is not used anywhere.
The only place where this method is being used (as far as I know) -

macaron/src/macaron/dependency_analyzer/cyclonedx.py

Line 426 in 4e2c3ab

dep_analyzer.remove_sboms(main_ctx.component.repository.fs_path)
.
I guess we could consider remove the return value of this method if we are not using it.

For this method, is the reason we always return True is because cyclonedx-py only produces one file as we specified from the command line options, instead of producing bom.json in multiple directories like for mvn and gradle?

Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Yes, that's correct. Cyclonedx-py does not behave like maven and gradle cyclonedx plugins and does not produce bom files in the target repo.

Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I think having a comment here stating that there are no files to remove would be useful.

Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

See commit a300d7d

tromai
tromai reviewed Jun 4, 2024
View reviewed changes
tromai
tromai reviewed Jun 5, 2024
View reviewed changes
tromai
tromai reviewed Jun 5, 2024
View reviewed changes
benmss
benmss reviewed Jun 5, 2024
View reviewed changes
tromai
tromai reviewed Jun 5, 2024
View reviewed changes
tromai
tromai reviewed Jun 5, 2024
View reviewed changes
tromai
tromai reviewed Jun 5, 2024
View reviewed changes
tromai
tromai reviewed Jun 5, 2024
View reviewed changes
tromai
tromai reviewed Jun 6, 2024
View reviewed changes
tromai
tromai reviewed Jun 6, 2024
View reviewed changes
tromai
tromai reviewed Jun 6, 2024
View reviewed changes
tromai
tromai reviewed Jun 6, 2024
View reviewed changes
tromai
tromai reviewed Jun 6, 2024
View reviewed changes
tromai
tromai reviewed Jun 6, 2024
View reviewed changes
tromai
tromai reviewed Jun 6, 2024
View reviewed changes
tromai
tromai reviewed Jun 7, 2024
View reviewed changes
tromai
tromai reviewed Jun 7, 2024
View reviewed changes
tromai
tromai reviewed Jun 7, 2024
View reviewed changes
tromai
tromai reviewed Jun 7, 2024
View reviewed changes
tromai
tromai reviewed Jun 7, 2024
View reviewed changes
@behnazh-w behnazh-w force-pushed the behnazh/python-sbom branch 3 times, most recently from 50b5669 to 799e750 Compare June 13, 2024 23:04
behnazh-w
behnazh-w commented Jun 14, 2024
View reviewed changes
@behnazh-w
Copy link
Member Author

@tromai I have added the tutorial examples on the website that were related to dependency analysis to the integration tests: 569b511

@behnazh-w behnazh-w force-pushed the behnazh/python-sbom branch from de28baf to e368ee0 Compare June 14, 2024 10:35
tromai
tromai approved these changes Jun 17, 2024
View reviewed changes
Copy link
Contributor

@tromai tromai left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM. Thanks for the changes!

benmss
benmss reviewed Jun 17, 2024
View reviewed changes
benmss
benmss reviewed Jun 17, 2024
View reviewed changes
benmss
benmss reviewed Jun 17, 2024
View reviewed changes
@nathanwn nathanwn mentioned this pull request Jun 17, 2024
Merged
behnazh-w added 8 commits June 18, 2024 11:09
@behnazh-w
feat: add dependency resolution for Python
2f4e726 
Signed-off-by: behnazh-w <behnaz.hassanshahi@oracle.com>
@behnazh-w
chore: address review comments
59af6dc 
Signed-off-by: behnazh-w <behnaz.hassanshahi@oracle.com>
@behnazh-w
test: fix target component in deps integration tests
d6f7e49 
Signed-off-by: behnazh-w <behnaz.hassanshahi@oracle.com>
@behnazh-w
test: remove the unnecessary multibuild_test.json file
797a213 
Signed-off-by: behnazh-w <behnaz.hassanshahi@oracle.com>
@behnazh-w
chore(deps): add the extra validation dep for cyclonedx-python-lib
f2c34d4 
Signed-off-by: behnazh-w <behnaz.hassanshahi@oracle.com>
@behnazh-w
test: fix tutorial examples and add them as tests
6393f32 
Signed-off-by: behnazh-w <behnaz.hassanshahi@oracle.com>
@behnazh-w
test: fix the datalog policies and clarify cyclonedx deserialization …
a468221 
…test

Signed-off-by: behnazh-w <behnaz.hassanshahi@oracle.com>
@behnazh-w
chore: add a check for remove_sboms function and fix tests and docs
a300d7d 
Signed-off-by: behnazh-w <behnaz.hassanshahi@oracle.com>
@behnazh-w behnazh-w force-pushed the behnazh/python-sbom branch from e368ee0 to a300d7d Compare June 18, 2024 01:12
@behnazh-w behnazh-w merged commit 5522ec1 into staging Jun 18, 2024
This was referenced Jun 26, 2024
Closed
Closed
@behnazh-w behnazh-w deleted the behnazh/python-sbom branch September 23, 2024 03:58
art1f1c3R pushed a commit that referenced this pull request Nov 29, 2024
@behnazh-w
feat: add dependency resolution for Python (#748)
071224d 
This PR adds a new feature to automatically analyze the dependencies of a Python package by providing the path to environment where the package is installed:

`run_macaron.sh analyze -purl pkg:pypi/django@5.0.6 --python-venv <path-to-the-virtual-environment>`

To implement this feature, it was necessary to detect the target software component in an SBOM to find its dependencies. So, the metadata section of an SBOM is only required for multi-module Java projects and for other projects we should be able to resolve the dependencies even if metadata is not present in the SBOM.

To enhance the processing of the SBOMs, two new dependencies are added:
* cyclonedx-bom
* cyclonedx-python-lib

cyclonedx-bom is used to automatically generate the SBOM from a virtual environment and cyclonedx-python-lib is used to improve deserialization of SBOMs and processing the components.

Signed-off-by: behnazh-w <behnaz.hassanshahi@oracle.com>
@tromai tromai mentioned this pull request Dec 4, 2024
Merged
10 tasks
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Reviewers

@benmss benmss benmss approved these changes

+1 more reviewer

@tromai tromai tromai approved these changes

Reviewers whose approvals may not affect merge requirements

Assignees

No one assigned

Labels

OCA Verified All contributors have signed the Oracle Contributor Agreement.

Projects

None yet

Milestone

No milestone

Development

Successfully merging this pull request may close these issues.

3 participants

@behnazh-w @benmss @tromai