Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Migrate ruby to new importers #799

Merged
merged 1 commit into from
Feb 13, 2024
Merged

Migrate ruby to new importers #799

merged 1 commit into from
Feb 13, 2024

Conversation

ziadhany
Copy link
Collaborator

Reference: #796
Signed-off-by: Ziad ziadhany2016@gmail.com

@ziadhany
Copy link
Collaborator Author

some of ruby logs. Can you have a look at categorize_versions, I am not sure if am made the right implementation for it .

Invalid Rubygems Version 1.13.2 , Invalid constraints sequence: [VersionConstraint(comparator='>=', version=RubygemsVersion(string='5.2.6')), VersionConstraint(comparator='>=', version=RubygemsVersion(string='5.2.6.2')), VersionConstraint(comparator='<', version=RubygemsVersion(string='5.3'))]
Invalid Rubygems Version 1.13.1 , Invalid constraints sequence: [VersionConstraint(comparator='>=', version=RubygemsVersion(string='5.2.6')), VersionConstraint(comparator='>=', version=RubygemsVersion(string='5.2.6.2')), VersionConstraint(comparator='<', version=RubygemsVersion(string='5.3'))]
Invalid Rubygems Version 1.13.0 , Invalid constraints sequence: [VersionConstraint(comparator='>=', version=RubygemsVersion(string='5.2.6')), VersionConstraint(comparator='>=', version=RubygemsVersion(string='5.2.6.2')), VersionConstraint(comparator='<', version=RubygemsVersion(string='5.3'))]
Invalid Rubygems Version 1.12.5 , Invalid constraints sequence: [VersionConstraint(comparator='>=', version=RubygemsVersion(string='5.2.6')), VersionConstraint(comparator='>=', version=RubygemsVersion(string='5.2.6.2')), VersionConstraint(comparator='<', version=RubygemsVersion(string='5.3'))]
Invalid Rubygems Version 1.12.4 , Invalid constraints sequence: [VersionConstraint(comparator='>=', version=RubygemsVersion(string='5.2.6')), VersionConstraint(comparator='>=', version=RubygemsVersion(string='5.2.6.2')), VersionConstraint(comparator='<', version=RubygemsVersion(string='5.3'))]
Invalid Rubygems Version 1.12.3 , Invalid constraints sequence: [VersionConstraint(comparator='>=', version=RubygemsVersion(string='5.2.6')), VersionConstraint(comparator='>=', version=RubygemsVersion(string='5.2.6.2')), VersionConstraint(comparator='<', version=RubygemsVersion(string='5.3'))]
Invalid Rubygems Version 1.12.2 , Invalid constraints sequence: [VersionConstraint(comparator='>=', version=RubygemsVersion(string='5.2.6')), VersionConstraint(comparator='>=', version=RubygemsVersion(string='5.2.6.2')), VersionConstraint(comparator='<', version=RubygemsVersion(string='5.3'))]
Invalid Rubygems Version 1.12.1 , Invalid constraints sequence: [VersionConstraint(comparator='>=', version=RubygemsVersion(string='5.2.6')), VersionConstraint(comparator='>=', version=RubygemsVersion(string='5.2.6.2')), VersionConstraint(comparator='<', version=RubygemsVersion(string='5.3'))]
Invalid Rubygems Version 1.12.0 , Invalid constraints sequence: [VersionConstraint(comparator='>=', version=RubygemsVersion(string='5.2.6')), VersionConstraint(comparator='>=', version=RubygemsVersion(string='5.2.6.2')), VersionConstraint(comparator='<', version=RubygemsVersion(string='5.3'))]
Invalid Rubygems Version 1.11.2 , Invalid constraints sequence: [VersionConstraint(comparator='>=', version=RubygemsVersion(string='5.2.6')), VersionConstraint(comparator='>=', version=RubygemsVersion(string='5.2.6.2')), VersionConstraint(comparator='<', version=RubygemsVersion(string='5.3'))]
Invalid Rubygems Version 1.11.1 , Invalid constraints sequence: [VersionConstraint(comparator='>=', version=RubygemsVersion(string='5.2.6')), VersionConstraint(comparator='>=', version=RubygemsVersion(string='5.2.6.2')), VersionConstraint(comparator='<', version=RubygemsVersion(string='5.3'))]
Invalid Rubygems Version 1.11.0 , Invalid constraints sequence: [VersionConstraint(comparator='>=', version=RubygemsVersion(string='5.2.6')), VersionConstraint(comparator='>=', version=RubygemsVersion(string='5.2.6.2')), VersionConstraint(comparator='<', version=RubygemsVersion(string='5.3'))]
Invalid Rubygems Version 1.10.2 , Invalid constraints sequence: [VersionConstraint(comparator='>=', version=RubygemsVersion(string='5.2.6')), VersionConstraint(comparator='>=', version=RubygemsVersion(string='5.2.6.2')), VersionConstraint(comparator='<', version=RubygemsVersion(string='5.3'))]
Invalid Rubygems Version 1.10.1 , Invalid constraints sequence: [VersionConstraint(comparator='>=', version=RubygemsVersion(string='5.2.6')), VersionConstraint(comparator='>=', version=RubygemsVersion(string='5.2.6.2')), VersionConstraint(comparator='<', version=RubygemsVersion(string='5.3'))]
Invalid Rubygems Version 1.9.1 , Invalid constraints sequence: [VersionConstraint(comparator='>=', version=RubygemsVersion(string='5.2.6')), VersionConstraint(comparator='>=', version=RubygemsVersion(string='5.2.6.2')), VersionConstraint(comparator='<', version=RubygemsVersion(string='5.3'))]
Invalid Rubygems Version 1.9.0 , Invalid constraints sequence: [VersionConstraint(comparator='>=', version=RubygemsVersion(string='5.2.6')), VersionConstraint(comparator='>=', version=RubygemsVersion(string='5.2.6.2')), VersionConstraint(comparator='<', version=RubygemsVersion(string='5.3'))]
Invalid Rubygems Version 1.8.1 , Invalid constraints sequence: [VersionConstraint(comparator='>=', version=RubygemsVersion(string='5.2.6')), VersionConstraint(comparator='>=', version=RubygemsVersion(string='5.2.6.2')), VersionConstraint(comparator='<', version=RubygemsVersion(string='5.3'))]
Invalid Rubygems Version 1.8.0 , Invalid constraints sequence: [VersionConstraint(comparator='>=', version=RubygemsVersion(string='5.2.6')), VersionConstraint(comparator='>=', version=RubygemsVersion(string='5.2.6.2')), VersionConstraint(comparator='<', version=RubygemsVersion(string='5.3'))]
Invalid Rubygems Version 1.7.0 , Invalid constraints sequence: [VersionConstraint(comparator='>=', version=RubygemsVersion(string='5.2.6')), VersionConstraint(comparator='>=', version=RubygemsVersion(string='5.2.6.2')), VersionConstraint(comparator='<', version=RubygemsVersion(string='5.3'))]
Invalid Rubygems Version 1.6.0 , Invalid constraints sequence: [VersionConstraint(comparator='>=', version=RubygemsVersion(string='5.2.6')), VersionConstraint(comparator='>=', version=RubygemsVersion(string='5.2.6.2')), VersionConstraint(comparator='<', version=RubygemsVersion(string='5.3'))]
Invalid Rubygems Version 1.5.1 , Invalid constraints sequence: [VersionConstraint(comparator='>=', version=RubygemsVersion(string='5.2.6')), VersionConstraint(comparator='>=', version=RubygemsVersion(string='5.2.6.2')), VersionConstraint(comparator='<', version=RubygemsVersion(string='5.3'))]
Invalid Rubygems Version 1.5.0 , Invalid constraints sequence: [VersionConstraint(comparator='>=', version=RubygemsVersion(string='5.2.6')), VersionConstraint(comparator='>=', version=RubygemsVersion(string='5.2.6.2')), VersionConstraint(comparator='<', version=RubygemsVersion(string='5.3'))]
Invalid Rubygems Version 1.4.0 , Invalid constraints sequence: [VersionConstraint(comparator='>=', version=RubygemsVersion(string='5.2.6')), VersionConstraint(comparator='>=', version=RubygemsVersion(string='5.2.6.2')), VersionConstraint(comparator='<', version=RubygemsVersion(string='5.3'))]
Invalid Rubygems Version 1.3.1 , Invalid constraints sequence: [VersionConstraint(comparator='>=', version=RubygemsVersion(string='5.2.6')), VersionConstraint(comparator='>=', version=RubygemsVersion(string='5.2.6.2')), VersionConstraint(comparator='<', version=RubygemsVersion(string='5.3'))]
Invalid Rubygems Version 1.3.0 , Invalid constraints sequence: [VersionConstraint(comparator='>=', version=RubygemsVersion(string='5.2.6')), VersionConstraint(comparator='>=', version=RubygemsVersion(string='5.2.6.2')), VersionConstraint(comparator='<', version=RubygemsVersion(string='5.3'))]
Invalid Rubygems Version 1.2.0 , Invalid constraints sequence: [VersionConstraint(comparator='>=', version=RubygemsVersion(string='5.2.6')), VersionConstraint(comparator='>=', version=RubygemsVersion(string='5.2.6.2')), VersionConstraint(comparator='<', version=RubygemsVersion(string='5.3'))]
Invalid Rubygems Version 1.1.0 , Invalid constraints sequence: [VersionConstraint(comparator='>=', version=RubygemsVersion(string='5.2.6')), VersionConstraint(comparator='>=', version=RubygemsVersion(string='5.2.6.2')), VersionConstraint(comparator='<', version=RubygemsVersion(string='5.3'))]
Invalid Rubygems Version 1.0.1 , Invalid constraints sequence: [VersionConstraint(comparator='>=', version=RubygemsVersion(string='5.2.6')), VersionConstraint(comparator='>=', version=RubygemsVersion(string='5.2.6.2')), VersionConstraint(comparator='<', version=RubygemsVersion(string='5.3'))]
Invalid Rubygems Version 1.0.0 , Invalid constraints sequence: [VersionConstraint(comparator='>=', version=RubygemsVersion(string='5.2.6')), VersionConstraint(comparator='>=', version=RubygemsVersion(string='5.2.6.2')), VersionConstraint(comparator='<', version=RubygemsVersion(string='5.3'))]
Invalid Rubygems Version 0.9.5 , Invalid constraints sequence: [VersionConstraint(comparator='>=', version=RubygemsVersion(string='5.2.6')), VersionConstraint(comparator='>=', version=RubygemsVersion(string='5.2.6.2')), VersionConstraint(comparator='<', version=RubygemsVersion(string='5.3'))]
Invalid Rubygems Version 0.9.0 , Invalid constraints sequence: [VersionConstraint(comparator='>=', version=RubygemsVersion(string='5.2.6')), VersionConstraint(comparator='>=', version=RubygemsVersion(string='5.2.6.2')), VersionConstraint(comparator='<', version=RubygemsVersion(string='5.3'))]
Successfully imported data using vulnerabilities.importers.ruby.RubyImporter

@ziadhany ziadhany marked this pull request as ready for review July 21, 2022 07:23
@ziadhany ziadhany mentioned this pull request Sep 22, 2022
9 tasks
@TG1999
Copy link
Contributor

TG1999 commented Nov 2, 2022

@ziadhany what's the status on this ?

@ziadhany
Copy link
Collaborator Author

ziadhany commented Nov 2, 2022

@ziadhany what's the status on this ?

I will work on it this week. 👍

@TG1999
Copy link
Contributor

TG1999 commented Nov 18, 2022

Dependent on https://github.com/nexB/univers/pull/92/files

@TG1999
Copy link
Contributor

TG1999 commented Nov 23, 2022

@ziadhany invert functionality has been added to univers, please use https://pypi.org/project/univers/30.9.1/

@TG1999
Copy link
Contributor

TG1999 commented Dec 14, 2022

@ziadhany gentle ping

@TG1999 TG1999 modified the milestones: v32.0.0, v33.0.0 Jan 13, 2023
@ziadhany
Copy link
Collaborator Author

Ruby importer and improver logs :
ruby_imp_logs.zip

Copy link
Contributor

@TG1999 TG1999 left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@ziadhany Thanks++, some review comments for your consideration, please rebase your branch as well.

vulnerabilities/importers/ruby.py Outdated Show resolved Hide resolved
vulnerabilities/importers/ruby.py Show resolved Hide resolved
@ziadhany
Copy link
Collaborator Author

logs : importer--improver-ruby.zip

@DennisClark
Copy link
Member

almost there -- needs one more review

@TG1999
Copy link
Contributor

TG1999 commented Dec 6, 2023

@ziadhany please rebase your PR

@ziadhany
Copy link
Collaborator Author

ziadhany commented Dec 8, 2023

@ziadhany please rebase your PR

Done

record = load_yaml(path)
class RubyImporter(Importer):
license_url = "https://github.com/rubysec/ruby-advisory-db/blob/master/LICENSE.txt"
spdx_license_expression = "unknown"
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@ziadhany @pombredanne what should be done for this?

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@ziadhany let's get some stats, how much data we can ingest once we filter out the data that is associated with OSVDB ?

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

https://scancode-licensedb.aboutcode.org/public-domain-disclaimer.html ,

spdx_license_expression = "LicenseRef-scancode-public-domain-disclaimer"
notice = """
If you submit code or data to the ruby-advisory-db that is copyrighted by
yourself, upon submission you hereby agree to release it into the public
domain.

The data imported from the ruby-advisory-db have been filtered to exclude 
any non-public domain data from the data copyrighted by the Open 
Source Vulnerability Database (http://osvdb.org).

THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR
IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY,
FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE
AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER
LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM,
OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE
SOFTWARE.
"""

@TG1999 TG1999 marked this pull request as draft December 12, 2023 19:35
@TG1999 TG1999 changed the title Migrate ruby to new importers [WIP] Migrate ruby to new importers Dec 12, 2023
@TG1999 TG1999 modified the milestones: v33.0.0, v34.0.0 Jan 9, 2024
@ziadhany ziadhany requested a review from TG1999 January 14, 2024 19:39
@TG1999 TG1999 changed the title [WIP] Migrate ruby to new importers Migrate ruby to new importers Jan 30, 2024
@TG1999 TG1999 marked this pull request as ready for review January 30, 2024 17:05
Copy link
Contributor

@TG1999 TG1999 left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@ziadhany thanks++, minor nit for your consideration

vulnerabilities/importers/ruby.py Show resolved Hide resolved
Copy link
Contributor

@TG1999 TG1999 left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM! thanks please do the suggested changes and feel free to squash and merge this PR : )

Drop cvss_v2
Add ruby importer_name and Rebase
Resolve merge conflicts
Add advisory_url to ruby importer
Add a notice and the spdx_license_expression
Resolve merge conflict
Add a docstring to get_affected_packages
Add a unite test for get_affected_packages function
Remove unused variables
Fix sorted affected_package_merge
Add ruby importer and improver
Fix style test
Fix test
Rewrite affected_packages
Ruby initial config
Reference: aboutcode-org#796

Clean imported data after import process
Signed-off-by: Tushar Goel <tushar.goel.dav@gmail.com>

Fix sorted affected_package_merge
Refactor Ruby importer and improver
Add ruby importer and improver
Fix style test
Fix test
Rewrite affected_packages
Ruby initial config
Reference: aboutcode-org#796

Signed-off-by: ziadhany <ziadhany2016@gmail.com>
@TG1999 TG1999 merged commit bca15bb into aboutcode-org:main Feb 13, 2024
6 of 7 checks passed
@TG1999
Copy link
Contributor

TG1999 commented Feb 13, 2024

@ziadhany thanks for your effort and persistence : )

@ziadhany ziadhany deleted the ruby branch February 13, 2024 10:30
TG1999 pushed a commit to TG1999/vulnerablecode that referenced this pull request Jul 19, 2024
…rg#799)

Drop cvss_v2
Add ruby importer_name and Rebase
Resolve merge conflicts
Add advisory_url to ruby importer
Add a notice and the spdx_license_expression
Resolve merge conflict
Add a docstring to get_affected_packages
Add a unite test for get_affected_packages function
Remove unused variables
Fix sorted affected_package_merge
Add ruby importer and improver
Fix style test
Fix test
Rewrite affected_packages
Ruby initial config
Reference: aboutcode-org#796

Clean imported data after import process


Fix sorted affected_package_merge
Refactor Ruby importer and improver
Add ruby importer and improver
Fix style test
Fix test
Rewrite affected_packages
Ruby initial config
Reference: aboutcode-org#796

Signed-off-by: ziadhany <ziadhany2016@gmail.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
None yet
Development

Successfully merging this pull request may close these issues.

4 participants