Migrate ruby to new importers #799

ziadhany · 2022-07-18T09:42:53Z

Reference: #796
Signed-off-by: Ziad ziadhany2016@gmail.com

ziadhany · 2022-07-21T07:21:41Z

some of ruby logs. Can you have a look at categorize_versions, I am not sure if am made the right implementation for it .

Invalid Rubygems Version 1.13.2 , Invalid constraints sequence: [VersionConstraint(comparator='>=', version=RubygemsVersion(string='5.2.6')), VersionConstraint(comparator='>=', version=RubygemsVersion(string='5.2.6.2')), VersionConstraint(comparator='<', version=RubygemsVersion(string='5.3'))]
Invalid Rubygems Version 1.13.1 , Invalid constraints sequence: [VersionConstraint(comparator='>=', version=RubygemsVersion(string='5.2.6')), VersionConstraint(comparator='>=', version=RubygemsVersion(string='5.2.6.2')), VersionConstraint(comparator='<', version=RubygemsVersion(string='5.3'))]
Invalid Rubygems Version 1.13.0 , Invalid constraints sequence: [VersionConstraint(comparator='>=', version=RubygemsVersion(string='5.2.6')), VersionConstraint(comparator='>=', version=RubygemsVersion(string='5.2.6.2')), VersionConstraint(comparator='<', version=RubygemsVersion(string='5.3'))]
Invalid Rubygems Version 1.12.5 , Invalid constraints sequence: [VersionConstraint(comparator='>=', version=RubygemsVersion(string='5.2.6')), VersionConstraint(comparator='>=', version=RubygemsVersion(string='5.2.6.2')), VersionConstraint(comparator='<', version=RubygemsVersion(string='5.3'))]
Invalid Rubygems Version 1.12.4 , Invalid constraints sequence: [VersionConstraint(comparator='>=', version=RubygemsVersion(string='5.2.6')), VersionConstraint(comparator='>=', version=RubygemsVersion(string='5.2.6.2')), VersionConstraint(comparator='<', version=RubygemsVersion(string='5.3'))]
Invalid Rubygems Version 1.12.3 , Invalid constraints sequence: [VersionConstraint(comparator='>=', version=RubygemsVersion(string='5.2.6')), VersionConstraint(comparator='>=', version=RubygemsVersion(string='5.2.6.2')), VersionConstraint(comparator='<', version=RubygemsVersion(string='5.3'))]
Invalid Rubygems Version 1.12.2 , Invalid constraints sequence: [VersionConstraint(comparator='>=', version=RubygemsVersion(string='5.2.6')), VersionConstraint(comparator='>=', version=RubygemsVersion(string='5.2.6.2')), VersionConstraint(comparator='<', version=RubygemsVersion(string='5.3'))]
Invalid Rubygems Version 1.12.1 , Invalid constraints sequence: [VersionConstraint(comparator='>=', version=RubygemsVersion(string='5.2.6')), VersionConstraint(comparator='>=', version=RubygemsVersion(string='5.2.6.2')), VersionConstraint(comparator='<', version=RubygemsVersion(string='5.3'))]
Invalid Rubygems Version 1.12.0 , Invalid constraints sequence: [VersionConstraint(comparator='>=', version=RubygemsVersion(string='5.2.6')), VersionConstraint(comparator='>=', version=RubygemsVersion(string='5.2.6.2')), VersionConstraint(comparator='<', version=RubygemsVersion(string='5.3'))]
Invalid Rubygems Version 1.11.2 , Invalid constraints sequence: [VersionConstraint(comparator='>=', version=RubygemsVersion(string='5.2.6')), VersionConstraint(comparator='>=', version=RubygemsVersion(string='5.2.6.2')), VersionConstraint(comparator='<', version=RubygemsVersion(string='5.3'))]
Invalid Rubygems Version 1.11.1 , Invalid constraints sequence: [VersionConstraint(comparator='>=', version=RubygemsVersion(string='5.2.6')), VersionConstraint(comparator='>=', version=RubygemsVersion(string='5.2.6.2')), VersionConstraint(comparator='<', version=RubygemsVersion(string='5.3'))]
Invalid Rubygems Version 1.11.0 , Invalid constraints sequence: [VersionConstraint(comparator='>=', version=RubygemsVersion(string='5.2.6')), VersionConstraint(comparator='>=', version=RubygemsVersion(string='5.2.6.2')), VersionConstraint(comparator='<', version=RubygemsVersion(string='5.3'))]
Invalid Rubygems Version 1.10.2 , Invalid constraints sequence: [VersionConstraint(comparator='>=', version=RubygemsVersion(string='5.2.6')), VersionConstraint(comparator='>=', version=RubygemsVersion(string='5.2.6.2')), VersionConstraint(comparator='<', version=RubygemsVersion(string='5.3'))]
Invalid Rubygems Version 1.10.1 , Invalid constraints sequence: [VersionConstraint(comparator='>=', version=RubygemsVersion(string='5.2.6')), VersionConstraint(comparator='>=', version=RubygemsVersion(string='5.2.6.2')), VersionConstraint(comparator='<', version=RubygemsVersion(string='5.3'))]
Invalid Rubygems Version 1.9.1 , Invalid constraints sequence: [VersionConstraint(comparator='>=', version=RubygemsVersion(string='5.2.6')), VersionConstraint(comparator='>=', version=RubygemsVersion(string='5.2.6.2')), VersionConstraint(comparator='<', version=RubygemsVersion(string='5.3'))]
Invalid Rubygems Version 1.9.0 , Invalid constraints sequence: [VersionConstraint(comparator='>=', version=RubygemsVersion(string='5.2.6')), VersionConstraint(comparator='>=', version=RubygemsVersion(string='5.2.6.2')), VersionConstraint(comparator='<', version=RubygemsVersion(string='5.3'))]
Invalid Rubygems Version 1.8.1 , Invalid constraints sequence: [VersionConstraint(comparator='>=', version=RubygemsVersion(string='5.2.6')), VersionConstraint(comparator='>=', version=RubygemsVersion(string='5.2.6.2')), VersionConstraint(comparator='<', version=RubygemsVersion(string='5.3'))]
Invalid Rubygems Version 1.8.0 , Invalid constraints sequence: [VersionConstraint(comparator='>=', version=RubygemsVersion(string='5.2.6')), VersionConstraint(comparator='>=', version=RubygemsVersion(string='5.2.6.2')), VersionConstraint(comparator='<', version=RubygemsVersion(string='5.3'))]
Invalid Rubygems Version 1.7.0 , Invalid constraints sequence: [VersionConstraint(comparator='>=', version=RubygemsVersion(string='5.2.6')), VersionConstraint(comparator='>=', version=RubygemsVersion(string='5.2.6.2')), VersionConstraint(comparator='<', version=RubygemsVersion(string='5.3'))]
Invalid Rubygems Version 1.6.0 , Invalid constraints sequence: [VersionConstraint(comparator='>=', version=RubygemsVersion(string='5.2.6')), VersionConstraint(comparator='>=', version=RubygemsVersion(string='5.2.6.2')), VersionConstraint(comparator='<', version=RubygemsVersion(string='5.3'))]
Invalid Rubygems Version 1.5.1 , Invalid constraints sequence: [VersionConstraint(comparator='>=', version=RubygemsVersion(string='5.2.6')), VersionConstraint(comparator='>=', version=RubygemsVersion(string='5.2.6.2')), VersionConstraint(comparator='<', version=RubygemsVersion(string='5.3'))]
Invalid Rubygems Version 1.5.0 , Invalid constraints sequence: [VersionConstraint(comparator='>=', version=RubygemsVersion(string='5.2.6')), VersionConstraint(comparator='>=', version=RubygemsVersion(string='5.2.6.2')), VersionConstraint(comparator='<', version=RubygemsVersion(string='5.3'))]
Invalid Rubygems Version 1.4.0 , Invalid constraints sequence: [VersionConstraint(comparator='>=', version=RubygemsVersion(string='5.2.6')), VersionConstraint(comparator='>=', version=RubygemsVersion(string='5.2.6.2')), VersionConstraint(comparator='<', version=RubygemsVersion(string='5.3'))]
Invalid Rubygems Version 1.3.1 , Invalid constraints sequence: [VersionConstraint(comparator='>=', version=RubygemsVersion(string='5.2.6')), VersionConstraint(comparator='>=', version=RubygemsVersion(string='5.2.6.2')), VersionConstraint(comparator='<', version=RubygemsVersion(string='5.3'))]
Invalid Rubygems Version 1.3.0 , Invalid constraints sequence: [VersionConstraint(comparator='>=', version=RubygemsVersion(string='5.2.6')), VersionConstraint(comparator='>=', version=RubygemsVersion(string='5.2.6.2')), VersionConstraint(comparator='<', version=RubygemsVersion(string='5.3'))]
Invalid Rubygems Version 1.2.0 , Invalid constraints sequence: [VersionConstraint(comparator='>=', version=RubygemsVersion(string='5.2.6')), VersionConstraint(comparator='>=', version=RubygemsVersion(string='5.2.6.2')), VersionConstraint(comparator='<', version=RubygemsVersion(string='5.3'))]
Invalid Rubygems Version 1.1.0 , Invalid constraints sequence: [VersionConstraint(comparator='>=', version=RubygemsVersion(string='5.2.6')), VersionConstraint(comparator='>=', version=RubygemsVersion(string='5.2.6.2')), VersionConstraint(comparator='<', version=RubygemsVersion(string='5.3'))]
Invalid Rubygems Version 1.0.1 , Invalid constraints sequence: [VersionConstraint(comparator='>=', version=RubygemsVersion(string='5.2.6')), VersionConstraint(comparator='>=', version=RubygemsVersion(string='5.2.6.2')), VersionConstraint(comparator='<', version=RubygemsVersion(string='5.3'))]
Invalid Rubygems Version 1.0.0 , Invalid constraints sequence: [VersionConstraint(comparator='>=', version=RubygemsVersion(string='5.2.6')), VersionConstraint(comparator='>=', version=RubygemsVersion(string='5.2.6.2')), VersionConstraint(comparator='<', version=RubygemsVersion(string='5.3'))]
Invalid Rubygems Version 0.9.5 , Invalid constraints sequence: [VersionConstraint(comparator='>=', version=RubygemsVersion(string='5.2.6')), VersionConstraint(comparator='>=', version=RubygemsVersion(string='5.2.6.2')), VersionConstraint(comparator='<', version=RubygemsVersion(string='5.3'))]
Invalid Rubygems Version 0.9.0 , Invalid constraints sequence: [VersionConstraint(comparator='>=', version=RubygemsVersion(string='5.2.6')), VersionConstraint(comparator='>=', version=RubygemsVersion(string='5.2.6.2')), VersionConstraint(comparator='<', version=RubygemsVersion(string='5.3'))]
Successfully imported data using vulnerabilities.importers.ruby.RubyImporter

vulnerabilities/importers/ruby.py

TG1999 · 2022-11-02T14:49:21Z

@ziadhany what's the status on this ?

ziadhany · 2022-11-02T15:15:15Z

@ziadhany what's the status on this ?

I will work on it this week. 👍

TG1999 · 2022-11-18T15:53:45Z

Dependent on https://github.com/nexB/univers/pull/92/files

TG1999 · 2022-11-23T11:29:59Z

@ziadhany invert functionality has been added to univers, please use https://pypi.org/project/univers/30.9.1/

TG1999 · 2022-12-14T12:11:06Z

@ziadhany gentle ping

ziadhany · 2023-01-31T15:58:07Z

Ruby importer and improver logs :
ruby_imp_logs.zip

TG1999

@ziadhany Thanks++, some review comments for your consideration, please rebase your branch as well.

vulnerabilities/importers/ruby.py

ziadhany · 2023-08-28T22:12:19Z

logs : importer--improver-ruby.zip

DennisClark · 2023-11-14T16:11:58Z

almost there -- needs one more review

TG1999 · 2023-12-06T13:56:40Z

@ziadhany please rebase your PR

ziadhany · 2023-12-08T11:41:55Z

@ziadhany please rebase your PR

Done

TG1999 · 2023-12-12T08:39:53Z

vulnerabilities/importers/ruby.py

-        record = load_yaml(path)
+class RubyImporter(Importer):
+    license_url = "https://github.com/rubysec/ruby-advisory-db/blob/master/LICENSE.txt"
+    spdx_license_expression = "unknown"


@ziadhany @pombredanne what should be done for this?

@ziadhany let's get some stats, how much data we can ingest once we filter out the data that is associated with OSVDB ?

https://scancode-licensedb.aboutcode.org/public-domain-disclaimer.html ,

spdx_license_expression = "LicenseRef-scancode-public-domain-disclaimer" notice = """ If you submit code or data to the ruby-advisory-db that is copyrighted by yourself, upon submission you hereby agree to release it into the public domain. The data imported from the ruby-advisory-db have been filtered to exclude any non-public domain data from the data copyrighted by the Open Source Vulnerability Database (http://osvdb.org). THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY, FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM, OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE SOFTWARE. """

TG1999

@ziadhany thanks++, minor nit for your consideration

vulnerabilities/importers/ruby.py

TG1999

LGTM! thanks please do the suggested changes and feel free to squash and merge this PR : )

vulnerabilities/importers/ruby.py

Drop cvss_v2 Add ruby importer_name and Rebase Resolve merge conflicts Add advisory_url to ruby importer Add a notice and the spdx_license_expression Resolve merge conflict Add a docstring to get_affected_packages Add a unite test for get_affected_packages function Remove unused variables Fix sorted affected_package_merge Add ruby importer and improver Fix style test Fix test Rewrite affected_packages Ruby initial config Reference: aboutcode-org#796 Clean imported data after import process Signed-off-by: Tushar Goel <tushar.goel.dav@gmail.com> Fix sorted affected_package_merge Refactor Ruby importer and improver Add ruby importer and improver Fix style test Fix test Rewrite affected_packages Ruby initial config Reference: aboutcode-org#796 Signed-off-by: ziadhany <ziadhany2016@gmail.com>

TG1999 · 2024-02-13T07:42:39Z

@ziadhany thanks for your effort and persistence : )

…rg#799) Drop cvss_v2 Add ruby importer_name and Rebase Resolve merge conflicts Add advisory_url to ruby importer Add a notice and the spdx_license_expression Resolve merge conflict Add a docstring to get_affected_packages Add a unite test for get_affected_packages function Remove unused variables Fix sorted affected_package_merge Add ruby importer and improver Fix style test Fix test Rewrite affected_packages Ruby initial config Reference: aboutcode-org#796 Clean imported data after import process Fix sorted affected_package_merge Refactor Ruby importer and improver Add ruby importer and improver Fix style test Fix test Rewrite affected_packages Ruby initial config Reference: aboutcode-org#796 Signed-off-by: ziadhany <ziadhany2016@gmail.com>

ziadhany marked this pull request as ready for review July 21, 2022 07:23

TG1999 reviewed Sep 22, 2022

View reviewed changes

vulnerabilities/importers/ruby.py Outdated Show resolved Hide resolved

ziadhany mentioned this pull request Sep 22, 2022

Plan for merging pull requestes #925

Open

9 tasks

TG1999 reviewed Oct 18, 2022

View reviewed changes

vulnerabilities/importers/ruby.py Outdated Show resolved Hide resolved

ziadhany mentioned this pull request Nov 8, 2022

How can we pass useful data from importer to improver? #994

Open

TG1999 added the import-improver-migration label Nov 23, 2022

pombredanne added this to the v32.0.0 milestone Dec 8, 2022

TG1999 modified the milestones: v32.0.0, v33.0.0 Jan 13, 2023

ziadhany force-pushed the ruby branch 2 times, most recently from 4e90f8d to 2c50219 Compare January 22, 2023 11:54

ziadhany force-pushed the ruby branch 2 times, most recently from 2fa6915 to c6091c9 Compare January 31, 2023 15:56

ziadhany force-pushed the ruby branch from c6091c9 to 6539804 Compare May 16, 2023 14:56

TG1999 added the Priority: high label Aug 15, 2023

TG1999 reviewed Aug 22, 2023

View reviewed changes

vulnerabilities/importers/ruby.py Outdated Show resolved Hide resolved

vulnerabilities/importers/ruby.py Show resolved Hide resolved

ziadhany force-pushed the ruby branch 4 times, most recently from 832fb3b to 2ed17d6 Compare August 28, 2023 21:04

ziadhany requested a review from TG1999 August 28, 2023 22:13

TG1999 force-pushed the ruby branch from 2ed17d6 to 941f307 Compare September 11, 2023 10:10

ziadhany force-pushed the ruby branch 2 times, most recently from 03e0b80 to 469df9f Compare November 13, 2023 10:31

ziadhany force-pushed the ruby branch from 469df9f to d93ebf7 Compare December 8, 2023 11:38

TG1999 requested changes Dec 12, 2023

View reviewed changes

TG1999 marked this pull request as draft December 12, 2023 19:35

TG1999 changed the title ~~Migrate ruby to new importers~~ [WIP] Migrate ruby to new importers Dec 12, 2023

ziadhany force-pushed the ruby branch from d93ebf7 to 2eedabc Compare December 13, 2023 15:11

TG1999 modified the milestones: v33.0.0, v34.0.0 Jan 9, 2024

ziadhany force-pushed the ruby branch 2 times, most recently from 8678d54 to 850c5a7 Compare January 14, 2024 19:35

ziadhany requested a review from TG1999 January 14, 2024 19:39

TG1999 added Priority: medium and removed Priority: high labels Jan 30, 2024

TG1999 changed the title ~~[WIP] Migrate ruby to new importers~~ Migrate ruby to new importers Jan 30, 2024

TG1999 marked this pull request as ready for review January 30, 2024 17:05

TG1999 force-pushed the ruby branch from 850c5a7 to e0ced89 Compare February 5, 2024 10:21

TG1999 requested changes Feb 5, 2024

View reviewed changes

vulnerabilities/importers/ruby.py Show resolved Hide resolved

TG1999 approved these changes Feb 5, 2024

View reviewed changes

TG1999 reviewed Feb 5, 2024

View reviewed changes

vulnerabilities/importers/ruby.py Outdated Show resolved Hide resolved

ziadhany force-pushed the ruby branch from e0ced89 to 4f8c69a Compare February 6, 2024 15:50

TG1999 force-pushed the ruby branch from 4f8c69a to 48c1cc4 Compare February 13, 2024 07:38

TG1999 merged commit bca15bb into aboutcode-org:main Feb 13, 2024
6 of 7 checks passed

ziadhany deleted the ruby branch February 13, 2024 10:30

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Migrate ruby to new importers #799

Migrate ruby to new importers #799

ziadhany commented Jul 18, 2022

ziadhany commented Jul 21, 2022

TG1999 commented Nov 2, 2022

ziadhany commented Nov 2, 2022

TG1999 commented Nov 18, 2022

TG1999 commented Nov 23, 2022

TG1999 commented Dec 14, 2022

ziadhany commented Jan 31, 2023

TG1999 left a comment

ziadhany commented Aug 28, 2023

DennisClark commented Nov 14, 2023

TG1999 commented Dec 6, 2023

ziadhany commented Dec 8, 2023

TG1999 Dec 12, 2023

TG1999 Dec 12, 2023

TG1999 Dec 12, 2023

TG1999 left a comment

TG1999 left a comment

TG1999 commented Feb 13, 2024

Migrate ruby to new importers #799

Migrate ruby to new importers #799

Conversation

ziadhany commented Jul 18, 2022

ziadhany commented Jul 21, 2022

TG1999 commented Nov 2, 2022

ziadhany commented Nov 2, 2022

TG1999 commented Nov 18, 2022

TG1999 commented Nov 23, 2022

TG1999 commented Dec 14, 2022

ziadhany commented Jan 31, 2023

TG1999 left a comment

Choose a reason for hiding this comment

ziadhany commented Aug 28, 2023

DennisClark commented Nov 14, 2023

TG1999 commented Dec 6, 2023

ziadhany commented Dec 8, 2023

TG1999 Dec 12, 2023

Choose a reason for hiding this comment

TG1999 Dec 12, 2023

Choose a reason for hiding this comment

TG1999 Dec 12, 2023

Choose a reason for hiding this comment

TG1999 left a comment

Choose a reason for hiding this comment

TG1999 left a comment

Choose a reason for hiding this comment

TG1999 commented Feb 13, 2024