Add support to Exploits model #1562

ziadhany · 2024-08-20T12:44:15Z

issue #95

Collect exploit pointers:

exploitdb issue Collect exploit from exploitdb #1529
metasploit issue Add support to metasploit #1539

I think it's best to handle these issues in a single pull request, as they're all closely related to our exploit model.

ziadhany · 2024-09-02T21:42:21Z

Here’s the updated view of the exploits tab. Let me know if any further changes are needed. @DennisClark

CVE-ID: CVE-2020-14871

keshav-space

@ziadhany thanks, see few nits for your considerations.

vulnerabilities/migrations/0064_rename_resources_and_notes_exploit_notes.py

vulnerabilities/improvers/exploitdb.py

vulnerabilities/improvers/metasploit.py

vulnerabilities/improvers/vulnerability_kev.py

keshav-space

@ziadhany some nits in metasploit.py for your consideration. I was wondering if we should rename this pipeline to something like enhance_with_metasploit.py? similar to how we name our scancode.io pipelines https://github.com/aboutcode-org/scancode.io/tree/main/scanpipe/pipelines.

vulnerabilities/pipelines/metasploit.py

keshav-space

@ziadhany few more suggestions for your consideration. How about using enhance_with_exploitdb.py and enhance_with_kev.py as pipeline names?
Since there are model changes, we need to make adjustments here too

vulnerablecode/vulnerabilities/api_extension.py

Lines 106 to 110 in ed17dbd

    
           class V2ExploitSerializer(ModelSerializer): 
        
               class Meta: 
        
                   model = Kev 
        
                   fields = ("description", "required_action", "date_added", "due_date", "resources_and_notes")

.

vulnerabilities/pipelines/exploitdb.py

vulnerabilities/pipelines/vulnerability_kev.py

Set data_source as the header for the exploit table. Squash the migration files into a single file. Add test for exploit-db , metasploit Add a missing migration file Rename resources_and_notes to notes Fix Api test Refactor metasploit , exploitdb , kev improver Rename Kev tab to exploit tab Add support for exploitdb , metasploit, kev Signed-off-by: ziadhany <ziadhany2016@gmail.com>

Refactor the error handling logic in the code. Signed-off-by: ziadhany <ziadhany2016@gmail.com>

Address the exploit in the API extension. Signed-off-by: ziadhany <ziadhany2016@gmail.com>

keshav-space · 2024-09-18T07:22:13Z

@ziadhany please mark the addressed review as Resolve Conversation. Also please avoid force-pushing after the review, as this removes the reviews from the Files Changed tab. Instead we should squash these commits while merging.

ziadhany · 2024-09-20T22:57:06Z

@ziadhany please mark the addressed review as Resolve Conversation. Also please avoid force-pushing after the review, as this removes the reviews from the Files Changed tab. Instead we should squash these commits while merging.

Got it, I try to avoid force-pushing

Remove unused logging module Signed-off-by: ziadhany <ziadhany2016@gmail.com>

# Conflicts: # vulnerabilities/improvers/__init__.py # vulnerabilities/improvers/vulnerability_kev.py

Set data_source as the header for the exploit table. Squash the migration files into a single file. Add test for exploit-db , metasploit Add a missing migration file Rename resources_and_notes to notes Fix Api test Refactor metasploit , exploitdb , kev improver Rename Kev tab to exploit tab Add support for exploitdb , metasploit, kev Signed-off-by: ziadhany <ziadhany2016@gmail.com>

Refactor the error handling logic in the code. Signed-off-by: ziadhany <ziadhany2016@gmail.com>

Address the exploit in the API extension. Signed-off-by: ziadhany <ziadhany2016@gmail.com>

Remove unused logging module Signed-off-by: ziadhany <ziadhany2016@gmail.com>

Add pipeline_id for ( kev, metasploit, exploit-db ) Signed-off-by: ziadhany <ziadhany2016@gmail.com>

# Conflicts: # vulnerabilities/pipelines/enhance_with_exploitdb.py # vulnerabilities/pipelines/enhance_with_kev.py # vulnerabilities/pipelines/enhance_with_metasploit.py

Signed-off-by: ziadhany <ziadhany2016@gmail.com>

ziadhany · 2024-09-21T16:12:23Z

@keshav-space I've merged all the required changes and hope I didn't overlook anything. Could you please review it once more?

keshav-space · 2024-09-23T15:17:41Z

@keshav-space I've merged all the required changes and hope I didn't overlook anything. Could you please review it once more?

@ziadhany Thanks ++, this looks good already.

There are still some unresolved conversation mostly related to csv, please use csv.DictReader insted of cv.reader that way code is much more readable Add support to Exploits model #1562 (comment), Add support to Exploits model #1562 (comment), Add support to Exploits model #1562 (comment), Add support to Exploits model #1562 (comment), and Add support to Exploits model #1562 (comment).
Also, whenever we catch an exception we should include the entire stack trace in the log to make debugging a bit easier, similar to what we do here

vulnerablecode/vulnerabilities/pipelines/__init__.py

Line 132 in 094c2bf

f"Failed to import advisory: {advisory!r} with error {e!r}:\n{traceback_format_exc()}",

Add missing logs Handle cases of one exploit for multiple vulnerabilities. Signed-off-by: ziadhany <ziadhany2016@gmail.com>

keshav-space

Thanks @ziadhany, LGTM!

Please run these pipelines locally and paste the logs here.

ziadhany · 2024-09-24T12:56:16Z

Thanks, @keshav-space

The logs are too large to paste directly, so I've compressed them.

logs.zip

keshav-space

@ziadhany Thanks for the logs.

Found these in the log you provided.

INFO 2024-09-24 12:40:10.640 No vulnerability found for aliases []
INFO 2024-09-24 12:40:10.640 No vulnerability found for aliases []
INFO 2024-09-24 12:40:10.640 No vulnerability found for aliases []
INFO 2024-09-24 12:40:10.640 No vulnerability found for aliases []

INFO 2024-09-24 12:40:28.783 No vulnerability found for aliases ['']
INFO 2024-09-24 12:40:28.783 No vulnerability found for aliases ['']
INFO 2024-09-24 12:40:28.783 No vulnerability found for aliases ['']
INFO 2024-09-24 12:40:28.784 No vulnerability found for aliases ['']

IMO we should skip the empty aliases altogether.

vulnerabilities/pipelines/enhance_with_exploitdb.py

vulnerabilities/pipelines/enhance_with_kev.py

vulnerabilities/pipelines/enhance_with_metasploit.py

Remove empty vulnerability_kev.py file Signed-off-by: ziadhany <ziadhany2016@gmail.com>

ziadhany · 2024-09-24T14:31:52Z

@keshav-space Here are the updated logs:
logs.zip

keshav-space

@ziadhany looking good. I'll run this locally on my side and then merge it.

vulnerabilities/pipelines/enhance_with_metasploit.py

Signed-off-by: ziadhany <ziadhany2016@gmail.com>

Signed-off-by: Keshav Priyadarshi <git@keshav.space>

keshav-space

Thanks @ziadhany, this is working alright. I've added some improvements to the vulnerability details template.
Merging this now!

This was referenced Aug 20, 2024

Add support to metasploit #1539

Closed

Collect exploit from exploitdb #1529

Closed

ziadhany force-pushed the vul-exploit branch from add381f to f78f89e Compare September 2, 2024 21:25

ziadhany marked this pull request as ready for review September 2, 2024 21:25

ziadhany requested review from keshav-space and TG1999 September 3, 2024 15:05

keshav-space requested changes Sep 10, 2024

View reviewed changes

ziadhany force-pushed the vul-exploit branch from 67ee99b to 0446fff Compare September 11, 2024 21:20

ziadhany requested a review from keshav-space September 11, 2024 21:22

keshav-space requested changes Sep 16, 2024

View reviewed changes

keshav-space requested changes Sep 17, 2024

View reviewed changes

ziadhany added 3 commits September 17, 2024 17:53

Implement the appropriate LoopProgress progress bar.

c513713

Refactor the error handling logic in the code. Signed-off-by: ziadhany <ziadhany2016@gmail.com>

Resolve migration conflicts.

fbde42f

Address the exploit in the API extension. Signed-off-by: ziadhany <ziadhany2016@gmail.com>

ziadhany force-pushed the vul-exploit branch from dfc40d2 to fbde42f Compare September 17, 2024 15:00

ziadhany added 9 commits September 21, 2024 02:28

Add any missing logs message

7165be6

Remove unused logging module Signed-off-by: ziadhany <ziadhany2016@gmail.com>

Merge remote-tracking branch 'origin/main' into vul-exploit

79973f0

# Conflicts: # vulnerabilities/improvers/__init__.py # vulnerabilities/improvers/vulnerability_kev.py

Implement the appropriate LoopProgress progress bar.

c83facd

Refactor the error handling logic in the code. Signed-off-by: ziadhany <ziadhany2016@gmail.com>

Resolve migration conflicts.

fe55cf2

Address the exploit in the API extension. Signed-off-by: ziadhany <ziadhany2016@gmail.com>

Add any missing logs message

b7a31cd

Remove unused logging module Signed-off-by: ziadhany <ziadhany2016@gmail.com>

Fix migration conflict

14fc920

Add pipeline_id for ( kev, metasploit, exploit-db ) Signed-off-by: ziadhany <ziadhany2016@gmail.com>

Merge remote-tracking branch 'origin/vul-exploit' into vul-exploit

46acf56

# Conflicts: # vulnerabilities/pipelines/enhance_with_exploitdb.py # vulnerabilities/pipelines/enhance_with_kev.py # vulnerabilities/pipelines/enhance_with_metasploit.py

Remove unwanted migration file

8136e9a

Signed-off-by: ziadhany <ziadhany2016@gmail.com>

ziadhany requested a review from keshav-space September 21, 2024 16:14

Add log traceback for all the errors.

3b96511

Add missing logs Handle cases of one exploit for multiple vulnerabilities. Signed-off-by: ziadhany <ziadhany2016@gmail.com>

keshav-space approved these changes Sep 24, 2024

View reviewed changes

keshav-space requested changes Sep 24, 2024

View reviewed changes

vulnerabilities/pipelines/enhance_with_exploitdb.py Show resolved Hide resolved

vulnerabilities/pipelines/enhance_with_kev.py Show resolved Hide resolved

vulnerabilities/pipelines/enhance_with_metasploit.py Outdated Show resolved Hide resolved

Skip empty aliases

5b35552

Remove empty vulnerability_kev.py file Signed-off-by: ziadhany <ziadhany2016@gmail.com>

ziadhany requested a review from keshav-space September 24, 2024 14:33

keshav-space reviewed Sep 24, 2024

View reviewed changes

vulnerabilities/pipelines/enhance_with_metasploit.py Outdated Show resolved Hide resolved

ziadhany and others added 5 commits September 24, 2024 18:49

Replace references log with interesting_references

ca853fa

Signed-off-by: ziadhany <ziadhany2016@gmail.com>

Merge remote-tracking branch 'origin/main' into vul-exploit

807568b

Use proper labels in vulnerability details

1b78956

Signed-off-by: Keshav Priyadarshi <git@keshav.space>

Display Known/Unknown for ransomware campaign use

d9e00db

Signed-off-by: Keshav Priyadarshi <git@keshav.space>

Merge remote-tracking branch 'origin/main' into vul-exploit

b1580a0

keshav-space approved these changes Sep 30, 2024

View reviewed changes

keshav-space merged commit e273c67 into aboutcode-org:main Sep 30, 2024
5 checks passed

ziadhany mentioned this pull request Nov 5, 2024

Collect CVE tagged with CERTCC fork of Metasploit Framework #1455

Closed

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Add support to Exploits model #1562

Add support to Exploits model #1562

ziadhany commented Aug 20, 2024

ziadhany commented Sep 2, 2024 •

edited

Loading

keshav-space left a comment

keshav-space left a comment

keshav-space left a comment

keshav-space commented Sep 18, 2024

ziadhany commented Sep 20, 2024

ziadhany commented Sep 21, 2024

keshav-space commented Sep 23, 2024

keshav-space left a comment

ziadhany commented Sep 24, 2024

keshav-space left a comment

ziadhany commented Sep 24, 2024 •

edited

Loading

keshav-space left a comment

keshav-space left a comment

	class V2ExploitSerializer(ModelSerializer):
	class Meta:
	model = Kev
	fields = ("description", "required_action", "date_added", "due_date", "resources_and_notes")

Add support to Exploits model #1562

Add support to Exploits model #1562

Conversation

ziadhany commented Aug 20, 2024

ziadhany commented Sep 2, 2024 • edited Loading

keshav-space left a comment

Choose a reason for hiding this comment

keshav-space left a comment

Choose a reason for hiding this comment

keshav-space left a comment

Choose a reason for hiding this comment

keshav-space commented Sep 18, 2024

ziadhany commented Sep 20, 2024

ziadhany commented Sep 21, 2024

keshav-space commented Sep 23, 2024

keshav-space left a comment

Choose a reason for hiding this comment

ziadhany commented Sep 24, 2024

keshav-space left a comment

Choose a reason for hiding this comment

ziadhany commented Sep 24, 2024 • edited Loading

keshav-space left a comment

Choose a reason for hiding this comment

keshav-space left a comment

Choose a reason for hiding this comment

ziadhany commented Sep 2, 2024 •

edited

Loading

ziadhany commented Sep 24, 2024 •

edited

Loading