feat(python): add license support for `requirement.txt` files #6782

DmitriyLewen · 2024-05-27T10:23:56Z

Description

Add license support for requirement.txt files.

➜  trivy -q fs --scanners license ./pkg/fanal/analyzer/language/python/pip/testdata/happy 

requirements.txt (license)

Total: 1 (UNKNOWN: 1, LOW: 0, MEDIUM: 0, HIGH: 0, CRITICAL: 0)

┌─────────┬─────────────┬────────────────┬──────────┐
│ Package │   License   │ Classification │ Severity │
├─────────┼─────────────┼────────────────┼──────────┤
│ Flask   │ BSD License │ Non Standard   │ UNKNOWN  │
└─────────┴─────────────┴────────────────┴──────────┘

Related issues

Close feat(pip): license support #6703

Checklist

I've read the guidelines for contributing to this repository.
I've followed the conventions in the PR title.
I've added tests that prove my fix is effective or that my feature works.
I've updated the documentation with the relevant information (if needed).
I've added usage information (if the PR introduces new options)
I've included a "before" and "after" example to the description (if the PR is a user interface change).

pkg/fanal/analyzer/language/python/pip/pip.go

knqyf263 · 2024-05-29T19:42:36Z

pkg/fanal/analyzer/language/python/pip/pip.go

+	sort.Slice(pythonDirs, func(i, j int) bool {
+		majorVer1, minorVer1, found1 := strings.Cut(pythonDirs[i], ".")
+		majorVer2, minorVer2, found2 := strings.Cut(pythonDirs[j], ".")
+
+		// if one of dir doesn't contain minor version => sort as strings
+		if !found1 || !found2 {
+			return pythonDirs[i] < pythonDirs[j]
+		}
+
+		// Sort major versions as strings
+		// e.g. `python2.7` and `python3.10`
+		if majorVer1 != majorVer2 {
+			return majorVer1 < majorVer2
+		}
+
+		// Convert minor versions
+		ver1, err1 := strconv.Atoi(minorVer1)
+		ver2, err2 := strconv.Atoi(minorVer2)
+
+		// If we can't convert minor versions => sort as strings
+		if err1 != nil || err2 != nil {
+			return pythonDirs[i] < pythonDirs[j]
+		}
+
+		return ver1 < ver2
+	})


I believe go-version helps.
https://go.dev/play/p/Ytgevsntqr7

hm... i didn't think about this 👍
Updated in 226b3da

knqyf263 · 2024-05-29T19:45:44Z

pkg/fanal/analyzer/language/python/pip/pip.go

+	var pythonDirs []string
+	for _, entry := range entries {
+		if entry.IsDir() && strings.HasPrefix(entry.Name(), "python") {
+			pythonDirs = append(pythonDirs, entry.Name())
+		}
+	}


Suggested change

var pythonDirs []string

for _, entry := range entries {

if entry.IsDir() && strings.HasPrefix(entry.Name(), "python") {

pythonDirs = append(pythonDirs, entry.Name())

}

}

var pythonVers []version.Version

for _, entry := range entries {

if entry.IsDir() && strings.HasPrefix(entry.Name(), "python") {

ver := strings.TrimPrefix(entry.Name(), "python")

v, err := version.Parse(ver)

if err != nil {

log.Debug("...", log.Err(err))

continue

}

pythonVers = append(pythonVers, v)

}

Signed-off-by: knqyf263 <knqyf263@gmail.com>

Signed-off-by: knqyf263 <knqyf263@gmail.com> Co-authored-by: knqyf263 <knqyf263@gmail.com>

DmitriyLewen added 8 commits May 27, 2024 10:54

refactor(pip): use PostAnalyze

b9751cf

feat(pip): detect licenses from VIRTUAL_ENV

cad6440

feat(pip): add check python executable files

c546116

update tests

8739155

test: add Test_getPythonExecutablePath

87fb037

fix linter errors

a505ca7

docs(coverage): update python page

0e8b8e2

fix tests

18627a1

DmitriyLewen self-assigned this May 27, 2024

DmitriyLewen changed the title ~~feat(python): add lincense support for requirement.txt files~~ feat(python): add license support for requirement.txt files May 27, 2024

DmitriyLewen added 3 commits May 27, 2024 16:29

refactor

1e61514

refactor

7fe5333

fix linter error

a1e471c

DmitriyLewen marked this pull request as ready for review May 27, 2024 11:11

DmitriyLewen requested a review from knqyf263 as a code owner May 27, 2024 11:11

DmitriyLewen marked this pull request as draft May 27, 2024 11:20

DmitriyLewen added 2 commits May 27, 2024 17:29

fix windows tests

544adcf

fix linter error

a96e5f8

DmitriyLewen marked this pull request as ready for review May 27, 2024 12:07

knqyf263 reviewed May 28, 2024

View reviewed changes

pkg/fanal/analyzer/language/python/pip/pip.go Outdated Show resolved Hide resolved

knqyf263 reviewed May 28, 2024

View reviewed changes

pkg/fanal/analyzer/language/python/pip/pip.go Outdated Show resolved Hide resolved

DmitriyLewen added 2 commits May 28, 2024 15:16

refactor: parse only required METADATA files

beaeb6e

refactor: remove unused packageID()

9ee8066

knqyf263 reviewed May 28, 2024

View reviewed changes

pkg/fanal/analyzer/language/python/pip/pip.go Outdated Show resolved Hide resolved

refactor: sort python dirs according to major and minor versions

48617b2

knqyf263 reviewed May 29, 2024

View reviewed changes

knqyf263 mentioned this pull request May 30, 2024

release: v0.52.0 [main] #6809

Merged

DmitriyLewen and others added 3 commits May 30, 2024 10:59

refactor: use go-version to sort python dirs

226b3da

refactor: fix typo in comment

d337819

chore: lower-case error messages

ec15629

Signed-off-by: knqyf263 <knqyf263@gmail.com>

knqyf263 added 5 commits May 30, 2024 11:32

refactor: use fsutils

bd54938

Signed-off-by: knqyf263 <knqyf263@gmail.com>

refactor: return the existing variable

0298bb4

Signed-off-by: knqyf263 <knqyf263@gmail.com>

refactor: make return statements consistent

a58205a

Signed-off-by: knqyf263 <knqyf263@gmail.com>

refactor: some tweaks

96cd594

Signed-off-by: knqyf263 <knqyf263@gmail.com>

docs: capitalize

71d0d10

Signed-off-by: knqyf263 <knqyf263@gmail.com>

knqyf263 approved these changes May 30, 2024

View reviewed changes

knqyf263 enabled auto-merge May 30, 2024 07:41

knqyf263 added this pull request to the merge queue May 30, 2024

github-merge-queue bot pushed a commit that referenced this pull request May 30, 2024

feat(python): add license support for requirement.txt files (#6782)

e62eced

Signed-off-by: knqyf263 <knqyf263@gmail.com> Co-authored-by: knqyf263 <knqyf263@gmail.com>

github-merge-queue bot removed this pull request from the merge queue due to failed status checks May 30, 2024

knqyf263 added this pull request to the merge queue May 30, 2024

Merged via the queue into aquasecurity:main with commit 29615be May 30, 2024
17 checks passed

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

feat(python): add license support for `requirement.txt` files #6782

feat(python): add license support for `requirement.txt` files #6782

DmitriyLewen commented May 27, 2024 •

edited

Loading

knqyf263 May 29, 2024

DmitriyLewen May 30, 2024

knqyf263 May 29, 2024

DmitriyLewen May 30, 2024

-	var pythonDirs []string
-	for _, entry := range entries {
-		if entry.IsDir() && strings.HasPrefix(entry.Name(), "python") {
-			pythonDirs = append(pythonDirs, entry.Name())
-		}
-	}
+	var pythonVers []version.Version
+	for _, entry := range entries {
+		if entry.IsDir() && strings.HasPrefix(entry.Name(), "python") {
+			ver := strings.TrimPrefix(entry.Name(), "python")
+			v, err := version.Parse(ver)
+			if err != nil {
+				log.Debug("...", log.Err(err))
+				continue
+			}
+			pythonVers = append(pythonVers, v)
+		}

feat(python): add license support for requirement.txt files #6782

feat(python): add license support for requirement.txt files #6782

Conversation

DmitriyLewen commented May 27, 2024 • edited Loading

Description

Related issues

Checklist

knqyf263 May 29, 2024

Choose a reason for hiding this comment

DmitriyLewen May 30, 2024

Choose a reason for hiding this comment

knqyf263 May 29, 2024

Choose a reason for hiding this comment

DmitriyLewen May 30, 2024

Choose a reason for hiding this comment

feat(python): add license support for `requirement.txt` files #6782

feat(python): add license support for `requirement.txt` files #6782

DmitriyLewen commented May 27, 2024 •

edited

Loading