New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Improve debian license detection #2390 #2558

Merged

pombredanne merged 34 commits into develop from 2390-improve-debian-license-detection

Jul 2, 2021

Member

pombredanne commented Jun 15, 2021 •

edited by AyanSinhaMahapatra

Loading

Improves Debian License Detection by making the following enhancements:

Adds license_matches property to get all matches detected in a Structured file.
Better Filtering in unstructured debian copyright file for license intros
Adds license rules from copyright files for debin-slim-buster to improve license detection
Adds tests to verify 100% license detection accuracy on debian-slim copyright files (no unknown-license-reference)
Adds consistency check option for debian copyright files
Detect structured copyright fle more accuately
Add checks if substitution is carried out propery
Remove none and unknown copyrights
Add license detection from extra_data
Delete old functions and debian_licenses.txt
Adds fix for Failure: "Bad arguments: all arguments must be an Expression" scancode.io#219

See:

Tasks

Reviewed contribution guidelines
PR is descriptively titled 📑 and links the original issue above 🔗
Tests pass -- look for a green checkbox ✔️ a few minutes after opening your PR
Run tests locally to check for errors.
Commits are in uniquely-named feature branch and has no merge conflicts 📁

AyanSinhaMahapatra added 3 commits

June 9, 2021 21:55


          Modify tests for unstructured debian copyrights

6ba8f15

Uses filter_licenses flag to return all license matches and not
only unique ones in the case of unstructured copyright files tests,
which have with_details as True.

Signed-off-by: Ayan Sinha Mahapatra <ayansmahapatra@gmail.com>


          Add consistency check for debian copyright files

4c423e4

Add functions to check consistency of a debian copuright file, which
if enabled, raises an exception in the following cases:

- Unstructured File
- Has other paragraphs detected
- Has dupicate license paragraphs
- Has paragraphs with license but not license name
- All licenses in license paragraphs are not used
- License expressions are not parsable

Signed-off-by: Ayan Sinha Mahapatra <ayansmahapatra@gmail.com>


          Fix test file for appstream

bdc5354

Signed-off-by: Ayan Sinha Mahapatra <ayansmahapatra@gmail.com>

AyanSinhaMahapatra force-pushed the 2390-improve-debian-license-detection branch 2 times, most recently from 942f592 to b81cb96 Compare

June 17, 2021 07:29

AyanSinhaMahapatra added 2 commits

June 17, 2021 14:57


          Add rules to improve debian-slim license detection

035d643

Signed-off-by: Ayan Sinha Mahapatra <ayansmahapatra@gmail.com>


          Regenerate tests after adding rules

9ae88d5

Signed-off-by: Ayan Sinha Mahapatra <ayansmahapatra@gmail.com>

AyanSinhaMahapatra force-pushed the 2390-improve-debian-license-detection branch from b81cb96 to cb074fc Compare

June 23, 2021 11:29

AyanSinhaMahapatra added 15 commits

June 28, 2021 21:41


          Add filter for license intros to reduce unknowns

dc81787

Adds a filter for unstructured debian copyright file, where if
license intros are perfectly matched, they are discarded, as
in the context of a debian copyright file, the license texts/notices
are also there and not just the intro.

Signed-off-by: Ayan Sinha Mahapatra <ayansmahapatra@gmail.com>


          Add tests to check for unknowns and issues

204c768

Adds tests which fail if there is a unknown license
detection or a license detection issue with low match
coverage present in the test cases. Also traces the
detections in case of failures. Fixes some text expectations.

Signed-off-by: Ayan Sinha Mahapatra <ayansmahapatra@gmail.com>


          Add and modify rules for debian-slim copyrights

46a57b3

Add new rules and modify existing rules to get debian-slim license
detections correct.

Signed-off-by: Ayan Sinha Mahapatra <ayansmahapatra@gmail.com>


          Modify test expectations for debian-slim copyright

1e16986

Modify test expectations after license detection improvements in debian-slim
copyright files.

Signed-off-by: Ayan Sinha Mahapatra <ayansmahapatra@gmail.com>


          Improve debian copyright and license detection

c2e9ea2

Improves license detection by modifying unstructured license
intro detection with full coverage. Fixes matched_text bug in unknown
debian license by setting the lines. Removes unknown copyright detections.

Signed-off-by: Ayan Sinha Mahapatra <ayansmahapatra@gmail.com>


          Improve tests tracing for unknown matches

bae8f3c

Signed-off-by: Ayan Sinha Mahapatra <ayansmahapatra@gmail.com>


          Add license_matches property for strucutred files

b0900d0

Adds license_matches property to get the LicenseMatch objects
out of LicenseDetection objects directly as a property.

Signed-off-by: Ayan Sinha Mahapatra <ayansmahapatra@gmail.com>


          Detect structured debian copyright files better

a405884

Structured debian copyright files are deteected by the 'format: '
first line, and this adds more format links commonly encountered in
debian copyright files, thus detecting structured copyright files
better.

Signed-off-by: Ayan Sinha Mahapatra <ayansmahapatra@gmail.com>


          Add rules to remove unknowns in debian copyrights

df62c78

Adds rules to remove unknown-license-references in common debian
copyright files. Regenerates test files with removed unknowns.

Signed-off-by: Ayan Sinha Mahapatra <ayansmahapatra@gmail.com>


          Bump license-expression to latest 21.6.14 version

e87ca00

Signed-off-by: Ayan Sinha Mahapatra <ayansmahapatra@gmail.com>


          Use licensing.dedup to simplify expressions

751fc13

Use the licensing.dedup function from license-expression library
to simplify the licenses without losing any license-expression
specific information.

Signed-off-by: Ayan Sinha Mahapatra <ayansmahapatra@gmail.com>


          Regenerate tests with deduped license-expressions

aaa3299

Signed-off-by: Ayan Sinha Mahapatra <ayansmahapatra@gmail.com>


          Add primary license for structured debian files

6cb3aa2

For structured debian copyright files, return an attribute
with the primary license detected, and for unstructured files
return None.

Signed-off-by: Ayan Sinha Mahapatra <ayansmahapatra@gmail.com>


          Regenerate tests with primary license included

8c1aea9

Signed-off-by: Ayan Sinha Mahapatra <ayansmahapatra@gmail.com>


          Reproduce bug in get_license_expression

20a69da

Adds the debian copyright file which caused an exception, as
it says it's a structured debian copyright file, but doesn't
have structured paragraphs in it.

See - aboutcode-org/scancode.io#219

Signed-off-by: Ayan Sinha Mahapatra <ayansmahapatra@gmail.com>

AyanSinhaMahapatra force-pushed the 2390-improve-debian-license-detection branch from 8b050bc to 95568bc Compare

June 28, 2021 16:28

AyanSinhaMahapatra added 2 commits

June 28, 2021 22:02


          Fix bug in get_license_expression

935be1a

Fixes bug in get_license_expression by not adding a LicenseDetection
object when there are no license matches in an other paragraph,
in a structured debian copyright file.

See - aboutcode-org/scancode.io#219

Signed-off-by: Ayan Sinha Mahapatra <ayansmahapatra@gmail.com>


          Modify processing None at license_expressions

ffdec4e

Modify get_license_expression to not pass None values to
combine_expression, and also handle if all license detections
are None, by getting a expression from the license_matches,
or raising an Error if no license_matches.

See - aboutcode-org/scancode.io#219

Signed-off-by: Ayan Sinha Mahapatra <ayansmahapatra@gmail.com>

AyanSinhaMahapatra force-pushed the 2390-improve-debian-license-detection branch from 95568bc to ffdec4e Compare

June 28, 2021 16:40

AyanSinhaMahapatra added 3 commits

June 29, 2021 01:20


          Add more consistency checks and fix test failures

1f0a277

Signed-off-by: Ayan Sinha Mahapatra <ayansmahapatra@gmail.com>


          Modify filter_licenses for license expressions

c87f513

Remove filter_licenses flag from get_license_expression functions
as licensing.dedup() makes this function redundant. Also renames
filter_licenses to filter_duplicates.

Signed-off-by: Ayan Sinha Mahapatra <ayansmahapatra@gmail.com>


          Remove "none" copyrights

70b521f

Remove copyrights which starts with "none", like
`Copyright: none`. Regenerate tests to remove nones
from results.

Signed-off-by: Ayan Sinha Mahapatra <ayansmahapatra@gmail.com>

AyanSinhaMahapatra added 5 commits

July 1, 2021 22:55


          Update docstrings and functions

b2e2144

Signed-off-by: Ayan Sinha Mahapatra <ayansmahapatra@gmail.com>


          Modify reporting unknown matches

b905a82

Signed-off-by: Ayan Sinha Mahapatra <ayansmahapatra@gmail.com>


          Move consistency errors function into EDC

a021480

Move consistency error function into the EnhancedDebianCopyright class
and have the error textx added directly instead of a dict lookup.

Signed-off-by: Ayan Sinha Mahapatra <ayansmahapatra@gmail.com>


          Rename rules to have appropriate names

da46cfd

Rename the rules appropriately, with some modifications and additions.

Signed-off-by: Ayan Sinha Mahapatra <ayansmahapatra@gmail.com>


          Add check for debian expression substitution

467d23d

Adds a function to check whether the keys in debian expression can be substituted
successfully, and adds an UnknownMatch if there are inconsistencies.

Signed-off-by: Ayan Sinha Mahapatra <ayansmahapatra@gmail.com>

AyanSinhaMahapatra force-pushed the 2390-improve-debian-license-detection branch from 4624992 to 467d23d Compare

July 2, 2021 06:53

AyanSinhaMahapatra added 4 commits

July 2, 2021 14:31


          Add license detections from extra_data

0ae9349

Signed-off-by: Ayan Sinha Mahapatra <ayansmahapatra@gmail.com>


          Fix subs bug for inconsistent cases

1ef4c89

Adds a function to do license detection on the license name
if it is not in the seen keys. Regenerates test and adds rules.

Signed-off-by: Ayan Sinha Mahapatra <ayansmahapatra@gmail.com>


          Remove old debian_copyright parsing functions

51cb4f0

Removes old debian copyright parsing functions and also
removes debian_licenses.txt.

Signed-off-by: Ayan Sinha Mahapatra <ayansmahapatra@gmail.com>


          Fix failing tests

3ce3d85

Signed-off-by: Ayan Sinha Mahapatra <ayansmahapatra@gmail.com>

Member

AyanSinhaMahapatra commented Jul 2, 2021

@pombredanne Tests are all green now.

pombredanne commented

View reviewed changes

Member Author

pombredanne left a comment

LGTM! merging ... thank you ++ that's a big one.
Separately we will need a CHANGELOG entry 👍

pombredanne merged commit 3f7da81 into develop

pombredanne deleted the 2390-improve-debian-license-detection branch

July 2, 2021 16:59

AyanSinhaMahapatra mentioned this pull request

meta issue: Improve Debian package reported license aboutcode-org/scancode.io#103

Closed

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet