Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Feature/add config params #71

Merged
merged 43 commits into from
Jan 13, 2025
Merged
Show file tree
Hide file tree
Changes from 36 commits
Commits
Show all changes
43 commits
Select commit Hold shift + click to select a range
a3cac85
changed each test to also check for warnings and specified which warn…
ant-u Apr 2, 2024
183d2b7
implemented change requests
ant-u Apr 8, 2024
473b6ab
Merge branch 'main' into feature/adapt_tests_for_warning
ant-u Apr 8, 2024
55e2230
linter issues
ant-u Apr 8, 2024
d21b4de
added location of check
ant-u Nov 11, 2024
6ca9be7
added check class
ant-u Nov 11, 2024
2da0559
added test for xml conformity
ant-u Nov 15, 2024
01d8ef7
added check for validating xml with scheme, output not yet processed …
ant-u Nov 15, 2024
7bf022e
created whole check out of validation function. introduced success an…
ant-u Nov 18, 2024
0a47e98
Merge branch 'main' into feature/adapt_tests_for_warning
ant-u Nov 21, 2024
03279cb
corrected double code
ant-u Nov 21, 2024
9dc4761
moved check to extra file
ant-u Nov 21, 2024
abf0c4d
fixed imports and whitespaces
ant-u Nov 21, 2024
8687856
changed if structure of _check method
ant-u Nov 22, 2024
3d57102
changed tests to all fullfill scheme version
ant-u Nov 25, 2024
dbef936
changed test data with source-file attribute to version 4
ant-u Nov 25, 2024
cd379e5
added case for package.xml version 4
ant-u Nov 28, 2024
9330eb2
removed todo
ant-u Nov 28, 2024
3ece023
adapted wrong test
ant-u Nov 28, 2024
bc4fabf
moved version checking and xml tree conversion to package.py
ant-u Nov 29, 2024
9716639
refactor of schema check
ant-u Dec 2, 2024
bab6691
fixed bug
ant-u Dec 2, 2024
b393418
added new args for config
ant-u Dec 5, 2024
e002704
added functionality for config options
ant-u Dec 5, 2024
7020bf3
added scheme 2 test
ant-u Dec 5, 2024
8baa2fd
added tests for all scheme versions
ant-u Dec 6, 2024
7b05817
added negative, violating tests for every scheme
ant-u Dec 6, 2024
e2ac79c
adapted tests to scan for schemacheck as well
ant-u Dec 6, 2024
f9c46d5
added online schema aquiration
ant-u Dec 9, 2024
396e73d
changed used xml validation files with online files, still need a fal…
ant-u Dec 9, 2024
7f7d0f5
changed readme due to updates on code
ant-u Dec 9, 2024
2cbfca5
changed readme
ant-u Dec 9, 2024
62ff892
changed content of arguments help
ant-u Dec 9, 2024
e3c6c89
added lazy schema request
ant-u Dec 9, 2024
19415cd
saving validation schema to reduce requests
ant-u Dec 9, 2024
125c185
deleted local shema files, rewrote schema_check cases and adapted sch…
ant-u Dec 9, 2024
711d56a
Resolved merge conflicts
ant-u Dec 12, 2024
fd38aff
implemented suggestions again
ant-u Dec 12, 2024
cf48bfd
implemented suggestions
ant-u Jan 9, 2025
56e7873
added fail case for no connection & adapted readme
ant-u Jan 13, 2025
0205104
linter issues
ant-u Jan 13, 2025
24350fe
fixing documentation bug
ant-u Jan 13, 2025
740e6d0
documentation fix
ant-u Jan 13, 2025
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
22 changes: 14 additions & 8 deletions README.md
Original file line number Diff line number Diff line change
Expand Up @@ -38,16 +38,18 @@ graph TD

This checks:

- [x] Is `package.xml` conform to it's schema?
[- SchemaCheck](src/ros_license_toolkit/license_checks/schema_check.py#L27)
- [x] Is any license defined in `package.xml`?
[- LicenseTagExistsCheck](src/ros_license_toolkit/checks.py#L90)
[- LicenseTagExistsCheck](src/ros_license_toolkit/license_checks/license_tag_exists_check.py#L24)
- [x] Has at most one license tag without a source-files declaration?
[- LicenseTagExistsCheck](src/ros_license_toolkit/checks.py#L90)
[- LicenseTagExistsCheck](src/ros_license_toolkit/license_checks/license_tag_exists_check.py#L24)
- [x] Do all licenses tags follow the SPDX standard?
[- LicenseTagIsInSpdxListCheck](src/ros_license_toolkit/checks.py#L104)
[- LicenseTagIsInSpdxListCheck](src/ros_license_toolkit/license_checks/license_tag_is_spdx.py#L24)
- [x] Are license texts available and correctly referenced for all declared licenses?
[- LicenseTextExistsCheck](src/ros_license_toolkit/checks.py#L123)
[- LicenseTextExistsCheck](src/ros_license_toolkit/license_checks/license_text_exists_check.py#L30)
- [x] Does the code contain licenses not declared in any license tags source-file attribute (source-files="src/something/**")?
[- LicensesInCodeCheck](src/ros_license_toolkit/checks.py#L182)
[- LicensesInCodeCheck](src/ros_license_toolkit/license_checks/license_in_code_check.py#L28)

## Usage

Expand All @@ -71,7 +73,7 @@ ros_license_toolkit my_ros_package

```bash
$ ros_license_toolkit -h
usage: ros_license_toolkit [-h] [-c] [-v] [-q] path
usage: ros_license_toolkit [-h] [-c] [-v] [-q] [-e] [-w] path

Checks ROS packages for correct license declaration.

Expand All @@ -84,6 +86,10 @@ options:
generate a copyright file
-v, --verbose enable verbose output
-q, --quiet disable most output
-e, --continue_on_error
treats all errors as warnings
-w, --warnings_as_error
treats all warnings as errors
```

Additionally, there is an option to ignore single files, folders and types of files.
Expand Down Expand Up @@ -140,9 +146,9 @@ In particular, the following things will have to be done:
- [x] Each LicenseTag should have SPDX id.
- [x] Single license tag without file attribute and single license text should match automatically.
- [x] Turn into github action.
- [ ] Evaluate runtime. If scancode-toolkit takes too long on too many cases, we will have to look for an alternative.
- [x] Evaluate runtime. If scancode-toolkit takes too long on too many cases, we will have to look for an alternative.
- [x] Error of `LicenseTagIsInSpdxListCheck` must be a warning
- [ ] Idea: Create pull requests for package maintainers automatically.
- [ ] Error of `LicenseTagIsInSpdxListCheck` must be a warning

## License

Expand Down
1 change: 1 addition & 0 deletions src/ros_license_toolkit/checks.py
Original file line number Diff line number Diff line change
Expand Up @@ -16,6 +16,7 @@

"""This module contains the checks for the linter."""


ant-u marked this conversation as resolved.
Show resolved Hide resolved
from enum import IntEnum

from ros_license_toolkit.package import Package, PackageException
Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -30,6 +30,7 @@ class LicensesInCodeCheck(Check):

def __init__(self: 'LicensesInCodeCheck'):
Check.__init__(self)
# self.package_xml_vers = package_xml_version
ant-u marked this conversation as resolved.
Show resolved Hide resolved
self.declared_licenses: Dict[str, LicenseTag] = {}
self.files_with_uncovered_licenses: Dict[str, List[str]] = {}
self.files_not_matched_by_any_license_tag: Dict[str, List[str]] = {}
Expand Down
78 changes: 78 additions & 0 deletions src/ros_license_toolkit/license_checks/schema_check.py
Original file line number Diff line number Diff line change
@@ -0,0 +1,78 @@
# Copyright (c) 2024 - for information on the respective copyright owner
ant-u marked this conversation as resolved.
Show resolved Hide resolved
# see the NOTICE file and/or the repository
# https://github.com/boschresearch/ros_license_toolkit

# Licensed under the Apache License, Version 2.0 (the "License");
# you may not use this file except in compliance with the License.
# You may obtain a copy of the License at

# http://www.apache.org/licenses/LICENSE-2.0

# Unless required by applicable law or agreed to in writing, software
# distributed under the License is distributed on an "AS IS" BASIS,
# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
# See the License for the specific language governing permissions and
# limitations under the License.

"""This Module contains SchemaCheck, which implements Check."""

from typing import Tuple
ant-u marked this conversation as resolved.
Show resolved Hide resolved

from lxml import etree

from ros_license_toolkit.checks import Check
from ros_license_toolkit.package import Package


class SchemaCheck(Check):
"""This checks the xml scheme and returns the version number."""
def __init__(self):
super().__init__()
self.accepted_versions = [1, 2, 3]
self.validation_schema: etree.XMLSchema = None
ant-u marked this conversation as resolved.
Show resolved Hide resolved

def _check(self, package: Package):
version: int = package.package_xml_format_ver
if version in self.accepted_versions:
status, message = self._validate(package)
if status:
self._success(f"Detected package.xml version {version}, "
"validation of scheme successful.")
else:
reason = f"package.xml contains errors: {message}"
self._failed(reason)
else:
# Temporary workaround for not implemented version 4
if version == 4:
reason = "couldn't check package.xml scheme. Version 4 is " +\
"not available right now"
self._warning(reason)
else:
reason = "package.xml does not contain correct package " +\
"format number. Please use a real version. " +\
"(e.g. <package format=\"3\">)"
self._failed(reason)

def _validate(self, package: Package) -> Tuple[bool, str]:
"""This is validating package.xml file from given package.
ant-u marked this conversation as resolved.
Show resolved Hide resolved
This can only validate for format version 1, 2 or 3. Every other
version WILL FAIL. Please check before calling, no check here. If
ant-u marked this conversation as resolved.
Show resolved Hide resolved
everything is correct, returns format number, else -1."""
version = package.package_xml_format_ver
message = ''
schema = self._get_validation_schema(version)
result = schema.validate(package.parsed_package_xml)
if not result:
message = schema.error_log.last_error
return result, message

def _get_validation_schema(self, version: int):
"""Return validation schema for version 1, 2 or 3. If called for other
version numbers, this WILL FAIL. Version is not checked again.
Only call with version 1, 2 or 3."""
if self.validation_schema is None:
address = 'http://download.ros.org/schema/' +\
f'package_format{version}.xsd'
schema = etree.parse(address)
self.validation_schema = etree.XMLSchema(schema)
return self.validation_schema
44 changes: 36 additions & 8 deletions src/ros_license_toolkit/main.py
Original file line number Diff line number Diff line change
Expand Up @@ -35,6 +35,7 @@
LicenseTagIsInSpdxListCheck
from ros_license_toolkit.license_checks.license_text_exists_check import \
LicenseTextExistsCheck
from ros_license_toolkit.license_checks.schema_check import SchemaCheck
from ros_license_toolkit.package import get_packages_in_path
from ros_license_toolkit.ui_elements import (FAILURE_STR, SUCCESS_STR,
WARNING_STR, Verbosity, major_sep,
Expand Down Expand Up @@ -66,6 +67,12 @@ def main(args: Optional[Sequence[str]] = None) -> int:
parser.add_argument(
'-q', '--quiet', dest='quiet', action='store_true',
default=False, help='disable most output')
parser.add_argument(
'-e', '--continue_on_error', action='store_true',
default=False, help='treats all errors as warnings')
ant-u marked this conversation as resolved.
Show resolved Hide resolved
parser.add_argument(
'-w', '--warnings_as_error', action='store_true',
default=False, help='treats all warnings as errors')
parsed_args = parser.parse_args(args)

# Determine the verbosity level
Expand Down Expand Up @@ -117,14 +124,7 @@ def main(args: Optional[Sequence[str]] = None) -> int:
rll_print(f'Execution time: {stop - start:.2f} seconds', Verbosity.QUIET)

# Print the overall results
if max(results_per_package.values()) == Status.SUCCESS:
rll_print(f"All packages:\n {SUCCESS_STR}", Verbosity.QUIET)
return os.EX_OK
if max(results_per_package.values()) == Status.WARNING:
rll_print(f"All packages:\n {WARNING_STR}", Verbosity.QUIET)
return os.EX_OK
rll_print(f"All packages:\n {FAILURE_STR}", Verbosity.QUIET)
return os.EX_DATAERR
return print_results(results_per_package, rll_print, parsed_args)


def generate_copyright_file(packages, rll_print):
Expand All @@ -143,6 +143,28 @@ def generate_copyright_file(packages, rll_print):
Verbosity.QUIET)


def print_results(result, rll_print, args):
"""Printing the result of package"""
if max(result.values()) == Status.SUCCESS:
rll_print(f"All packages:\n {SUCCESS_STR}", Verbosity.QUIET)
return os.EX_OK

if max(result.values()) == Status.WARNING:
if args.warnings_as_error:
rll_print(f"All packages:\n {FAILURE_STR} "
+ "(Treating warnings as failure)", Verbosity.QUIET)
return os.EX_DATAERR
rll_print(f"All packages:\n {WARNING_STR}", Verbosity.QUIET)
return os.EX_OK

if args.continue_on_error: # Error is Warning, still displayed red
rll_print(f"All packages:\n {WARNING_STR} "
+ "(Treating errors as warnings)", Verbosity.QUIET)
return os.EX_OK
rll_print(f"All packages:\n {FAILURE_STR}", Verbosity.QUIET)
return os.EX_DATAERR


def process_one_pkg(rll_print, package):
"""Perform checks on one package, print results and return them."""
results_per_package = {}
Expand All @@ -151,6 +173,11 @@ def process_one_pkg(rll_print, package):
rll_print(
f'git hash of ({package.repo.get_path()}):'
f' {package.repo.get_hash()}')
# check for scheme here, then insert scheme version in constructor
schema_check = SchemaCheck()
schema_check.check(package)
rll_print(schema_check)
rll_print(schema_check.verbose(), Verbosity.VERBOSE)
checks_to_perform = [
LicenseTagExistsCheck(),
LicenseTagIsInSpdxListCheck(),
Expand All @@ -163,6 +190,7 @@ def process_one_pkg(rll_print, package):
rll_print(check)
rll_print(check.verbose(), Verbosity.VERBOSE)

checks_to_perform.append(schema_check)
rll_print(minor_sep())
# Every check is successful, no warning
if max(check.status for check in checks_to_perform) == Status.SUCCESS:
Expand Down
36 changes: 36 additions & 0 deletions src/ros_license_toolkit/package.py
Original file line number Diff line number Diff line change
Expand Up @@ -22,6 +22,7 @@
import xml.etree.ElementTree as ET
from typing import Any, Dict, List, Optional

from lxml import etree
from rospkg import RosPack, list_by_path
from rospkg.common import PACKAGE_FILE
from scancode.api import get_licenses
Expand All @@ -32,6 +33,8 @@
from ros_license_toolkit.license_tag import LicenseTag
from ros_license_toolkit.repo import NotARepoError, Repo

INVALID = -1


class PackageException(Exception):
"""Exception raised when a package is invalid."""
Expand Down Expand Up @@ -83,6 +86,12 @@ def __init__(self, path: str, repo: Optional[Repo] = None):
# All ignored files and folders
self._ignored_content: List[str] = get_ignored_content(self.abspath)

# The package.xml file, parsed as etree
self._parsed_package_xml: etree = None

# The package.xml version as set in <package format="x">
self._package_xml_format_ver: int = 0

def _get_path_relative_to_pkg(self, path: str) -> str:
"""Get path relative to pkg root"""
return os.path.relpath(path, self.abspath)
Expand Down Expand Up @@ -230,6 +239,33 @@ def license_tags(self) -> Dict[str, LicenseTag]:

return self._license_tags

@property
def package_xml_format_ver(self) -> int:
"""Returns version of package.xml format as seen in
<package format="3">. If Version is not valid,
INVALID (-1) is returned."""
if self._package_xml_format_ver == 0:
root = self.parsed_package_xml.getroot()
if root.tag == 'package':
if 'format' in root.attrib:
version = root.attrib['format']
try:
self._package_xml_format_ver = int(version)
except ValueError:
self._package_xml_format_ver = INVALID
return self._package_xml_format_ver
self._package_xml_format_ver = INVALID
return self._package_xml_format_ver

@property
def parsed_package_xml(self) -> etree:
"""Returns the package.xml content parsed as etree."""
if self._parsed_package_xml is None:
path = self.abspath + "/package.xml"
assert os.path.exists(path), f'Path {path} does not exist.'
self._parsed_package_xml = etree.parse(path)
return self._parsed_package_xml

@property
def repo_url(self) -> Optional[str]:
"""Get the url of the repo this package is in if the package is in a
Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -2,5 +2,8 @@
<?xml-model href="http://download.ros.org/schema/package_format3.xsd" schematypens="http://www.w3.org/2001/XMLSchema"?>
<package format="3">
<name>test_pkg_deep</name>
<version>1.2.3</version>
<description>describing words</description>
<maintainer email="test@test.com"></maintainer>
<license file="LICENSE">BSD</license>
</package>
7 changes: 5 additions & 2 deletions test/_test_data/test_pkg_both_tags_not_spdx/package.xml
Original file line number Diff line number Diff line change
@@ -1,7 +1,10 @@
<?xml version="1.0"?>
<?xml-model href="http://download.ros.org/schema/package_format3.xsd" schematypens="http://www.w3.org/2001/XMLSchema"?>
<package format="3">
<?xml-model href="http://download.ros.org/schema/package_format4.xsd" schematypens="http://www.w3.org/2001/XMLSchema"?>
<package format="4">
<name>test_pkg_both_tags_not_spdx</name>
<version>1.2.3</version>
<description>describing words</description>
<maintainer email="test@test.com"></maintainer>
<license file="BSD.LICENSE">BSD</license>
<license file="MPL.LICENSE" source-files="code_with_MPL.py">MPL</license>
</package>
Original file line number Diff line number Diff line change
@@ -1,7 +1,10 @@
<?xml version="1.0"?>
<?xml-model href="http://download.ros.org/schema/package_format3.xsd" schematypens="http://www.w3.org/2001/XMLSchema"?>
<package format="3">
<?xml-model href="http://download.ros.org/schema/package_format4.xsd" schematypens="http://www.w3.org/2001/XMLSchema"?>
<package format="4">
<name>test_pkg_both_tags_not_spdx_one_file_own</name>
<version>3.2.1</version>
<description>very suggestive description</description>
<maintainer email="user@bosch.com"></maintainer>
<license file="LICENSE">Self Created License</license>
<license file="MPL.LICENSE" source-files="code_with_MPL.py">MPL</license>
</package>
3 changes: 3 additions & 0 deletions test/_test_data/test_pkg_code_has_no_license/package.xml
Original file line number Diff line number Diff line change
Expand Up @@ -2,5 +2,8 @@
<?xml-model href="http://download.ros.org/schema/package_format3.xsd" schematypens="http://www.w3.org/2001/XMLSchema"?>
<package format="3">
<name>test_pkg_code_has_no_license</name>
<version>1.2.3</version>
<description>describing words</description>
<maintainer email="test@test.com"></maintainer>
<license file="LICENSE">Apache-2.0</license>
</package>
7 changes: 5 additions & 2 deletions test/_test_data/test_pkg_has_code_disjoint/package.xml
Original file line number Diff line number Diff line change
@@ -1,7 +1,10 @@
<?xml version="1.0"?>
<?xml-model href="http://download.ros.org/schema/package_format3.xsd" schematypens="http://www.w3.org/2001/XMLSchema"?>
<package format="3">
<?xml-model href="http://download.ros.org/schema/package_format4.xsd" schematypens="http://www.w3.org/2001/XMLSchema"?>
<package format="4">
<name>test_pkg_has_code_disjoint</name>
<version>1.2.3</version>
<description>describing words</description>
<maintainer email="test@test.com"></maintainer>
<license file="LICENSE">BSD-3-Clause</license>
<license file="LICENSE_APACHE" source-files="src/** also_src/**">Apache-2.0</license>
</package>
Original file line number Diff line number Diff line change
Expand Up @@ -2,5 +2,8 @@
<?xml-model href="http://download.ros.org/schema/package_format3.xsd" schematypens="http://www.w3.org/2001/XMLSchema"?>
<package format="3">
<name>test_pkg_has_code_of_different_license</name>
<version>1.2.3</version>
<description>describing words</description>
<maintainer email="test@test.com"></maintainer>
<license file="LICENSE">BSD-3-Clause</license>
</package>
Original file line number Diff line number Diff line change
@@ -1,7 +1,10 @@
<?xml version="1.0"?>
<?xml-model href="http://download.ros.org/schema/package_format3.xsd" schematypens="http://www.w3.org/2001/XMLSchema"?>
<package format="3">
<?xml-model href="http://download.ros.org/schema/package_format4.xsd" schematypens="http://www.w3.org/2001/XMLSchema"?>
<package format="4">
<name>test_pkg_has_code_of_different_license_and_tag</name>
<version>1.2.3</version>
<description>describing words</description>
<maintainer email="test@test.com"></maintainer>
<license file="LICENSE">BSD-3-Clause</license>
<license file="LICENSE_LGPL_2_1" source-files="src/**">LGPL-2.1-only</license>
</package>
Original file line number Diff line number Diff line change
@@ -1,7 +1,10 @@
<?xml version="1.0"?>
<?xml-model href="http://download.ros.org/schema/package_format3.xsd" schematypens="http://www.w3.org/2001/XMLSchema"?>
<package format="3">
<?xml-model href="http://download.ros.org/schema/package_format4.xsd" schematypens="http://www.w3.org/2001/XMLSchema"?>
<package format="4">
<name>test_pkg_has_code_of_different_license_and_wrong_tag</name>
<version>3.2.1</version>
<description>very suggestive description</description>
<maintainer email="user@bosch.com"></maintainer>
<license file="LICENSE">BSD-3-Clause</license>
<license file="LICENSE_LGPL_2_1_or_later" source-files="not_src/**">LGPL-2.1-or-later</license>
</package>
Loading