Refine the license policies tutorial #337

Signed-off-by: Thomas Druez <tdruez@nexb.com>
aboutcode-org · Nov 15, 2021 · ec9f49b · ec9f49b
1 parent 13d1ca6
commit ec9f49b
Show file tree

Hide file tree

Showing 3 changed files with 97 additions and 69 deletions.
diff --git a/docs/images/tutorial-license-policies-results.png b/docs/images/tutorial-license-policies-results.png
diff --git a/docs/scancodeio-settings.rst b/docs/scancodeio-settings.rst
@@ -121,6 +121,8 @@ of additional pipelines directories::
 
     SCANCODEIO_PIPELINES_DIRS=/var/scancodeio/pipelines/,/home/user/pipelines/
 
+.. _scancodeio_settings_policies_file:
+
 SCANCODEIO_POLICIES_FILE
 ------------------------
 
@@ -142,12 +144,16 @@ A valid policies file is required to enable compliance-related features.
 
 - Licenses are referenced by the ``license_key``.
 - A Policy is defined with ``label`` and ``compliance_alert``.
-- The ``compliance_alert`` accepts 3 values: "" for an empty string, warning, and error.
+- The ``compliance_alert`` accepts 3 values: '' for an empty string, warning, and error.
 
 .. note::
     When the policy feature is enabled, the ``compliance_alert`` values are
     displayed in the UI and returned in all downloadable results.
 
+.. tip::
+    Check out the :ref:`tutorial_license_policies` tutorial for in-depth coverage of
+    this feature.
+
 SCANCODEIO_REST_API_PAGE_SIZE
 -----------------------------
 

diff --git a/docs/tutorial_license_policies.rst b/docs/tutorial_license_policies.rst
@@ -3,27 +3,24 @@
 License Policies and Compliance Alerts
 ======================================
 
-In this tutorial, we'll introduce ScanCode.io's license policies and compliance
-alert system and use the results of a pipeline run to demonstrate an example
-license and compliance alert output.
+In this tutorial, we'll introduce ScanCode.io's **license policies** and **compliance
+alerts** system and use the **results of a pipeline run** to demonstrate an example
+of the license policies and compliance alerts output.
 
-.. note::
-    For example tutorials and pipeline runs, you can view our ``Tutorial``
-    section.
+As already mentioned, ScanCode.io automates the process of **Software Composition
+Analysis "SCA"** to identify existing open source components and their license
+compliance data in an application's codebase.
 
-As already mentioned, ScanCode.io automates the process of Software Composition
-Analysis—SCA—to identify existing open source components and their license
-compliance data in an application's codebase. ScanCode.io also gives users the
-ability to define a set of policies—license policies—to have their projects
-checked against with a ``ok``, ``missing``, ``warning``, and ``error`` compliance
-system.
+ScanCode.io also gives users the ability to define a set of **license policies** to
+have their projects checked against with a **compliance system**.
 
 Creating Policies Files
 -----------------------
 
-A valid policies file is required to enable compliance-related features. The
-policies file, by default ``policies.yml``, is a YAML file with a format
-similar to the following,
+A valid policies file is required to **enable compliance-related features**.
+
+The policies file, by default ``policies.yml``, is a **YAML file** with a structure
+similar to the following:
 
 .. code-block:: yaml
 
@@ -41,17 +38,22 @@ similar to the following,
 - In the above policies file, licenses are referenced by the ``license_key``,
   such as mit and gpl-3.0, which represents the ScanCode license key to match
   against detected licenses in the scan results.
-- A policy is defined with ``label`` and ``compliance_alert``.
-- The ``compliance_alert`` accepts 4 values: ``''`` for ``Approved License``,
-  ``warning`` for ``Restricted License``, ``error`` for ``Prohibited License``,
-  and ``missing`` for ``Missing License``.
+- A policy is defined with a ``label`` and a ``compliance_alert``.
+  The labels can be customized to your prefered wording.
+- The ``compliance_alert`` accepts 3 values:
+
+   - ``''`` (empty string)
+   - ``warning``
+   - ``error``
 
 Policies File Location
 ----------------------
 
-During the installation of ScanCode.io, an ``.env`` file is created at the root
-of ScanCode.io's codebase. You can configure the location of policies files using
-the ``SCANCODEIO_POLICIES_FILE`` setting in the ``.env`` file.
+By default, ScanCode.io will look for a ``policies.yml`` file at the root of its
+codebase.
+
+Alternatively, you can configure the location of policies files using the
+dedicated :ref:`scancodeio_settings_policies_file` setting in your ``.env`` file.
 
 .. tip::
     Check out our :ref:`scancodeio_settings` section for a comprehensive list of
@@ -62,62 +64,82 @@ How Does The Compliance Alert Work?
 
 The compliance system works by following a ``Precedence of Policies`` principal
 allowing the highest precedence policy to be applied in case of resources or
-packages with complex license expressions. This principal means a given resource
-with ``Prohibited`` AND ``Restricted`` AND ``Approved`` license expression would
-have an overall policy of ``Prohibited``.
+packages with complex license expressions:
 
-The code block below implements the previous principal by checking alerts for a
-given resource or package and having ``error`` as the highest precedence and
-``ok`` as the lowest.
+- **error > warning > missing > '' (empty string)**
 
-.. code-block:: python
+This principal means a given resource with ``error AND warning AND ''``
+license expression would have an overall compliance alert of ``error``.
 
-    ok = self.Compliance.OK
-    error = self.Compliance.ERROR
-    warning = self.Compliance.WARNING
-    missing = self.Compliance.MISSING
-
-     if error in alerts:
-         return error
-     elif warning in alerts:
-         return warning
-     elif missing in alerts:
-         return missing
-     return ok
+.. warning::
+    The ``missing`` compliance alert value is applied for licenses not present in the
+    policies file.
 
 Example Output
 --------------
 
-We followed the :ref:`tutorial_cli_analyze_docker_image` tutorial to generate
-scan results that show the ``""`` compliance alert for a given package.
+Create a ``policies.yml`` file in the root directory of your ScanCode.io codebase, with
+the following content:
+
+.. code-block:: yaml
+
+    license_policies:
+    -   license_key: mit
+        label: Approved License
+        compliance_alert: ''
+    -   license_key: gpl-3.0
+        label: Prohibited License
+        compliance_alert: error
+
+Run the following command to create a project and run the ``scan_codebase`` pipeline:
+
+.. code-block:: bash
+
+    $ scanpipe create-project cuckoo-filter-with-policies \
+        --input-url https://files.pythonhosted.org/packages/75/fc/f5b2e466d763dcc381d5127b73ffc265e8cdaf39ddafa422b7896e625432/cuckoo_filter-1.0.6.tar.gz \
+        --pipeline scan_codebase \
+        --execute
+
+Generate results:
+
+.. code-block:: bash
+
+    $ scanpipe output --project cuckoo-filter-with-policies
+
+The computed compliance alerts are now included in the results, available for each
+detected licenses, and computed at the codebase resource level, for example:
 
 .. code-block:: json
 
     {
-      "for_packages": [
-        "pkg:alpine/busybox@1.31.1-r9?arch=x86_64"
+      "for_packages": [],
+      "compliance_alert": "error",
+      "path": "cuckoo_filter-1.0.6.tar.gz-extract/cuckoo_filter-1.0.6/README.md",
+      "licenses": [
+        {
+          "key": "mit",
+          "name": "MIT License",
+          "policy": {
+            "label": "Recommended License",
+            "compliance_alert": ""
+          },
+        }
+        {
+          "key": "gpl-3.0",
+          "name": "GNU General Public License 3.0",
+          "policy": {
+            "label": "Prohibited License",
+            "compliance_alert": "error"
+        },
+      ]
+      "license_expressions": [
+        "mit OR gpl-3.0",
       ],
-      "compliance_alert": "",
-      "path": "30-alpine-nickolashkraus-staticbox-latest.tar-extract/5216338b40a7b96416b8b9858974bbe4acc3096ee60acbc4dfb1ee02aecceb10/bin/busybox",
-      "size": 841288,
-      "extra_data": {},
-      "copyrights": [],
-      "holders": [],
-      "authors": [],
-      "licenses": [],
-      "license_expressions": [],
-      "emails": [],
-      "urls": [],
-      "status": "system-package",
-      "type": "file",
-      "name": "busybox",
-      "extension": "",
-      "programming_language": "",
-      "mime_type": "application/x-pie-executable",
-      "file_type": "ELF 64-bit LSB pie executable, x86-64, version 1 (SYSV), dynamically linked, interpreter /lib/ld-musl-x86_64.so.1, stripped",
-      "is_binary": true,
-      "is_text": false,
-      "is_archive": false,
-      "is_key_file": false,
-      "is_media": false
+      "status": "scanned",
+      "name": "README",
+      "[...]": "[...]"
     }
+
+The compliance alert are also displayed in the Web UI:
+
+.. image:: images/tutorial-license-policies-results.png