v32.3.0

What's Changed

• Deploy to develop pipeline #659 by @tdruez in #666
• Add attribution generation as a new downloadable output #684 by @tdruez in #712
• Add pathmap pipes module for path matching using Aho-Corasick #711 by @tdruez in #713
• Improve performances of map_jar_to_source pipe #711 by @tdruez in #723
• File viewer search and full screen by @tdruez in #725
• Upgrade the worker related dependencies to latest versions #726 by @tdruez in #727
• Improve speed of reset and delete using the _raw_delete API #729 by @tdruez in #730
• Upgrade ScanCode-toolkit to latest v32 #569 by @tdruez in #715
• Add step in d2d to scan mapped from/ files #569 by @tdruez in #736
• Add support for XLSX input in the load_inventory pipeline #735 by @tdruez in #741
• Use bulk_create for collect_and_create_codebase_resources step #659 by @tdruez in #747
• Specify update_fields during each save() related to tasks #726 by @tdruez in #748
• Upgrade ScanCode-toolkit to latest v32.0.2 #569 by @tdruez in #752
• Add support for unknown licenses in attribution output #749 by @tdruez in #750
• Add d2d step for JavaScript/TypeScript match #714 by @tdruez in #728
• Add ProjectCodebaseView to browse project codebase tree #744 by @tdruez in #754
• Add support for rocky linux in docker scans #757 by @pombredanne in #771
• Remove flag_to_meta_inf_files step from d2d pipeline by @keshav-space in #760
• Project configuration model and view #685 by @tdruez in #756
• Dark Mode for documentation added by @swastkk in #665
• Multiple file upload with drag option by @swastkk in #751
• Upgrade ScanCode-toolkit to v32.0.4 by @tdruez in #772
• Add License objects to the packages for attribution generation #775 by @tdruez in #776
• Add toggle button for the "Project data" panel #210 #774 by @tdruez in #777
• Support ABOUT files for code that is patched or vendored #740 by @tdruez in #778

New Contributors

• @swastkk made their first contribution in #665

Full Changelog: v32.2.0...v32.3.0

v32.2.0

What's Changed

• Enhance the output command to support multiple formats #646 by @tdruez in #655
• Enhance resolve_about_packages to handle filename and checksum values by @tdruez in #656
• Use radio for pipeline choices in modal #618 by @tdruez in #641
• Move the cyclonedx and spdx root modules into the pipes module by @tdruez in #657
• Add LAYERS sheet in the xlsx output for docker pipeline run #578 by @tdruez in #660
• Upgrade Django to 4.2.0 and to Psycopg to version 3 by @tdruez in #663
• Add support for manifest types using ScanCode-toolkit handlers #658 by @tdruez in #669
• Add cli option to copy source directory to project codebase by @tdruez in #672
• Improve the resolution of CycloneDX BOM and SPDX document by @tdruez in #688
• Add section to display codebase content in project view by @tdruez in #673
• Enhance the Resource details view to use the tabset system #215 by @tdruez in #691
• Fix logic in update_or_create_package #681 by @JonoYang in #682
• Bump ScanCode toolkit to 31.2.6 #693 by @pombredanne in #694
• v32.2.0 release by @tdruez in #695

Full Changelog: v32.1.0...v32.2.0

v32.1.0

• Add support for ScanCode.io results in the "load_inventory" pipeline. #609

• Add support for CycloneDX 1.4 to the "inspect-manifest" pipeline to import SBOM into
  a Project. #583

• Add fields in CycloneDX BOM output using the component properties.
  See registered properties at https://github.com/nexB/aboutcode-cyclonedx-taxonomy #637

• Upgrade to Python 3.11 in the Dockerfile. #611

• Refine the "Command Line Interface" documentation about the scanpipe command
  usages in the Docker context.
  Add the /app workdir in the "PYTHONPATH" env of the Docker file to make the
  scanpipe entry point available while running docker compose commands. #616

• Add a new tutorial about the "find vulnerabilities" pipeline and the vulnerablecode
  integration in the documentation. #600

• Rewrite the CLI tutorials for a Docker-based installation. #440

• Use CodebaseResource path instead of id as slug_field in URL navigation. #242

• Remove dead code related to the project_tree view #623

• Update scanpipe.pipes.ProjectCodebase and related code to work properly
  with the current Project/CodebaseResource path scheme. #624

• Add the SCANCODEIO_PAGINATE_BY setting to customize the number of items displayed per
  page for each object type. #563

• Add a setting for a per-file timeout. The maximum time allowed for a file to be
  analyzed when scanning a codebase is configurable with SCANCODEIO_SCAN_FILE_TIMEOUT
  while the maximum time allowed for a pipeline to complete can be defined using
  SCANCODEIO_TASK_TIMEOUT. #593

v32.0.1

• Upgrade ScanCode-toolkit and related dependencies to solve installation issues. #586

• Add support for Python 3.11 #611

• Populate documentDescribes field with Package and Dependency SPDX IDs in SPDX BOM output. #564

v32.0.0

• Add a new "find vulnerabilities" pipeline to lookup vulnerabilities in the
  VulnerableCode database for all project discovered packages.
  Vulnerability data is stored in the extra_data field of each package.
  More details about VulnerableCode at https://github.com/nexB/vulnerablecode/ #101

• Add a new "inspect manifest" pipeline to resolve packages from manifest, lockfile,
  and SBOM. The resolved packages are created as discovered packages.
  Support PyPI "requirements.txt" files, SPDX document as JSON ".spdx.json",
  and AboutCode ".ABOUT" files. #284

• Generate SBOM (Software Bill of Materials) compliant with the SPDX 2.3 specification
  as a new downloadable output. #389

• Generate CycloneDX SBOM (Software Bill of Materials) as a new downloadable output. #389

• Display Webhook status in the Run modal.
  The WebhookSubscription model was refined to capture delivery data. #389

• Display the current active step of a running pipeline in the "Pipeline" section of
  the project details view, inside the run status tag. #300

• Add proper pagination for API actions: resources, packages, dependencies, and errors.

• Refine the fields ordering in API Serializers based on the toolkit order. #546

• Keep the current filters state when submitting a search in list views. #541

• Improve the performances of the project details view to load faster by deferring the
  the charts rendering. This is especially noticeable on projects with a large amount
  of codebase resources and discovered packages. #193

• Add support for filtering by "Other" values when filtering from the charts in the
  Project details view. #526

• CodebaseResource.for_packages now returns a list of
  DiscoveredPackage.package_uid or DiscoveredPackage.package_url if
  DiscoveredPackage.package_uid is not present. This is done to reflect the
  how scancode-toolkit's JSON output returns package_uids in the
  for_packages field for Resources.

• Add the model DiscoveredDependency. This represents Package dependencies
  discovered in a Project. The scan_codebase and scan_packages pipelines
  have been updated to create DiscoveredDepdendency objects. The Project API has
  been updated with new fields:

  • dependency_count: The number of DiscoveredDependencies associated with the project.

  • discovered_dependencies_summary: A mapping that contains following fields:

    • total: The number of DiscoveredDependencies associated with the project.
    • is_runtime: The number of runtime dependencies.
    • is_optional: The number of optional dependencies.
    • is_resolved: The number of resolved dependencies.

  These values are also available on the Project view. #447

• The dependencies field has been removed from the DiscoveredPackage model.

• Create directory CodebaseResources in the rootfs pipeline. #515

• Add ProjectErrors when the DiscoveredPackage could not be fetched using the
  provided package_uid during the assemble_package step instead of failing the whole
  pipeline. #525

• Escape paths before using them in regular expressions in CodebaseResource.walk(). #525

• Disable multiprocessing and threading by default on macOS ("spawn" start method). #522

v31.0.0

• WARNING: Drop support for Python 3.6 and 3.7. Add support for Python 3.10.
  Upgrade Django to version 4.1 series.

• Upgrade ScanCode-toolkit to version 31.0.x.
  See https://github.com/nexB/scancode-toolkit/blob/develop/CHANGELOG.rst for an
  overview of the changes in the v31 compared to v30.

• Implement run status auto-refresh using the htmx JavaScript library.
  The statuses of queued and running pipeline are now automatically refreshed
  in the project list and project details views every 10 seconds.
  A new "toast" type of notification is displayed along the status update. #390

• Ensure the worker service waits for migrations completion before starting.
  To solve this issue we install the wait-for-it script available in
  Debian by @vishnubob and as suggested in the Docker documentation.
  In the docker-compose.yml, we let the worker wait for the web processing
  to be complete when gunicorn exposes port 8000 and web container is available. #387
  Reference: https://docs.docker.com/compose/startup-order/
  Reference: https://github.com/vishnubob/wait-for-it
  Reference: https://tracker.debian.org/pkg/wait-for-it

• Add a "create-user" management command to create new user with its API key. #458

• Add a "tag" field on the CodebaseResource model.
  The layer details are stored in this field in the "docker" pipeline. #443

• Add support for multiple inputs in the LoadInventory pipeline. #451

• Add new SCANCODEIO_REDIS_PASSWORD environment variable and setting
  to optionally set Redis instance password.

• Ensure a project cannot be deleted through the API while a pipeline is running. #402

• Display "License clarity" and "Scan summary" values as new panel in the project
  details view. The summary is generated during the scan_package pipeline. #411

• Enhance Project list view page: #413

  • 20 projects are now displayed per page
  • Creation date displayed under the project name
  • Add ability to sort by date and name
  • Add ability to filter by pipeline type
  • Add ability to filter by run status

• Correctly extract symlinks in docker images. We now use the latest
  container-inspector to fix symlinks extraction in docker image tarballs.
  In particular broken symlinks are not treated as an error anymore
  and symlinks are extracted correctly. #471 #407

• Add a Package details view including all model fields and resources.
  Display only 5 resources per package in the list view. #164 #464

• Add the ability to filter by empty and none values providing the
  "EMPTY" magic value to any filters. #296

• CodebaseResource.name now contains both the bare file name with extension, as
  opposed to just the bare file name without extension.
  Using a name stripped from its extension was something that was not used in
  other AboutCode project or tools. #467

• Export current results as XLSX for resource, packages, and errors list views. #48

• Add support for .tgz extension for input files in Docker pipeline #499

• Add support for resource missing file content in details view.
  Refine the annotation using the new className instead of type. #495

• Change the worksheet names in XLSX output, using the
  "PACKAGES", "RESOURCES", "DEPENDENCIES", and "ERRORS" names. #511

• Update application Package scanning step to reflect the updates in
  scancode-toolkit package scanning.

  • Package data detected from a file are now stored on the
    CodebaseResource.package_data field.
  • A second processing step is now done after scanning for Package data, where
    Package Resources are determined and DiscoveredPackages are created. #444

• CodebaseResource.for_packages now returns a list of
  DiscoveredPackage.package_uid or DiscoveredPackage.package_url if
  DiscoveredPackage.package_uid is not present. This is done to reflect the
  how scancode-toolkit's JSON output returns package_uid in the
  for_packages field for Resources.

Full Changelog: v30.2.0...v31.0.0

v30.2.0

• Add authentication for the Web UI views and REST API endpoint.
  The authentication is disabled by default and can be enabled using the
  SCANCODEIO_REQUIRE_AUTHENTICATION settings.
  When enabled, users have to authenticate through a login form in the Web UI, or using
  their API Key in the REST API.
  The API Key can be viewed in the Web UI "Profile settings" view once logged-in.
  Users can be created using the Django "createsuperuser" management command. #359

• Include project errors in XLSX results output. #364

• Add input_sources used to fetch inputs to JSON results output. #351

• Refactor the update_or_create_package pipe to support the ProjectError system
  and fix a database transaction error. #381

• Add webhook subscription available when creating project from REST API. #98

• Add the project "reset" feature in the UI, CLI, and REST API. #375

• Add a new GitHub action that build the docker-compose images and run the test suite.
  This ensure that the app is properly working and tested when running with Docker. #367

• Add --no-install-recommends in the Dockerfile apt-get install and add the
  linux-image-amd64 package. This packages makes available the kernels
  required by extractcode and libguestfs for proper VM images extraction. #367

• Add a new list-project CLI command to list projects. #365

v30.1.1

• Remove the --no-install-recommends in the Dockerfile apt-get install to include
  required dependencies for proper VM extraction. #367

v30.1.0

• Synchronize QUEUED and RUNNING pipeline runs with their related worker jobs during
  worker maintenance tasks scheduled every 10 minutes.
  If a container was taken down while a pipeline was running, or if pipeline process
  was killed unexpectedly, that pipeline run status will be updated to a FAILED state
  during the next maintenance tasks.
  QUEUED pipeline will be restored in the queue as the worker redis cache backend data
  is now persistent and reloaded on starting the image.
  Note that internally, a running job emits a "heartbeat" every 60 seconds to let all the
  workers know that it is properly running.
  After 90 seconds without any heartbeats, a worker will determine that the job is not
  active anymore and that job will be moved to the failed registry during the worker
  maintenance tasks. The pipeline run will be updated as well to reflect this failure
  in the Web UI, the REST API, and the command line interface. #130

• Enable redis data persistence using the "Append Only File" with the default policy of
  fsync every second in the docker-compose. #130

• Add a new tutorial chapter about license policies and compliance alerts. #337

• Include layers in docker image data. #175

• Fix a server error on resource details view when the compliance alert is "missing". #344

• Migrate the ScanCodebase pipeline from scancode.run_scancode subprocess to
  scancode.scan_for_application_packages and scancode.scan_for_files. #340

v30.0.1

• Fix a build failure related to dependency conflict #342