v34.1.0

Changelog:

• Add support for importing CycloneDX SBOM 1.2, 1.3, 1.4 and 1.5 spec formats. #1045
• The pipeline help modal is now available from all project views: form, list, details.
  The docstring are converted from markdown to html for proper rendering. #1105
• Add a new CollectSymbols pipeline (addon) for collecting codebase symbols using
  Universal Ctags. #1116
• Capture errors during the inspect_elf_binaries pipeline execution.
  Errors on resource inspection are stored as project error message instead of global
  pipeline failure.
  The problematic resource path is stored in the message details and displayed in the
  message list UI as a link to the resource details view. #1121 #1122
• Use the package_only option in scancode get_package_data API in
  inspect_packages pipeline, to skip license and copyright detection in
  extracted license and copyright statements found in package metadata. aboutcode-org/scancode-toolkit#3689
• Rename the match_to_purldb pipeline to match_to_matchcode, and add
  MatchCode.io API settings to ScanCode.io settings.
• In the DiscoveredPackage model, rename the "datasource_id" attribute to
  "datasource_ids" and add a new attribute "datafile_paths". This is aligned
  with the scancode-toolkit Package model, and package detection information
  is now stored correctly. Also update the UI for discovered packages to
  show the corresponding package datafiles and their datasource IDs.
  A data migration is included to facilitate the migration of existing data. #1099
• Add PurlDB tab, displayed when the PURLDB_URL settings is configured.
  When loading the package details view, a request is made on the PurlDB to fetch and
  and display any available data. #1125
• Create a new management command purldb-scan-queue-worker, that runs
  scancode.io as a Package scan queue worker for PurlDB.
  purldb-scan-queue-worker gets the next available Package to be scanned and
  the list of pipeline names to be run on the Package from PurlDB, creates a
  Project, fetches the Package, runs the specified pipelines, and returns the
  results to PurlDB. #1078 aboutcode-org/purldb#236
• Update matchcode-toolkit to v4.0.0

Full Changelog: v34.1.0...v34.1.0

v34.0.0

Changelog:

• Add ability to "group" pipeline steps to control their inclusion in a pipeline run.
  The groups can be selected in the UI, or provided using the
  "pipeline_name:group1,group2" syntax in CLI and REST API. #1045

• Refine pipeline choices in the "Add pipeline" modal based on the project context.

  • When there is at least one existing pipeline in the project, the modal now includes
    all addon pipelines along with the existing pipeline for selection.
  • In cases where no pipelines are assigned to the project, the modal displays all
    base (non-addon) pipelines for user selection. #1071

• Rename pipeline for consistency and precision:

  • scan_codebase_packages: inspect_packages

  Restructure the inspect_manifest pipeline into:

  • load_sbom: for loading SPDX/CycloneDX SBOMs and ABOUT files
  • resolve_dependencies: for resolving package dependencies
  • inspect_packages: gets package data from package manifests/lockfiles

  A data migration is included to facilitate the migration of existing data.
  Only the new names are available in the web UI but the REST API and CLI are backward
  compatible with the old names. #1034 https://github.com/nexB/scancode.io/discussions/1035

• Remove "packageFileName" entry from SPDX output. #1076

• Add an add-on pipeline for collecting DWARF debug symbol compilation
  unit paths when available from elfs. aboutcode-org/purldb#260

• Extract all archives recursively in the scan_single_package pipeline. #1081

• Add URL scheme validation with explicit error messages for input URLs. #1047

• All supported output_format can now be downloaded using the results_download API
  action providing a value for the new output_format parameter. #1091

• Add settings related to fetching private files. Those settings allow to
  define credentials for various authentication types. #620 #203

• Update matchcode-toolkit to v3.0.0

What's Changed

• Refine pipeline choices in the "Add pipeline" modal #1071 by @tdruez in #1072
• Add ability to "group" pipeline steps to control inclusion #1045 by @tdruez in #1055
• Improve the documentation about external service (integrations) by @tdruez in #1073
• Restructure pipelines for verbosity by @AyanSinhaMahapatra in #1074
• Add an add-on pipeline for collecting dwarfs from elfs by @TG1999 in #1068
• Run the extract_archives step in ScanSinglePackage #1081 by @tdruez in #1083
• Use new /collect/index_packages/ endpoint to populate PurlDB by @keshav-space in #1084
• Add URL scheme validation with explicit error messages #1047 by @tdruez in #1085
• Bump matchcode-toolkit version to v3.0.0 by @JonoYang in #1088
• Improve the results_download API action to accept output_format #1091 by @tdruez in #1092
• Add support for fetching authentications #620 by @tdruez in #1097
• Add settings for providing Skopeo credentials #203 by @tdruez in #1098

Full Changelog: v33.1.0...v34.0.0

v33.1.0

Changelog:

• Rename multiple pipelines for consistency and precision:
  • docker: analyze_docker_image
  • root_filesystems: analyze_root_filesystem_or_vm_image
  • docker_windows: analyze_windows_docker_image
  • inspect_manifest: inspect_packages
  • deploy_to_develop: map_deploy_to_develop
  • scan_package: scan_single_package
    A data migration is included to facilitate the migration of existing data.
    Only the new names are available in the web UI but the REST API and CLI are backward
    compatible with the old names. #1044
• Generate CycloneDX SBOM in 1.5 spec format, migrated from 1.4 previously.
  The Package vulnerabilities are now included in the CycloneDX SBOM when available. #807
• Improve the inspect_manifest pipeline to accept archives as inputs. #1034
• Add support for "tagging" download URL inputs using the "#" section of URLs.
  This feature is particularly useful in the map_develop_to_deploy pipeline when
  download URLs are utilized as inputs. Tags such as "from" and "to" can be specified
  by adding "#from" or "#to" fragments at the end of the download URLs.
  Using the CLI, the uploaded files can be tagged using the "filename:tag" syntax
  while using the --input-file arguments.
  In the UI, tags can be edited from the Project details view "Inputs" panel.
  On the REST API, a new upload_file_tag field is available to use along the
  upload_file. #708

What's Changed

• Rename multiple pipelines #1044 by @tdruez in #1053
• Removed extra spaces from package copyright section. by @Divyansh044 in #1054
• Display the entire content of map files by @keshav-space in #1014
• Upgrade CycloneDX output to SPEC v1.5 #807 by @tdruez in #1057
• Update inspect_manifest to accept archives by @AyanSinhaMahapatra in #1037
• Add support for "tagging" URL inputs using # #708 by @tdruez in #1062
• Create "client" matching pipeline by @JonoYang in #1042
• scancode-action docs #599 by @tdruez in #1065
• Support patterns in ABOUT resource paths by @AyanSinhaMahapatra in #982
• Add support for tagging input files in CLI and UI #708 by @tdruez in #1069

New Contributors

• @Divyansh044 made their first contribution in #1054

Full Changelog: v33.0.0...v33.1.0

v33.0.0

Changelog:

• Upgrade Django to version 5.0 and drop support for Python 3.8 and 3.9 #1020
• Fetching "Download URL" inputs is now delegated to an initial pipeline step that is
  always run as the start of a pipeline.
  This allows to run pipelines on workers running from a remote location, external to
  the main ScanCode.io app server. #410
• Migrate the Project.input_sources field into a InputSource model. #410
• Refactor run_scancode to not fail on scan errors happening at the resource level,
  such as a timeout. Project error message are created instead. #1018
• Add support for the SCANCODEIO_SCAN_FILE_TIMEOUT setting in the scan_package pipeline. #1018
• Add support for non-archive single file in the scan_package pipeline. #1009
• Do not include "add-on" pipelines in the "New project" form choices. #1041
• Display a "Run pipelines" button in the "Pipelines" panel.
  Remove the ability to run a single pipeline in favor of running all "not started"
  project pipeline. #997
• Fix an issue where the pipeline details cannot be fetched when using URLs that
  include credentials such as "user:pass@domain". #998

What's Changed

• Ignore whitespace files by @keshav-space in #976
• Upgrade multiple dependencies to their latest version by @tdruez in #984
• Provide context for project_resources_url in summary views by @keshav-space in #986
• Fix processing embedded archives by @AyanSinhaMahapatra in #1008
• Expand the scope of flag_whitespace_files by @keshav-space in #987
• Choose best package for PurlDB matched resources by @keshav-space in #975
• Enable local file packages in d2d pipeline by @AyanSinhaMahapatra in #992
• Use package_uid for cdx bom-ref by @keshav-space in #1016
• Refactor run_scancode to handle errors along success #1018 by @tdruez in #1019
• Upgrade Django to version 5.0.0 #1020 by @tdruez in #1021
• Fix minor doc typo in tutorial_api_analyze_package_archive.rst by @pombredanne in #1026
• feat: Resolve make envfile command by @jayanth-kumar-morem in #1029
• Add support for non-archive file in scan_package #1009 by @tdruez in #1031
• Bump matchcode-toolkit to 2.0.1 by @JonoYang in #1033
• Migrate the Project.input_sources field to its own model #410 by @tdruez in #1039
• Do not include add-on pipelines in the "New project" form choices #1041 by @tdruez in #1043
• Display a "Run pipelines" button in the "Pipelines" panel #997 by @tdruez in #1046
• Fix the pipeline details fetch for URLs with credentials #998 by @tdruez in #1048
• v33.0.0 by @tdruez in #1049

New Contributors

• @jayanth-kumar-morem made their first contribution in #1029

Full Changelog: v32.7.0...v33.0.0

v32.7.0

• Display the Run.scancodeio_version in the Pipeline run modal.
  When possible this value is displayed as a link to the diff view between the current
  ScanCode.io version and the version used when the Pipeline was run. #956

• Improve presentation of the "Resources detected license expressions" project section. #937

• Add ability to sort by Package URL in the package list #938

• Fix an issue where the empty project settings were overriding the settings loaded
  from a config file. #961

• Control the execution order of Pipelines within a Project. Pipelines are not allowed
  to start anymore unless all the previous ones within a Project have been completed. #901

• Add support for webhook subscriptions in project clone. #910

• Add resources license expression summary panel in the project details view.
  This panel displays the list of licenses detected in the project and includes links
  to the resources list. #355

• Add the tag field on the DiscoveredPackage model. This new field is used to store
  the layer id where the package was found in the Docker context. #919

• Add to apply actions, such as archive, delete, and reset to a selection of projects
  from the main list. #488

• Add new "Outputs" panel in the Project details view.
  Output files are listed and can be downloaded from the panel. #678

• Add a step in the deploy_to_develop pipelines to create "local-files" packages
  with from-side resource files that have one or more relations with to-side resources
  that are not part of a package.
  This allows to include those files in the SBOMs and attribution outputs. #914

• Enable sorting the packages list by resource count. #978

What's Changed

• Use bulk creation for resource in ScanCodebase pipeline by @tdruez in #897
• Refactor the pipes.log_progress as a context manager by @tdruez in #898
• Control the execution order of Pipelines within a Project #901 by @tdruez in #906
• Add progress logging to scan_resources pipe #908 by @tdruez in #909
• Add support for Webhook subscriptions in project clone by @tdruez in #910
• Modify pipelines to get purls from package_data by @AyanSinhaMahapatra in #904
• Add resources license summary panel in the project details view #355 by @tdruez in #916
• Include link to the resource archive details view in breadcrumb #918 by @tdruez in #920
• Add tag on DiscoveredPackage model and store the docker layer id #919 by @tdruez in #923
• Send only unique PURLs to PurlDB by @keshav-space in #836
• Add UI and JS logic for the list selection and actions #488 by @tdruez in #924
• Add "Outputs" panel in the Project details view #678 by @tdruez in #928
• 903 update purldb matching by @JonoYang in #913
• Base implementation for local-files packages creation #914 by @tdruez in #927
• D2D: Add house-keeping step by @keshav-space in #911
• Re-enable offline docker installation by @pombredanne in #540
• D2d pipeline updates by @JonoYang in #934
• Update status for deployed files without license by @keshav-space in #932
• Don't directory match anything mapped or matched by @JonoYang in #945
• Remove too-many-maps status from path mapping by @keshav-space in #946
• Use more specific status for PurlDB matchings by @keshav-space in #948
• Bump scancode-toolkit to v32.0.7 by @AyanSinhaMahapatra in #947
• Update d2d pipeline to use specific license flags by @AyanSinhaMahapatra in #954
• Skip matched archives in directory matching by @keshav-space in #957
• Limit npm package lookup to resource without status by @keshav-space in #952
• Display only 5 packages per resource in list view by @keshav-space in #955
• [Workaround] Handle dangling legal files by @keshav-space in #949
• Add ability to sort by Package URL in package list #938 by @tdruez in #962
• Improve presentation "Resources detected license expressions" #937 by @tdruez in #964
• Display the Run.scancodeio_version in the Pipeline run modal #956 by @tdruez in #965
• Improve the QS performances of each view #958 by @tdruez in #972
• Bump scancode-toolkit to v32.0.8 by @AyanSinhaMahapatra in #974
• Load the Project license and status summary panels lazily by @tdruez in #980
• Add GitHub action to build Docker image and publish it to GHCR by @tdruez in #983

Full Changelog: v32.6.0...v32.7.0

v32.6.0

• Improve the performance of the codebase relations list view to support large number
  of entries. #858
• Improve DiscoveredPackageListView query performances refining the prefetch_related. #856
• Fix the map_java_to_class d2d pipe to skip if no .java file is found. #853
• Enhance Package search to handle full pkg: purls and segment of purls. #859
• Add a new step in the deploy_to_develop pipeline where we tag archives as
  processed, if all the resources in their extracted directory is mapped/processed. #827
• Add the ability to clone a project. #874
• Improve perceived display performance of projects charts and stats on home page.
  The charts are displayed when the number of resources or packages are less than
  5000 records. Else, a button to load the charts is displayed. #844
• Add advanced search query system to all list views. Refer to the documentation for details about the search syntax. #871
• Migrate the ProjectError model to a global ProjectMessage.
  3 level of severity available: INFO, WARNING, and ERROR. #338
• Add label/tag system that can be used to group and filters projects. #769

What's Changed

• Map files with empty status in map_javascript step by @keshav-space in #852
• Improve DiscoveredPackageListView query performances #856 by @tdruez in #860
• Refactor the CodebaseRelationListView #858 by @tdruez in #864
• Fix the map_java_to_class pipe to skip if no .java is found #853 by @tdruez in #865
• Assign status to processed archives by @AyanSinhaMahapatra in #861
• Fix duplicate results in in_package query by @keshav-space in #879
• Fix processed archive bug by @AyanSinhaMahapatra in #877
• Fix run with docker link and grammar by @Hritik14 in #867
• Add clone method on Project model #822 by @tdruez in #874
• Add toggle button for graphs when more than 5000 resources #844 by @tdruez in #884
• Add advanced search query system to all list views #871 by @tdruez in #892
• Migrate the ProjectError model to a global ProjectMessage #338 by @tdruez in #820
• Add errors if ABOUT file resource not found by @AyanSinhaMahapatra in #891
• Avoid path map solely based on file name by @keshav-space in #882
• Handle thirdparty package.json file in d2d by @keshav-space in #881
• Document rosetta support by @Hritik14 in #894
• Add label/tag system on Projects using django-taggit #769 by @tdruez in #893
• purldb matching updates #781 by @JonoYang in #782

New Contributors

• @Hritik14 made their first contribution in #867

Full Changelog: v32.5.2...v32.6.0

v32.5.2 (Security Release)

Security

This release addresses the security issue detailed below. We encourage all users of ScanCode.io to upgrade as soon as possible.

• GHSA-6xcx-gx7r-rccj: Reflected Cross-Site Scripting (XSS) in license endpoint
  The license_details_view function was subject to cross-site scripting (XSS)
  attack due to inadequate validation and sanitization of the key parameter.
  The license views were migrated class-based views are the inputs are now properly
  sanitized.
  Credit to @0xmpij for reporting the vulnerability.

What's Changed

• Add JavaScript colocation mapping step in d2d pipeline by @keshav-space in #803
• Migrate license views to CBV and main UI #847 by @tdruez in #849
• Remove the subprocess need in the scan_package pipeline #798 by @tdruez in #855
• Security checks #850 by @tdruez in #851

Full Changelog: v32.5.1...v32.5.2

v32.5.1 (Security Release)

Security

This release addresses the security issue detailed below. We encourage all users of ScanCode.io to upgrade as soon as possible.

• GHSA-2ggp-cmvm-f62f: Command injection in docker image fetch process The fetch_docker_image function was subject to potential injection attack. The user inputs are now sanitized before calling the subprocess function. GHSA-2ggp-cmvm-f62f

What's Changed

• Tag about files and companions correctly #825 by @AyanSinhaMahapatra in #837
• Add new documentation chapter about automation #828 by @tdruez in #845
• Add vulnerability support for discovered dependencies #835 by @tdruez in #846
• fix: update rootfs to scan resource path by @philcali in #840

New Contributors

• @philcali made their first contribution in #840

Full Changelog: v32.5.0...v32.5.1

v32.5.0

WARNING

After upgrading the ScanCode.io codebase to this version, and following the docker compose build, the permissions of the /var/scancodeio/ directory of the Docker volumes require to be updated for the new app user, using:

docker compose run -u 0:0 web chown -R app:app /var/scancodeio/

What's Changed

• Upgrade ScanCode-toolkit to v32.0.6 by @AyanSinhaMahapatra in #806
• Support ignored_resources attribute in ABOUT files by @AyanSinhaMahapatra in #810
• Add pipeline to populate PurlDB by @keshav-space in #788
• Update project inputs #813 by @tdruez in #814
• Add a ScanCodebasePackage pipeline #815 by @tdruez in #818
• Run Docker as non-root user using virtualenv #399 by @tdruez in #821
• Add column sort and filters in dependency list view #823 by @tdruez in #830
• Fix the SPDX output to include license texts for LicenseRef #841 by @tdruez in #842

Full Changelog: v32.4.0...v32.5.0

v32.4.0

What's Changed

• Append "compliance_alert" to update_fields during save() #784 by @tdruez in #786
• Add slug field on project model and use it in URLs #745 by @tdruez in #789
• Enhance list view filters #216 by @tdruez in #791
• D2D: Enhance Javascript mapping by @keshav-space in #779
• Upgrade ScanCode-toolkit to v32.0.5rc3 by @AyanSinhaMahapatra in #792
• Fix attribution output when package license attribute is empty by @tdruez in #793
• Upgrade VulnerableCode integration #600 by @tdruez in #797
• Display relations in a resource details tab #799 by @tdruez in #801
• Include commit tag in the ScanCode.io version by @tdruez in #802
• Add ScanCode License Score as a Project setting #395 by @tdruez in #804
• Add policies and compliance alert support for Discovered Packages #151 by @tdruez in #805
• Refactor license_matches in summary following ScanCode-toolkit upgrade by @tdruez in #808

Full Changelog: v32.3.0...v32.4.0