411 toolkit upgrade 2

[JonoYang]

This PR updates scancode-toolkit in scancode.io, expected test results, as well as updating code that uses deprecated scancode-toolkit functions.

[tdruez]

@JonoYang can we make the new dependency system as a separate feature from the toolkit upgrade?

[JonoYang]

@tdruez No problem, I'll revert the model changes.

[JonoYang]

@tdruez

I've regenerated the test data using the new command you created and I run into some problems where I would like your opinion on what should be done.

1. ProjectCodebase and the lack of a root codebase directory

I run regen_test_data.py and the tests for ProjectCodebase all fail now because the ProjectCodebase code has no good way of determining the root of a scan. Previous scancode.io Resource creation behavior was to create the codebase directory Resource, and then every Resource under codebase. The paths of all Resources created starts with codebase. The ProjectCodebase object considers the root directory
to be the codebase directory (https://github.com/nexB/scancode.io/blob/main/scanpipe/pipes/codebase.py#L71) Since I regenerated the test fixtures, the codebase directory Resource is no longer created and Resource paths no longer start with codebase

I have some previous attempts where I worked on different solutions. In the branch starting-paths, we had the idea of considering the directories under codebase directory as individual roots. (https://github.com/nexB/scancode.io/blob/fa6cd0c597bcb53d449c0c119208f0f314955ea2/scanpipe/pipes/codebase.py#L81) Should I expand on the work I did there or should I remove ProjectCodebase entirely until we come up with something satisfactory?

In a scancode JSON output, all Packages detected in a scan are present in a top-level attribute named packages. Likewise, all detected Dependencies are placed in the Dependencies attribute. Multiple copies of the same package can be present in the packages field, if a particular package was detected multiple times in the same codebase. Each copy of this package will have a different package_uid. A package_uid is the purl of that package with a qualifier named uuid that is specific to the scancode run. e.g. pkg:pypi/django-audit-tools@0.4.0?uuid=9c19275c-c3fe-43dd-b6ec-a4f2bf65810f

For each Resource that is for a Package, the for_packages for those Resources will be populated with the package_uid of the Package they are for.

When I first modified the pipes code to handle the new top-level packages field from a scan, I just put the package_uid in the extra_data field. When I wanted to assign a Resource to a Package, I would then query for that package_uid in the DiscoveredPackage table (https://github.com/nexB/scancode.io/blob/411-toolkit-upgrade-2/scanpipe/pipes/scancode.py#L392).

I started testing the pipeline using the package django-audit-tools-0.4.0.tar.gz and I was getting errors when I tried to look up a package_uid from a Resource's for_packages field in the DiscoveredPackage table. The issue was because the same package was detected multiple times throughout the scan, so there were multiple instances of that package (with different package_uids), but only 1 version of that package was saved to the database (since I am assuming that we can only have unique instances of packages in the DiscoveredPackage table).

I decided to update how I stored the package_uid in extra_data by creating the field package_uids whose value is a list of package_uids. (https://github.com/nexB/scancode.io/blob/774c5b727b52f7d93601e016a9c3a05941ec3d98/scanpipe/pipes/__init__.py#L95)

The extra_data probably isn't the best place to keep the package_uids, so I'd like to know how we should handle package_uids in DiscoveredPackages. What is the best way to store package_uids on DiscoveredPackages?

The scan output asgiref-3.3.0_scan.json created by regen_test_data.py is slightly different than the scan output generated by scancode-toolkit:

• No top-level dependencies field
• the for_packages field on Resources is populated with purls instead of package_uids

This causes tests like https://github.com/nexB/scancode.io/blob/411-toolkit-upgrade-2/scanpipe/tests/test_pipes.py#L521 to fail since we expect the json to be in the sctk json output format.

Properly handling package_uids in (2) will probably help with this.

[tdruez]

ProjectCodebase and the lack of a root codebase directory

The test data was generated from a time where the codebase/ prefix was stored in the DB. The test data should have been updated at the time of the code change.

Should I expand on the work I did there or should I remove ProjectCodebase entirely until we come up with something satisfactory?

Why not completing the #378 (review) PR instead? It seems that my review points were never addressed but it looks like we were close to a merge. Will this PR still relevant to solve this?

The extra_data probably isn't the best place to keep the package_uids

Probably not the best place, but I think the goal of this first PR is to make upgrade the toolkit with the least amount of models changes to get it out there asap. We should then have tickets and more PRs to work on the optimizations, such as new models and fields for Packages and Dependencies

The scan output asgiref-3.3.0_scan.json created by regen_test_data.py is slightly different than the scan output generated by scancode-toolkit:

We probably have to update the JSONResultsGenerator to include the new format and data.

[JonoYang]

@tdruez I'll revisit #378 and see if I can get it to work with what I have in this branch.

Probably not the best place, but I think the goal of this first PR is to make upgrade the toolkit with the least amount of models changes to get it out there asap. We should then have tickets and more PRs to work on the optimizations, such as new models and fields for Packages and Dependencies

ack. I will create tickets for some of the further work that needs to be done.

[JonoYang]

@tdruez

I tagged the ProjectCodebase tests with expectedFailure and regenerated the asgiref-3.3.0 test files using regen_test_data.py

I've added scanpipe/tests/data/asgiref-3.3.0_scancode_scan.json for use in test_scanpipe_pipes_scancode_create_codebase_resources_inject_policy (https://github.com/nexB/scancode.io/blob/411-toolkit-upgrade-2/scanpipe/tests/test_pipes.py#L553) because scanpipe/tests/data/asgiref-3.3.0_scan.json (generated by regen_test_data.py) is not exactly in the same format as what scancode-toolkit outputs.

[tdruez]

While running pipelines on this branch:

module 'packagedcode.debian' has no attribute 'get_installed_packages'

Traceback:
  File "/app/scanpipe/pipelines/__init__.py", line 115, in execute
    step(self)
  File "/app/scanpipe/pipelines/docker.py", line 93, in collect_and_create_system_packages
    docker.scan_image_for_system_packages(self.project, image)
  File "/app/scanpipe/pipes/docker.py", line 166, in scan_image_for_system_packages
    for i, (purl, package, layer) in enumerate(installed_packages):
  File "/usr/local/lib/python3.9/site-packages/container_inspector/image.py", line 446, in get_installed_packages
    for purl, package in layer.get_installed_packages(packages_getter):
  File "/app/scanpipe/pipes/debian.py", line 31, in package_getter
    packages = debian.get_installed_packages(

1. our test suite did not catch this.
2. The packager getter get_installed_packages system was changed on the toolkit side, this needs to be upgraded on the ScanCode.io too

See aboutcode-org/scancode-toolkit@fbdba2e#diff-29c4686d506744c5a6ae71725c1fa392956829a0c6009c53d26536ef54e01858L57

The following pipe modules are impacted:

• alpine
• debian
• rpm

[tdruez]

Scanning https://github.com/bastikr/boolean.py/archive/refs/heads/master.zip with scan_package create 1 discovered package in the DB but the summary returns 4 times the same entry in the summary key_files_packages.

https://staging.scancode.io/api/projects/ac26f606-4a7b-4359-b585-72a9616a921c/summary/
https://staging.scancode.io/api/projects/ac26f606-4a7b-4359-b585-72a9616a921c/results/

See https://github.com/nexB/scancode.io/blob/411-toolkit-upgrade-2/scanpipe/pipes/scancode.py#L487

[JonoYang]

While running pipelines on this branch:

module 'packagedcode.debian' has no attribute 'get_installed_packages'

Traceback:
  File "/app/scanpipe/pipelines/__init__.py", line 115, in execute
    step(self)
  File "/app/scanpipe/pipelines/docker.py", line 93, in collect_and_create_system_packages
    docker.scan_image_for_system_packages(self.project, image)
  File "/app/scanpipe/pipes/docker.py", line 166, in scan_image_for_system_packages
    for i, (purl, package, layer) in enumerate(installed_packages):
  File "/usr/local/lib/python3.9/site-packages/container_inspector/image.py", line 446, in get_installed_packages
    for purl, package in layer.get_installed_packages(packages_getter):
  File "/app/scanpipe/pipes/debian.py", line 31, in package_getter
    packages = debian.get_installed_packages(

1. our test suite did not catch this.
2. The packager getter get_installed_packages system was changed on the toolkit side, this needs to be upgraded on the ScanCode.io too

See nexB/scancode-toolkit@fbdba2e#diff-29c4686d506744c5a6ae71725c1fa392956829a0c6009c53d26536ef54e01858L57

The following pipe modules are impacted:

• alpine
• debian
• rpm

The get_installed_packages function has been removed from the alpine, debian/ubuntu, and rpm package handlers from packagedcode. There are now multiple "datafile handlers" that handle the different OS files, like the debian status files. The datafile handler then assemble all of the detected packages, dependencies, etc it finds. For example, this is the assemble method for the datafile handler that deals with installed Debian packages. (https://github.com/nexB/scancode-toolkit/blob/develop/src/packagedcode/debian.py#L223)

There doesn't seem to be an easy way to get the previous get_installed_packages functionality back from the new way we process package data. I think the logic from the assemble method I linked earlier is what we want, but we'll have to modify it to fit our model in scancode.io.

[JonoYang]

@tdruez

I've brought in the old code that handled processing installed Debian package files and updated enough of it to get the pipeline to work. I had some issues with license detection, but it was resolved when I ran scancode --reindex-licenses.

[tdruez]

I've brought in the old code that handled processing installed Debian package files and updated enough of it to get the pipeline to work.

I don't think we want to bring old toolkit code over in ScanCode.io, this defeat the purpose of a toolkit upgrade, but we should rather make use of the new code.

We had discussion today with @pombredanne to discuss this and he will provide solution on the toolkit side.

In the mean time, we should:

• Add a simple unit test that triggers the various get_installed_packages in debian, alpine and rpm (the test will fail for now)
• Solve 411 toolkit upgrade 2 #435 (comment)

[tdruez]

It seems a bit much to test on a whole data structure such as package_data.
Could we use a more specific id instead?

[tdruez]

@JonoYang The code looks pretty good.
Do you have a list of what's left to do to complete this PR?

[JonoYang]

@tdruez

The last few things I finished up were adding the package_uid field on the DiscoveredPackage model and replacing the ubuntu and redhat test Docker images with smaller images. Other than that, I need to merge aboutcode-org/scancode-toolkit#2974 and aboutcode-org/scancode-toolkit#2988 into the develop branch and then create a new release of scancode so we can test this branch out on staging.