Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Display scan summary data on project pages #411

Closed
JonoYang opened this issue Mar 17, 2022 · 0 comments
Closed

Display scan summary data on project pages #411

JonoYang opened this issue Mar 17, 2022 · 0 comments
Assignees
@JonoYang

Comments

@JonoYang
Copy link
Member

We are overhauling how the --summary plugin works in scancode-toolkit by updating how license clarity scoring works (aboutcode-org/scancode-toolkit#2861), and changing the output of the --summary plugin to show a primary license expression and the license clarity score. The new version of the --summary plugin is available on the develop branch of scancode-toolkit as --summary2. It would be nice to be able to show the new summary information on a scancode.io project page.

Since scanning a codebase in scancode-toolkit is different than scanning a codebase in scancode.io, we cannot directly use the scancode-toolkit plugins in scancode.io. We'll have to port the --classify, --license-clarity-score, and --summary2 plugins so they work in a scancode.io context.
@JonoYang JonoYang self-assigned this Mar 17, 2022
JonoYang added a commit that referenced this issue Mar 24, 2022
@JonoYang
Show license clarity data in project view
b07e71c 
Reference: #411

Signed-off-by: Jono Yang <jyang@nexb.com>
JonoYang added a commit that referenced this issue Mar 24, 2022
@JonoYang
Add styling to license clarity data table
4af6573 
    * Update help text

Reference: #411

Signed-off-by: Jono Yang <jyang@nexb.com>
JonoYang added a commit that referenced this issue Mar 26, 2022
@JonoYang
Pass more summary data to project view
7ac6a3a 
    * Add fontawesome v 5.13.0

Reference: #411

Signed-off-by: Jono Yang <jyang@nexb.com>
JonoYang added a commit that referenced this issue Mar 29, 2022
@JonoYang
Show summary data in transposed table
06048d6 
Reference: #411

Signed-off-by: Jono Yang <jyang@nexb.com>
JonoYang added a commit that referenced this issue Mar 29, 2022
@JonoYang
Pass summary data to view in a dictionary
a2c794e 
Reference: #411

Signed-off-by: Jono Yang <jyang@nexb.com>
JonoYang added a commit that referenced this issue Mar 30, 2022
@JonoYang
Display summary data using lists
7cb2a87 
Reference: #411

Signed-off-by: Jono Yang <jyang@nexb.com>
JonoYang added a commit that referenced this issue Apr 21, 2022
@JonoYang
Show license clarity data in project view
bec79aa 
Reference: #411

Signed-off-by: Jono Yang <jyang@nexb.com>
JonoYang added a commit that referenced this issue Apr 21, 2022
@JonoYang
Add styling to license clarity data table
af54fe4 
    * Update help text

Reference: #411

Signed-off-by: Jono Yang <jyang@nexb.com>
JonoYang added a commit that referenced this issue Apr 21, 2022
@JonoYang
Pass more summary data to project view
058fc96 
    * Add fontawesome v 5.13.0

Reference: #411

Signed-off-by: Jono Yang <jyang@nexb.com>
JonoYang added a commit that referenced this issue Apr 21, 2022
@JonoYang
Show summary data in transposed table
da19a2c 
Reference: #411

Signed-off-by: Jono Yang <jyang@nexb.com>
JonoYang added a commit that referenced this issue Apr 21, 2022
@JonoYang
Pass summary data to view in a dictionary
6cebe7c 
Reference: #411

Signed-off-by: Jono Yang <jyang@nexb.com>
JonoYang added a commit that referenced this issue Apr 21, 2022
@JonoYang
Display summary data using lists
6477c97 
Reference: #411

Signed-off-by: Jono Yang <jyang@nexb.com>
JonoYang added a commit that referenced this issue Apr 21, 2022
@JonoYang
Separate code into methods
dfec6e4 
Reference: #411

Signed-off-by: Jono Yang <jyang@nexb.com>
JonoYang added a commit that referenced this issue Apr 21, 2022
@JonoYang
Update scancode options for scan_package.py
a6be58a 
Reference: #411
Signed-off-by: Jono Yang <jyang@nexb.com>
JonoYang added a commit that referenced this issue Apr 21, 2022
@JonoYang
Show new license clarity score fields
b9e1df5 
Reference: #411
Signed-off-by: Jono Yang <jyang@nexb.com>
tdruez added a commit that referenced this issue May 2, 2022
@tdruez
Upgrade scancode-toolkit to latest beta release #411
14f36b7 
Signed-off-by: Thomas Druez <tdruez@nexb.com>
JonoYang added a commit that referenced this issue May 5, 2022
@JonoYang
Handle new scan format in scancode pipes #411
1957ab8 
Signed-off-by: Jono Yang <jyang@nexb.com>
JonoYang added a commit that referenced this issue May 6, 2022
@JonoYang
Handle package_uids for DiscoveredPackages #411
b406ac0 
    * Remove create_discovered_packages2 and create_codebase_resources2

Signed-off-by: Jono Yang <jyang@nexb.com>
JonoYang added a commit that referenced this issue May 6, 2022
@JonoYang
Update deprecated code #411
f573cb9 
    * Normalize package_uids before comparing results in tests
    * Update expected test results

Signed-off-by: Jono Yang <jyang@nexb.com>
JonoYang added a commit that referenced this issue May 6, 2022
@JonoYang
Update deprecated code #411
f0bc99c 
    * Normalize package_uids before comparing results in tests
    * Update expected test results

Signed-off-by: Jono Yang <jyang@nexb.com>
JonoYang added a commit that referenced this issue May 6, 2022
@JonoYang
Update deprecated code #411
58d4909 
    * Normalize package_uids before comparing results in tests
    * Update expected test results

Signed-off-by: Jono Yang <jyang@nexb.com>
JonoYang added a commit that referenced this issue May 6, 2022
@JonoYang
Create DiscoveredDependency model #411
d62f75c 
    * Create functions to process dependencies from scans
    * Modify DiscoveredPackages so we can relate DiscoveredDependencies to it

Signed-off-by: Jono Yang <jyang@nexb.com>
tdruez added a commit that referenced this issue May 9, 2022
@tdruez
Add a test class to regen test data #411
e5b7696 
Signed-off-by: Thomas Druez <tdruez@nexb.com>
tdruez added a commit that referenced this issue May 9, 2022
@tdruez
Upgrade container_inspector to latest 31.0.0 version #411
3519009 
Signed-off-by: Thomas Druez <tdruez@nexb.com>
JonoYang added a commit that referenced this issue May 9, 2022
@JonoYang
Handle new scan format in scancode pipes #411
0110163 
Signed-off-by: Jono Yang <jyang@nexb.com>
JonoYang added a commit that referenced this issue May 9, 2022
@JonoYang
Handle package_uids for DiscoveredPackages #411
a029c26 
    * Remove create_discovered_packages2 and create_codebase_resources2

Signed-off-by: Jono Yang <jyang@nexb.com>
JonoYang added a commit that referenced this issue May 9, 2022
@JonoYang
Update deprecated code #411
774c5b7 
    * Normalize package_uids before comparing results in tests
    * Update expected test results

Signed-off-by: Jono Yang <jyang@nexb.com>
JonoYang added a commit that referenced this issue May 10, 2022
@JonoYang
Regenerate asgiref 3.3.0 test data #411
a6bffc5 
    * Mark ProjectCodebase tests with expectedFailure
    * We will revisit ProjectCodebase and update it to fit our current models

Signed-off-by: Jono Yang <jyang@nexb.com>
JonoYang added a commit that referenced this issue May 10, 2022
@JonoYang
Add asgiref-3.3.0_scancode_scan.json #411
4c354f1 
    * We are using a scancode scan results for tests since asgiref-3.3.0_scan.json is not exactly the same format as scancode's json output

Signed-off-by: Jono Yang <jyang@nexb.com>
JonoYang added a commit that referenced this issue Jun 13, 2022
@JonoYang
Consider all PURL fields when ordering Packages #411 #438
942e06d 
Signed-off-by: Jono Yang <jyang@nexb.com>31.0.0rc1
JonoYang added a commit that referenced this issue Jun 13, 2022
@JonoYang
Consider all PURL fields when ordering Packages #411 #438
9fcec67 
Signed-off-by: Jono Yang <jyang@nexb.com>
JonoYang added a commit that referenced this issue Jun 14, 2022
@JonoYang
Create Packages before Resources #411 #438
20ffbe0 
    * In the LoadInventory pipeline, create the DiscoveredPackages from a scan before creating the CodebaseResources

Signed-off-by: Jono Yang <jyang@nexb.com>
JonoYang added a commit that referenced this issue Jun 14, 2022
@JonoYang
Add test for load_inventory pipeline #411
bb44b4e 
Signed-off-by: Jono Yang <jyang@nexb.com>
JonoYang added a commit that referenced this issue Jun 14, 2022
@JonoYang
Add test for load_inventory pipeline #411
784dbbc 
Signed-off-by: Jono Yang <jyang@nexb.com>
tdruez added a commit that referenced this issue Jun 14, 2022
@tdruez
Code cleanups and formatting #411
d57afa6 
Signed-off-by: Thomas Druez <tdruez@nexb.com>
tdruez added a commit that referenced this issue Jun 14, 2022
@tdruez
Upgrade dependencies #411
413cf0d 
Signed-off-by: Thomas Druez <tdruez@nexb.com>
tdruez added a commit that referenced this issue Jun 14, 2022
@tdruez
Refactor create_inventory_from_scan to remove duplicated code #411
c8cb574 
Signed-off-by: Thomas Druez <tdruez@nexb.com>
tdruez added a commit that referenced this issue Jun 14, 2022
@tdruez
Add changelog entry #411
5aed5e8 
Signed-off-by: Thomas Druez <tdruez@nexb.com>
tdruez added a commit that referenced this issue Jun 14, 2022
@JonoYang @tdruez
Upgrade ScanCode-toolkit to version v31 #411
ba4695a 
* Upgrade scancode-toolkit to latest beta release #411

Signed-off-by: Thomas Druez <tdruez@nexb.com>

* Add a test class to regen test data #411

Signed-off-by: Thomas Druez <tdruez@nexb.com>

* Upgrade container_inspector to latest 31.0.0 version #411

Signed-off-by: Thomas Druez <tdruez@nexb.com>

* Handle new scan format in scancode pipes #411

Signed-off-by: Jono Yang <jyang@nexb.com>

* Handle package_uids for DiscoveredPackages #411

    * Remove create_discovered_packages2 and create_codebase_resources2

Signed-off-by: Jono Yang <jyang@nexb.com>

* Update deprecated code #411

    * Normalize package_uids before comparing results in tests
    * Update expected test results

Signed-off-by: Jono Yang <jyang@nexb.com>

* Regenerate asgiref 3.3.0 test data #411

    * Mark ProjectCodebase tests with expectedFailure
    * We will revisit ProjectCodebase and update it to fit our current models

Signed-off-by: Jono Yang <jyang@nexb.com>

* Add asgiref-3.3.0_scancode_scan.json #411

    * We are using a scancode scan results for tests since asgiref-3.3.0_scan.json is not exactly the same format as scancode's json output

Signed-off-by: Jono Yang <jyang@nexb.com>

* Add asgiref-3.3.0_walk_test_fixtures.json #411

    * Update regen_test_data.py to generate asgiref-3.3.0_walk_test_fixtures.json

Signed-off-by: Jono Yang <jyang@nexb.com>

* Signed-off-by: Jono Yang <jyang@nexb.com>

* Update make_results_summary() #411

    * No need to explicity get license_clarity_score in make_results_summary()
    * Update expected test results

Signed-off-by: Jono Yang <jyang@nexb.com>

* Exclude system_environment from diff #411

    * Add .vscode to .gitignore

Signed-off-by: Jono Yang <jyang@nexb.com>

* Upgrade scancode-toolkit and extractcode to latest version #411

Signed-off-by: Thomas Druez <tdruez@nexb.com>

* Update package_getter #434 #438

    * Adapt code from previous version of scancode-toolkit for use in the debian pipeline

Signed-off-by: Jono Yang <jyang@nexb.com>

* Allow packages to be created without versions #438

    * Update DiscoveredPackage.create_from_data to create packages without a version

Signed-off-by: Jono Yang <jyang@nexb.com>

* Update expected test results

Signed-off-by: Jono Yang <jyang@nexb.com>

* Report DiscoveredPackage correctly in summary #411

    * Ensure that DiscoveredPackages are reported one time in the scan_package pipeline summary
   * Add test to check key_file_packages field in the summary output

Signed-off-by: Jono Yang <jyang@nexb.com>

* Add test for docker pipeline for alpine #411

Signed-off-by: Jono Yang <jyang@nexb.com>

* Add docker pipeline test for rpm images #411

Signed-off-by: Jono Yang <jyang@nexb.com>

* Track package_uids in make_results_summary #435

    * Avoid checking if package_data dictionary is already in the key_files_packages list
    * Keep track of package_uids instead

Signed-off-by: Jono Yang <jyang@nexb.com>

* Add truncated ubuntu docker image for testing #435

Signed-off-by: Jono Yang <jyang@nexb.com>

* Bump scancode and commoncode versions #435

Signed-off-by: Jono Yang <jyang@nexb.com>

* Update docker pipeline #435

    * We now run scancode-toolkit on the docker image resources using the new --system-package option
    * This gives us the installed system packages in the returned scan
    * We use the scan to create the DiscoveredPackages and CodebaseResources
    * The rest of the pipeline is unchanged

Signed-off-by: Jono Yang <jyang@nexb.com>

* Fix code validity #411

Signed-off-by: Thomas Druez <tdruez@nexb.com>

* Simplify the filtering of key_files_packages using a QuerySet #411

Signed-off-by: Thomas Druez <tdruez@nexb.com>

* Remove copied code from docker.py #411 #435

    * Create Docker pipeline from combining the rootfs pipeline and scan_package pipeline

Signed-off-by: Jono Yang <jyang@nexb.com>

* Update alpine test image and results #411 #435

    * TODO: create smaller test images for ubuntu and redhat docker image tests

Signed-off-by: Jono Yang <jyang@nexb.com>

* Properly create multiple package instances #411

    * Do not attempt to combine multiple instances of the same package
    * Store package_uid in extra data by itself
    * Add test for multiple package instances

Signed-off-by: Jono Yang <jyang@nexb.com>

* Sort packages in JSON output by type and name #411

    * Normalize package_uid in extra_data fields

Signed-off-by: Jono Yang <jyang@nexb.com>

* Get file info and packages in initial scan #438

    * Remove step for scanning application packages

Signed-off-by: Jono Yang <jyang@nexb.com>

* Revert changes to docker pipes and pipeline #438

    * Check for existence of installed_file attribute before using it

Signed-off-by: Jono Yang <jyang@nexb.com>

* Use generic package_getter for all distros #438

    * Ensure both installed_file and codebase_resource have the same checksum field before comparing them

Signed-off-by: Jono Yang <jyang@nexb.com>

* Use get_path() with strip_root to get paths #438

    * Update mappings_keys_by_fieldname
    * Look for package data in package_data field instead of packages in save_scan_package_results

Signed-off-by: Jono Yang <jyang@nexb.com>

* Remove distro specific pipes #438

    * Move get_installed_packages to rootfs.py
    * Use get_package_data instead of get_package_info
    * Rename all instances of packages to package_data when scanning for application packages
    * Update test docker images and test results
    * Add test for basic rootfs

Signed-off-by: Jono Yang <jyang@nexb.com>

* Use list comprehension for key_file_packages #438

Signed-off-by: Jono Yang <jyang@nexb.com>

* Add package_uid field to DiscoveredPackage #411

    * Update expected test results

Signed-off-by: Jono Yang <jyang@nexb.com>

* Add test docker image for Ubuntu #438

    * Update expected test results
    * Remove old ubuntu.tar

Signed-off-by: Jono Yang <jyang@nexb.com>

* Update formatting #411 #438

Signed-off-by: Jono Yang <jyang@nexb.com>

* Use smaller rpm docker image for testing #438

Signed-off-by: Jono Yang <jyang@nexb.com>

* Replace ubuntu docker test image #438

Signed-off-by: Jono Yang <jyang@nexb.com>

* Use purl data in update_or_create_packages #438

    * Add package_uid to test package data
    * Update expected test result

Signed-off-by: Jono Yang <jyang@nexb.com>

* Bump scancode version to v31.0.0rc1 #438 #411

Signed-off-by: Jono Yang <jyang@nexb.com>

* Consider all PURL fields when ordering Packages #411 #438

Signed-off-by: Jono Yang <jyang@nexb.com>

* Create Packages before Resources  #411 #438

    * In the LoadInventory pipeline, create the DiscoveredPackages from a scan before creating the CodebaseResources

Signed-off-by: Jono Yang <jyang@nexb.com>

* Add test for load_inventory pipeline #411

Signed-off-by: Jono Yang <jyang@nexb.com>

* Code cleanups and formatting #411

Signed-off-by: Thomas Druez <tdruez@nexb.com>

* Upgrade dependencies #411

Signed-off-by: Thomas Druez <tdruez@nexb.com>

* Refactor create_inventory_from_scan to remove duplicated code #411

Signed-off-by: Thomas Druez <tdruez@nexb.com>

* Add changelog entry #411

Signed-off-by: Thomas Druez <tdruez@nexb.com>

Co-authored-by: Thomas Druez <tdruez@nexb.com>
JonoYang added a commit that referenced this issue Jun 17, 2022
@JonoYang
Show license clarity data in project view
4d83798 
Reference: #411

Signed-off-by: Jono Yang <jyang@nexb.com>
JonoYang added a commit that referenced this issue Jun 17, 2022
@JonoYang
Add styling to license clarity data table
c2961e1 
    * Update help text

Reference: #411

Signed-off-by: Jono Yang <jyang@nexb.com>
JonoYang added a commit that referenced this issue Jun 17, 2022
@JonoYang
Pass more summary data to project view
6c69aaa 
    * Add fontawesome v 5.13.0

Reference: #411

Signed-off-by: Jono Yang <jyang@nexb.com>
JonoYang added a commit that referenced this issue Jun 17, 2022
@JonoYang
Show summary data in transposed table
22e64af 
Reference: #411

Signed-off-by: Jono Yang <jyang@nexb.com>
JonoYang added a commit that referenced this issue Jun 17, 2022
@JonoYang
Pass summary data to view in a dictionary
d46d5e4 
Reference: #411

Signed-off-by: Jono Yang <jyang@nexb.com>
JonoYang added a commit that referenced this issue Jun 17, 2022
@JonoYang
Display summary data using lists
24bc656 
Reference: #411

Signed-off-by: Jono Yang <jyang@nexb.com>
JonoYang added a commit that referenced this issue Jun 17, 2022
@JonoYang
Separate code into methods
147dd24 
Reference: #411

Signed-off-by: Jono Yang <jyang@nexb.com>
JonoYang added a commit that referenced this issue Jun 17, 2022
@JonoYang
Show new license clarity score fields
94daf79 
Reference: #411
Signed-off-by: Jono Yang <jyang@nexb.com>
JonoYang added a commit that referenced this issue Jun 17, 2022
@JonoYang
Show license clarity data in project view
1e5d1cf 
Reference: #411

Signed-off-by: Jono Yang <jyang@nexb.com>
JonoYang added a commit that referenced this issue Jun 17, 2022
@JonoYang
Add styling to license clarity data table
3f71faa 
    * Update help text

Reference: #411

Signed-off-by: Jono Yang <jyang@nexb.com>
JonoYang added a commit that referenced this issue Jun 17, 2022
@JonoYang
Pass more summary data to project view
9f2679d 
    * Add fontawesome v 5.13.0

Reference: #411

Signed-off-by: Jono Yang <jyang@nexb.com>
JonoYang added a commit that referenced this issue Jun 17, 2022
@JonoYang
Show summary data in transposed table
b71b6fb 
Reference: #411

Signed-off-by: Jono Yang <jyang@nexb.com>
JonoYang added a commit that referenced this issue Jun 17, 2022
@JonoYang
Pass summary data to view in a dictionary
da29af7 
Reference: #411

Signed-off-by: Jono Yang <jyang@nexb.com>
JonoYang added a commit that referenced this issue Jun 17, 2022
@JonoYang
Display summary data using lists
4e151fb 
Reference: #411

Signed-off-by: Jono Yang <jyang@nexb.com>
JonoYang added a commit that referenced this issue Jun 17, 2022
@JonoYang
Separate code into methods
2bfe10c 
Reference: #411

Signed-off-by: Jono Yang <jyang@nexb.com>
JonoYang added a commit that referenced this issue Jun 17, 2022
@JonoYang
Show new license clarity score fields
8306869 
Reference: #411
Signed-off-by: Jono Yang <jyang@nexb.com>
JonoYang added a commit that referenced this issue Jul 14, 2022
@JonoYang
Update license clarity field help text #411
1f0efd5 
Signed-off-by: Jono Yang <jyang@nexb.com>
JonoYang added a commit that referenced this issue Jul 15, 2022
@JonoYang
Update license clarity field help text #411
d83d73f 
    * Clean logic in get_license_clarity_data

Signed-off-by: Jono Yang <jyang@nexb.com>
tdruez pushed a commit that referenced this issue Jul 19, 2022
@JonoYang
Display "License clarity" and "Scan summary" values in the project de…
7f3de27 
…tails view #411

Signed-off-by: Thomas Druez <tdruez@nexb.com>
@tdruez tdruez closed this as completed Jul 19, 2022
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees

@JonoYang JonoYang

Labels
None yet
Projects
None yet
Milestone
No milestone
Development

No branches or pull requests
2 participants
@tdruez @JonoYang