Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Incomplete/Incorrect result for yarn.lock and package-lock.json from package scan for pacakge that have '@' in namespace #1993

Closed
chinyeungli opened this issue Apr 2, 2020 · 1 comment · Fixed by #2010
Closed

Incomplete/Incorrect result for yarn.lock and package-lock.json from package scan for pacakge that have '@' in namespace #1993

chinyeungli opened this issue Apr 2, 2020 · 1 comment · Fixed by #2010
Labels
bug must have

Comments

@chinyeungli
Copy link
Contributor

Description

With the following yarn.lock context

# THIS IS AN AUTOGENERATED FILE. DO NOT EDIT THIS FILE DIRECTLY.
# yarn lockfile v1


agent-base@^4.1.0:
  version "4.2.1"
  resolved "https://registry.yarnpkg.com/agent-base/-/agent-base-4.2.1.tgz#d89e5999f797875674c07d87f260fc41e83e8ca9"
  integrity sha512-JVwXMr9nHYTUXsBFKUqhJwvlcYU/blreOEUkhNR2eXZIvwd+c+o5V4MgDPKWnMS/56awN3TRzIP+KoPn+roQtg==
  dependencies:
    es6-promisify "^5.0.0"

es6-promise@^4.0.3:
  version "4.2.5"
  resolved "https://registry.yarnpkg.com/es6-promise/-/es6-promise-4.2.5.tgz#da6d0d5692efb461e082c14817fe2427d8f5d054"
  integrity sha512-n6wvpdE43VFtJq+lUDYDBFUwV8TZbuGXLV4D6wKafg13ldznKsyEvatubnmUe31zcvelSzOHF+XbaT+Bl9ObDg==

es6-promisify@^5.0.0:
  version "5.0.0"
  resolved "https://registry.yarnpkg.com/es6-promisify/-/es6-promisify-5.0.0.tgz#5109d62f3e56ea967c4b63505aef08291c8a5203"
  integrity sha1-UQnWLz5W6pZ8S2NQWu8IKRyKUgM=
  dependencies:
    es6-promise "^4.0.3"

"@angular-builders/custom-webpack@^7.0.0":
  version "7.5.2"
  resolved "https://registry.yarnpkg.com/@angular-builders/custom-webpack/-/custom-webpack-7.5.2.tgz#de475d211bbb2838cbdee5744432f2eaaab787b0"
  dependencies:
    lodash "^4.17.10"

lodash@^4.0.0, lodash@^4.13.1, lodash@^4.15.0, lodash@^4.17.10, lodash@^4.17.11, lodash@^4.17.13, lodash@^4.17.14, lodash@^4.17.4, lodash@^4.17.5, lodash@^4.5.0, lodash@~4.17.10, lodash@~4.17.11:
  version "4.17.15"
  resolved "https://registry.yarnpkg.com/lodash/-/lodash-4.17.15.tgz#b447f6670a0455bbfeedd11392eff330ea097548"

The tool parses the first 3 alright. However, it fails to parse the forth element "@angular-builders/custom-webpack@^7.0.0":, correctly. The following is the result:

        {
          "type": "npm",
          "namespace": null,
          "name": "resolved \"https://registry.yarnpkg.com/",
          "version": "7.5.2",

The expected result should be

        {
          "type": "npm",
          "namespace": "@angular-builders",
          "name": "custom-webpack",
          "version": "7.5.2",

In addition, the lodash does not exist in the output as well.

Following is the complete output from the package scan for the above yarn.lock file:

{
  "headers": [
    {
      "tool_name": "scancode-toolkit",
      "tool_version": "3.1.1.post381.4a5c596a2.dirty.20200330020342",
      "options": {
        "input": [
          "C:\\tmp\\yarn.lock"
        ],
        "--csv": "C:\\tmp\\lock-info.csv",
        "--json-pp": "C:\\tmp\\lock-info.json",
        "--package": true
      },
      "notice": "Generated with ScanCode and provided on an \"AS IS\" BASIS, WITHOUT WARRANTIES\nOR CONDITIONS OF ANY KIND, either express or implied. No content created from\nScanCode should be considered or used as legal advice. Consult an Attorney\nfor any legal advice.\nScanCode is a free software code scanning tool from nexB Inc. and others.\nVisit https://github.com/nexB/scancode-toolkit/ for support and download.",
      "start_timestamp": "2020-04-02T015638.986000",
      "end_timestamp": "2020-04-02T015653.364000",
      "duration": 14.378000020980835,
      "message": null,
      "errors": [],
      "extra_data": {
        "files_count": 1
      }
    }
  ],
  "files": [
    {
      "path": "yarn.lock",
      "type": "file",
      "packages": [
        {
          "type": "npm",
          "namespace": null,
          "name": null,
          "version": null,
          "qualifiers": {},
          "subpath": null,
          "primary_language": "JavaScript",
          "description": null,
          "release_date": null,
          "parties": [],
          "keywords": [],
          "homepage_url": null,
          "download_url": null,
          "size": null,
          "sha1": null,
          "md5": null,
          "sha256": null,
          "sha512": null,
          "bug_tracking_url": null,
          "code_view_url": null,
          "vcs_url": null,
          "copyright": null,
          "license_expression": null,
          "declared_license": null,
          "notice_text": null,
          "root_path": null,
          "dependencies": [
            {
              "purl": "pkg:npm/agent-base@4.2.1",
              "requirement": "^4.1.0",
              "scope": "dependencies",
              "is_runtime": true,
              "is_optional": false,
              "is_resolved": true
            },
            {
              "purl": "pkg:npm/es6-promise@4.2.5",
              "requirement": "^4.0.3",
              "scope": "dependencies",
              "is_runtime": true,
              "is_optional": false,
              "is_resolved": true
            },
            {
              "purl": "pkg:npm/es6-promisify@5.0.0",
              "requirement": "^5.0.0",
              "scope": "dependencies",
              "is_runtime": true,
              "is_optional": false,
              "is_resolved": true
            },
            {
              "purl": "pkg:npm/resolved%20%22https://registry.yarnpkg.com@7.5.2",
              "requirement": "angular-builders/custom-webpack/-/custom-webpack-7.5.2.tgz#de475d211bbb2838cbdee5744432f2eaaab787b0",
              "scope": "dependencies",
              "is_runtime": true,
              "is_optional": false,
              "is_resolved": true
            }
          ],
          "contains_source_code": null,
          "source_packages": [],
          "purl": null,
          "repository_homepage_url": "https://www.npmjs.com/package/None",
          "repository_download_url": "https://registry.npmjs.org/None/-/None-None.tgz",
          "api_data_url": "https://registry.npmjs.org/None"
        },
        {
          "type": "npm",
          "namespace": null,
          "name": "agent-base",
          "version": "4.2.1",
          "qualifiers": {},
          "subpath": null,
          "primary_language": "JavaScript",
          "description": null,
          "release_date": null,
          "parties": [],
          "keywords": [],
          "homepage_url": null,
          "download_url": "https://registry.yarnpkg.com/agent-base/-/agent-base-4.2.1.tgz#d89e5999f797875674c07d87f260fc41e83e8ca9",
          "size": null,
          "sha1": null,
          "md5": null,
          "sha256": null,
          "sha512": null,
          "bug_tracking_url": null,
          "code_view_url": null,
          "vcs_url": null,
          "copyright": null,
          "license_expression": null,
          "declared_license": null,
          "notice_text": null,
          "root_path": null,
          "dependencies": [
            {
              "purl": "pkg:npm/es6-promisify",
              "requirement": "^5.0.0",
              "scope": "dependencies",
              "is_runtime": true,
              "is_optional": false,
              "is_resolved": true
            }
          ],
          "contains_source_code": null,
          "source_packages": [],
          "purl": "pkg:npm/agent-base@4.2.1",
          "repository_homepage_url": "https://www.npmjs.com/package/agent-base",
          "repository_download_url": "https://registry.npmjs.org/agent-base/-/agent-base-4.2.1.tgz",
          "api_data_url": "https://registry.npmjs.org/agent-base/4.2.1"
        },
        {
          "type": "npm",
          "namespace": null,
          "name": "es6-promise",
          "version": "4.2.5",
          "qualifiers": {},
          "subpath": null,
          "primary_language": "JavaScript",
          "description": null,
          "release_date": null,
          "parties": [],
          "keywords": [],
          "homepage_url": null,
          "download_url": "https://registry.yarnpkg.com/es6-promise/-/es6-promise-4.2.5.tgz#da6d0d5692efb461e082c14817fe2427d8f5d054",
          "size": null,
          "sha1": null,
          "md5": null,
          "sha256": null,
          "sha512": null,
          "bug_tracking_url": null,
          "code_view_url": null,
          "vcs_url": null,
          "copyright": null,
          "license_expression": null,
          "declared_license": null,
          "notice_text": null,
          "root_path": null,
          "dependencies": [],
          "contains_source_code": null,
          "source_packages": [],
          "purl": "pkg:npm/es6-promise@4.2.5",
          "repository_homepage_url": "https://www.npmjs.com/package/es6-promise",
          "repository_download_url": "https://registry.npmjs.org/es6-promise/-/es6-promise-4.2.5.tgz",
          "api_data_url": "https://registry.npmjs.org/es6-promise/4.2.5"
        },
        {
          "type": "npm",
          "namespace": null,
          "name": "es6-promisify",
          "version": "5.0.0",
          "qualifiers": {},
          "subpath": null,
          "primary_language": "JavaScript",
          "description": null,
          "release_date": null,
          "parties": [],
          "keywords": [],
          "homepage_url": null,
          "download_url": "https://registry.yarnpkg.com/es6-promisify/-/es6-promisify-5.0.0.tgz#5109d62f3e56ea967c4b63505aef08291c8a5203",
          "size": null,
          "sha1": null,
          "md5": null,
          "sha256": null,
          "sha512": null,
          "bug_tracking_url": null,
          "code_view_url": null,
          "vcs_url": null,
          "copyright": null,
          "license_expression": null,
          "declared_license": null,
          "notice_text": null,
          "root_path": null,
          "dependencies": [
            {
              "purl": "pkg:npm/es6-promise",
              "requirement": "^4.0.3",
              "scope": "dependencies",
              "is_runtime": true,
              "is_optional": false,
              "is_resolved": true
            }
          ],
          "contains_source_code": null,
          "source_packages": [],
          "purl": "pkg:npm/es6-promisify@5.0.0",
          "repository_homepage_url": "https://www.npmjs.com/package/es6-promisify",
          "repository_download_url": "https://registry.npmjs.org/es6-promisify/-/es6-promisify-5.0.0.tgz",
          "api_data_url": "https://registry.npmjs.org/es6-promisify/5.0.0"
        },
        {
          "type": "npm",
          "namespace": null,
          "name": "resolved \"https://registry.yarnpkg.com/",
          "version": "7.5.2",
          "qualifiers": {},
          "subpath": null,
          "primary_language": "JavaScript",
          "description": null,
          "release_date": null,
          "parties": [],
          "keywords": [],
          "homepage_url": null,
          "download_url": "https://registry.yarnpkg.com/@angular-builders/custom-webpack/-/custom-webpack-7.5.2.tgz#de475d211bbb2838cbdee5744432f2eaaab787b0",
          "size": null,
          "sha1": null,
          "md5": null,
          "sha256": null,
          "sha512": null,
          "bug_tracking_url": null,
          "code_view_url": null,
          "vcs_url": null,
          "copyright": null,
          "license_expression": null,
          "declared_license": null,
          "notice_text": null,
          "root_path": null,
          "dependencies": [
            {
              "purl": "pkg:npm/lodash",
              "requirement": "^4.17.10",
              "scope": "dependencies",
              "is_runtime": true,
              "is_optional": false,
              "is_resolved": true
            }
          ],
          "contains_source_code": null,
          "source_packages": [],
          "purl": "pkg:npm/resolved%20%22https://registry.yarnpkg.com@7.5.2",
          "repository_homepage_url": "https://www.npmjs.com/package/resolved \"https://registry.yarnpkg.com/",
          "repository_download_url": "https://registry.npmjs.org/resolved \"https://registry.yarnpkg.com//-/resolved \"https://registry.yarnpkg.com/-7.5.2.tgz",
          "api_data_url": "https://registry.npmjs.org/resolved \"https://registry.yarnpkg.com//7.5.2"
        }
      ],
      "scan_errors": []
    }
  ]
}
@chinyeungli chinyeungli changed the title Imcomplete result for yarn.lock from package scan Incomplete/Incorrect result for yarn.lock from package scan Apr 2, 2020
chinyeungli added a commit that referenced this issue Apr 3, 2020
@chinyeungli
potential fix for #1993
6d54855 
 * The tool should now be able to parse the packages with namespace in
yarn.lock correctly. Need to check for others such as package.lock etc.
@chinyeungli chinyeungli changed the title Incomplete/Incorrect result for yarn.lock from package scan Incomplete/Incorrect result for yarn.lock and package-lock.json from package scan for pacakge that have '@' in namespace Apr 9, 2020
@chinyeungli
Copy link
Contributor Author

For the package-lock.json
"@angular/animations",
is parsed as

          "namespace": null,
          "name": "@angular/animations",

The @angular should fall in "namespace" and the "name" should be animations only

In addition,
the purl from the json output is

            {
              "purl": "pkg:npm/%40angular/animations@7.2.15",
              "requirement": null,
              "scope": "dependencies",
              "is_runtime": true,
              "is_optional": false,
              "is_resolved": true
            },

which is incorrect as the @ sign has been translated to %40

chinyeungli added a commit that referenced this issue Apr 13, 2020
@chinyeungli
Fixed #1993 correct namespace for package-lock.json
210ec0f 
 * Parse the namespace data for package-lock.json

The weird (encoding related) character in purl has to be fixed in
package_url.
@chinyeungli chinyeungli mentioned this issue Apr 14, 2020
Merged
4 tasks
chinyeungli added a commit that referenced this issue Apr 14, 2020
@chinyeungli
potential fix for #1993
daf67fe 
 * The tool should now be able to parse the packages with namespace in
yarn.lock correctly. Need to check for others such as package.lock etc.

Signed-off-by: Chin Yeung Li <tli@nexb.com>
chinyeungli added a commit that referenced this issue Apr 14, 2020
@chinyeungli
Fixed #1993 correct namespace for package-lock.json
3af6f67 
 * Parse the namespace data for package-lock.json

The weird (encoding related) character in purl has to be fixed in
package_url.

Signed-off-by: Chin Yeung Li <tli@nexb.com>
chinyeungli added a commit that referenced this issue Apr 14, 2020
@chinyeungli
Fixed #1993 - Update the partition statements into single statements
426b74b
chinyeungli added a commit that referenced this issue Apr 14, 2020
@chinyeungli
Fixed #1993 - Update the partition statements into single statements
eb9f025 
Signed-off-by: Chin Yeung Li <tli@nexb.com>
pombredanne added a commit that referenced this issue Apr 15, 2020
@pombredanne
Merge pull request #2010 from nexB/1993_package_scan_bug_for_lock_files
a8125da 
Collect yarn.lock dependencies correctly #1993
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees
No one assigned
Labels
bug must have
Projects
None yet
Milestone
No milestone
Development

Successfully merging a pull request may close this issue.

Collect yarn.lock dependencies correctly #1993 aboutcode-org/scancode-toolkit
1 participant
@chinyeungli