Accesses gemfileparser which is not installed

[danielthieleog]

Description

Installation of scancode-toolkit using pip (version 32.2.1) also installs gemfileparser2, but code still depends on gemfileparser, which is not installed automatically.
File spec.py imports from gemfileparser, not from gemfileparser2.

How To Reproduce

In a python environment with all the necessary OS libraries, e. g. a docker container for image python:3.9-slim:

# I started in a directory /app
# mkdir /app
# cd /app

python -m venv venv-scancode
. venv-scancode/bin/activate
pip install --upgrade pip
pip install wheel
pip install scancode-toolkit

./venv-scancode/bin/scancode --example

Results in

========================================================================
ERROR: UNABLE TO IMPORT SCANCODE PLUGINS.
Check your installation configuration (setup.py) or re-install/re-configure ScanCode.
The following plugin(s) are referenced and cannot be loaded/imported:
No module named 'gemfileparser'
========================================================================
Traceback (most recent call last):
  File "/app/./venv-scancode/bin/scancode", line 5, in <module>
    from scancode.cli import scancode
  File "/app/venv-scancode/lib/python3.9/site-packages/scancode/cli.py", line 126, in <module>
    raise e
  File "/app/venv-scancode/lib/python3.9/site-packages/scancode/cli.py", line 113, in <module>
    plugin_classes, plugin_options = PluginManager.load_plugins()
  File "/app/venv-scancode/lib/python3.9/site-packages/plugincode/__init__.py", line 196, in load_plugins
    mgr_setup = manager.setup()
  File "/app/venv-scancode/lib/python3.9/site-packages/plugincode/__init__.py", line 222, in setup
    raise e
  File "/app/venv-scancode/lib/python3.9/site-packages/plugincode/__init__.py", line 220, in setup
    self.manager.load_setuptools_entrypoints(entrypoint)
  File "/app/venv-scancode/lib/python3.9/site-packages/pluggy/_manager.py", line 287, in load_setuptools_entrypoints
    plugin = ep.load()
  File "/usr/local/lib/python3.9/importlib/metadata.py", line 86, in load
    module = import_module(match.group('module'))
  File "/usr/local/lib/python3.9/importlib/__init__.py", line 127, in import_module
    return _bootstrap._gcd_import(name[level:], package, level)
  File "/app/venv-scancode/lib/python3.9/site-packages/packagedcode/__init__.py", line 24, in <module>
    from packagedcode import cocoapods
  File "/app/venv-scancode/lib/python3.9/site-packages/packagedcode/cocoapods.py", line 21, in <module>
    from packagedcode import spec
  File "/app/venv-scancode/lib/python3.9/site-packages/packagedcode/spec.py", line 12, in <module>
    from gemfileparser import GemfileParser
ModuleNotFoundError: No module named 'gemfileparser'

After

pip install gemfileparser

calling scancode again succeeds.

System configuration

• Linux
• scancode-toolkit version 32.2.1
• installation method pip

[pombredanne]

@danielthieleog thank you for the report. As it happens this has just been fixed in a branch in https://github.com/nexB/scancode-toolkit/tree/new-ruby-lexer

There is also extensive work towards a new, better parser for gemspec/Gemfile/podspec/Podfile manifests.

[pombredanne]

If this is a blocker for you in the very short I could backport just this fix and trigger a release.

[danielthieleog]

@pombredanne , thank you for taking care. Currently it is OK for me to install gemfileparser additionally. Will the fix be part of the next regular release?

[pombredanne]

@danielthieleog you wrote:

thank you for taking care.

Thank you for using the tool!

Currently it is OK for me to install gemfileparser additionally. > Will the fix be part of the next regular release?

Yes.

[pombredanne]

@danielthieleog @tdruez @multiplemonomials this is now fixed by pushing a fix for backward compat to https://github.com/nexB/gemfileparser2/releases/tag/v0.9.3

[pombredanne]

Note that the only cure I can think of is this:

• Run everyday a schedule CI test that will install scancode-toolkit using pip therefore using the latest compatible version of the deps as released on PyPI.

• Run this for the latest tagged release and the head of of the main branch.

• Run this as part of the standard test suite on every commits.

[pombredanne]

@danielthieleog I am keeping this open for now until we have tests for this to ensure there will be no future regression.