Fail to scan on user home with ~

[pombredanne]

Description

A scan to ~/ fails:

$ ./scancode -i ~/misc/licenses/ --json -
Setup plugins...
Collect file inventory...
Removing temporary files...done.
Usage: scancode [OPTIONS] <OUTPUT FORMAT OPTION(s)> <input>...
Try 'scancode --help' for help on options and arguments.

Error: Invalid value: ERROR: failed to collect codebase at: '/home/misc/licenses/'
Traceback (most recent call last):
  File "/scancode-toolkit/src/scancode/cli.py", line 860, in run_scan
    max_in_memory=max_in_memory
  File "/scancode-toolkit/src/scancode/resource.py", line 284, in __init__
    self._populate()
  File "/scancode-toolkit/src/scancode/resource.py", line 443, in _populate
    parent = parent_by_loc.pop(top)
KeyError: '/home/licenses/javascript/rcs

System configuration

For bug reports, it really helps us to know:

• What OS are you running on? Linux
• What version of scancode-toolkit was used to generate the scan file? https://github.com/nexB/scancode-toolkit/tree/2529cd7e2b3f9e41522d657fe26697be142b4645
• What installation method was used to install/run scancode? git clone

[JonoYang]

@pombredanne Is scancode failing to scan a folder in your home directory? I am unable to reproduce the error.

[pombredanne]

@JonoYang good catch... this is not the root cause indeed.
The small zip attached fails with:

unzip scan.zip
scancode --json-pp - scan

scan.zip

[JonoYang]

@pombredanne Thanks, I am able to reproduce the error with the attached zip.

[JonoYang]

@pombredanne The issue is with the directory scan.zip/rcs/. ScanCode will ignore the rcs directory since it's shares the same name as RCS, the version control system.

[pombredanne]

OK, I think we can now drop ignoring RCS ten... I have not seen RCS directories in ages, and digging in them if we ever meet some real ones is not a problem IMHO.

We still need to avoid this kind of errors too BEFORE removing skipping rcs

[JonoYang]

We now scan the RCS directory and we properly ignore directories when creating the codebase. Closing.

[adambhere]

I've come across this same problem running scancode 3.0.2 on ubuntu 18.04. The directory named "rcs" was ignored by scancode, but not its contents. It seems to me that it might be a good idea not only to remove "RCS" from the ignored list but also to make checks case sensitive (except on Windows) -- get_matches() in fileste.py changes the case of the item to be ignored to lower and I suggest it shouldn't.

[pombredanne]

@adambhere Let's create a new ticket for that then. Do you have the same problem using the tip of the develop branh?

[adambhere]

@pombredanne We have plans to move to it, but the decision is out of my hands.