Use paths in codebase

[pombredanne]

This PR merges the latest skeleton and implements new internals for resource Codebase and Resource.
The key change is dropping using numeric resource ids and using a simpler map of path->Resource instead.
The not-yet-implemented part is focusing a codebase on the subset the new Codebase(paths) argument.
This is still a work in progress because of this. But early feedback is welcomed and needed.

[JonoYang]

I'm running into errors when using the VirtualCodebase. I am passing in two JSON scans into VirtualCodebase to test the functionality we have where we can create a VirtualCodebase from multiple scans. I am able to create the VirtualCodebase but using the walk() method causes an exception to be raised:

Traceback (most recent call last):
  File "/home/jono/nexb/src/commoncode/src/commoncode/resource.py", line 1228, in children
    return sorted(children, key=_sorter)
  File "/home/jono/nexb/src/commoncode/src/commoncode/resource.py", line 1220, in <lambda>
    _sorter = lambda r: (r.has_children(), r.name.lower(), r.name)
AttributeError: 'NoneType' object has no attribute 'has_children'

The above exception was the direct cause of the following exception:

Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
  File "/home/jono/nexb/src/commoncode/src/commoncode/resource.py", line 867, in walk
    for res in root.walk(self, topdown=topdown, ignored=ignored):
  File "/home/jono/nexb/src/commoncode/src/commoncode/resource.py", line 1196, in walk
    ignored=ignored,
  File "/home/jono/nexb/src/commoncode/src/commoncode/resource.py", line 1187, in walk
    for child in self.children(codebase):
  File "/home/jono/nexb/src/commoncode/src/commoncode/resource.py", line 1230, in children
    raise Exception(f'Cannot sort children: {children!r}:\n{children_paths!r}') from e
Exception: Cannot sort children: [None, None]:
['codebase/package', 'codebase/django-audit-tools-0.4.0']

I took a look at the paths of the VirtualCodebase I created by using the resources_by_path attribute and I saw that the paths root Resources of the two scans I used do not start with virtual_root, whereas all of the other Resources in the VirtualCodebase do. I think that the exception is occurring because the Resources doesn't have virtual_root in its path and it cannot find any other children because of the difference in the path prefix.

[pombredanne]

I'm running into errors when using the VirtualCodebase. I am passing in two JSON scans into VirtualCodebase to test the functionality we have where we can create a VirtualCodebase from multiple scans. I am able to create the VirtualCodebase but using the walk() method causes an exception to be raised:

@JonoYang the latest push is fixing this. And several other issues. Thanks!

[AyanSinhaMahapatra]

Looks great! Just a few nits.

[AyanSinhaMahapatra]

We could add a line about paths in the docstring.

[JonoYang]

I think it looks okay. I like that you set the iter dunder on Codebase to yield its resources.

I feel that the solution we have for creating a codebase from multiple scans is a bit kludgy, but reasonable. Treating each scan as an individual codebase is easier than attempting to always merge scan directory structure/data together. I had not realized the implication of getting scancode to merge two directories named codebase from two different scans.

On a whim, I've updated commoncode in scancode and all the references to rid in scancode/cli.py will need to be updated.