Implement new library loading approach #20
Conversation
Signed-off-by: Philippe Ombredanne <pombredanne@nexb.com>
Signed-off-by: Philippe Ombredanne <pombredanne@nexb.com>
Signed-off-by: Steven Esser <sesser@nexb.com>
Fix .gitattributes
This allows scancode to work on platforms not supported by typecode-libmagic e.g. M1 Macs or windows WSL Most of this code is from https://github.com/ahupp/python-magic/blob/e0ccc6d/magic/loader.py by Adam Hupp, MIT License, compatible with typecode's Apache Signed-off-by: Pierre Tardy <pierre.tardy@renault.com>
Use virtualenv-embedded libraries
Signed-off-by: Philippe Ombredanne <pombredanne@nexb.com>
This will work even from a git archive or when git is not installed. Signed-off-by: Philippe Ombredanne <pombredanne@nexb.com>
Signed-off-by: Philippe Ombredanne <pombredanne@nexb.com>
Signed-off-by: Philippe Ombredanne <pombredanne@nexb.com>
Signed-off-by: Philippe Ombredanne <pombredanne@nexb.com>
Signed-off-by: Philippe Ombredanne <pombredanne@nexb.com>
Signed-off-by: Philippe Ombredanne <pombredanne@nexb.com>
Signed-off-by: Philippe Ombredanne <pombredanne@nexb.com>
This was they do not end up in the template CHANGELOG.rst Signed-off-by: Philippe Ombredanne <pombredanne@nexb.com>
We now load native libraries and executables from: 1. an envt. variable path 2. OR a locatin provider plugin 3. OR the PATH or we fail Signed-off-by: Philippe Ombredanne <pombredanne@nexb.com>
New configure
Signed-off-by: Philippe Ombredanne <pombredanne@nexb.com>
Signed-off-by: Philippe Ombredanne <pombredanne@nexb.com>
Signed-off-by: Philippe Ombredanne <pombredanne@nexb.com>
Signed-off-by: Philippe Ombredanne <pombredanne@nexb.com>
Signed-off-by: Philippe Ombredanne <pombredanne@nexb.com>
Signed-off-by: Philippe Ombredanne <pombredanne@nexb.com>
Its was skipping plugin loading. Also add tracing. Signed-off-by: Philippe Ombredanne <pombredanne@nexb.com>
Only do content-based detection for programing language if there is no file extension as this avoids a large number of detection issues. Signed-off-by: Philippe Ombredanne <pombredanne@nexb.com>
Aslo ensure we use assertion in the correct order. Signed-off-by: Philippe Ombredanne <pombredanne@nexb.com>
Signed-off-by: Philippe Ombredanne <pombredanne@nexb.com>
Signed-off-by: Philippe Ombredanne <pombredanne@nexb.com>
Signed-off-by: Philippe Ombredanne <pombredanne@nexb.com>
We now load the libmagic native library and its database from paths found in: 1. environment variables, 2. OR a location provider plugin, 3. OR the system PATH, 4. OR we fail with an informative error message. Based on original code contributed by @priv-kweihmann to scancode-plugins in aboutcode-org/scancode-plugins#9 and moved here and adapted for use in the core code rather than in a plugin. Contributed-by: Konrad Weihmann <kweihmann@outlook.com> Signed-off-by: Philippe Ombredanne <pombredanne@nexb.com>
Signed-off-by: Philippe Ombredanne <pombredanne@nexb.com>
Signed-off-by: Philippe Ombredanne <pombredanne@nexb.com>
Signed-off-by: Philippe Ombredanne <pombredanne@nexb.com>
Signed-off-by: Philippe Ombredanne <pombredanne@nexb.com>
|
@tardyp I merged 17 in this branch which also carries over in a single place the code originally contributed by @priv-kweihmann for the Getting both of your reviews would be great before I merge |
pombredanne
changed the title
pombredanne
changed the title
Signed-off-by: Philippe Ombredanne <pombredanne@nexb.com>
There was a problem hiding this comment.
Hello,
Thanks for pinging me.
I am having hard time reviewing this 272 files PR.
Franckly I am not a huge fan of vendoring for opensource projects, although I do understand the need to limit dependency variablility for support reason.
I didn't file too much issue though, but I cannot claim a deep review. :-)
configure.bat
Outdated
| set PYTHON_EXECUTABLE=%PROVIDED_PYTHON% | ||
| goto run | ||
| if "%1" EQU "--python" ( | ||
| echo "The --python is now DEPRECATED. Use the PYTHON_EXECUTABLE environment |
There was a problem hiding this comment.
Thanks. Fixed in https://github.com/nexB/skeleton and merged here
| logger_debug('get_magicdb_location:', 'got path magicdb location:', magicdb_loc) | ||
|
|
||
| if not magicdb_loc: | ||
| warnings.warn( |
There was a problem hiding this comment.
I think this not needed as libmagic will (probably) already complain if it does not find its db in standard location.
There was a problem hiding this comment.
Good point... I like keeping the warning on our side too though as the message would be more explicit
Signed-off-by: Philippe Ombredanne <pombredanne@nexb.com>
Signed-off-by: Philippe Ombredanne <pombredanne@nexb.com>
From #20 Reported-by: Pierre Tardy <pierre.tardy@renault.com> Signed-off-by: Philippe Ombredanne <pombredanne@nexb.com>
Signed-off-by: Philippe Ombredanne <pombredanne@nexb.com>
|
@tardyp you wrote:
I agree yet the Pygments vendoring here is a must here as we have dependencies on specific versions and vendoring is the sane approach to avoid conflicts (that popped up often in the past) when this fairly common Pygments library is installed otherwise. |
|
@tardyp note also that we should be able to support Apple ARM chips as soon as I can find some CI supporting it. |
And remove v prefix from fallback version Signed-off-by: Philippe Ombredanne <pombredanne@nexb.com>
Signed-off-by: Philippe Ombredanne <pombredanne@nexb.com>
|
This looks decent now. Merging and releasing! |
We now load the libmagic native library and its database from paths found in:
Based on original code contributed by @priv-kweihmann to scancode-plugins
in aboutcode-org/scancode-plugins#9 and
moved here and adapted for use in the core code rather than in a plugin.
This also includes the merge of @tardyp #17 to search for common locations as a failover
Signed-off-by: Philippe Ombredanne pombredanne@nexb.com
Signed-off-by: Philippe Ombredanne pombredanne@nexb.com