How to use --extract on Linux-32 ? What extract archive mean?

[K-Rex]

I want to use -c -l to scan, but it scan all the files like .c .h and all, I don't want to scan .svn/ directory. How can I do?

I use : ./scancode -e ../gpl/
I get this error:

Extracting archives...
[------------------------------------] 0
Traceback (most recent call last):
File "/home/rex/qdu_claro/yuki/tools/scancode-toolkit-1.2.4/bin/scancode", line 9, in
load_entry_point('scancode-toolkit==1.2.4', 'console_scripts', 'scancode')()
File "/home/rex/qdu_claro/yuki/tools/scancode-toolkit-1.2.4/local/lib/python2.7/site-packages/click/core.py", line 664, in call
return self.main(_args, *_kwargs)
File "/home/rex/qdu_claro/yuki/tools/scancode-toolkit-1.2.4/local/lib/python2.7/site-packages/click/core.py", line 644, in main
rv = self.invoke(ctx)
File "/home/rex/qdu_claro/yuki/tools/scancode-toolkit-1.2.4/local/lib/python2.7/site-packages/click/core.py", line 837, in invoke
return ctx.invoke(self.callback, *_ctx.params)
File "/home/rex/qdu_claro/yuki/tools/scancode-toolkit-1.2.4/local/lib/python2.7/site-packages/click/core.py", line 464, in invoke
return callback(_args, **kwargs)
File "/home/rex/qdu_claro/yuki/tools/scancode-toolkit-1.2.4/src/scancode/cli.py", line 258, in scancode
extract_with_progress(abs_input, verbose)
File "/home/rex/qdu_claro/yuki/tools/scancode-toolkit-1.2.4/src/scancode/cli.py", line 345, in extract_with_progress
for xevent in extractions:
File "/home/rex/qdu_claro/yuki/tools/scancode-toolkit-1.2.4/local/lib/python2.7/site-packages/click/_termui_impl.py", line 240, in next
rv = next(self.iter)
File "/home/rex/qdu_claro/yuki/tools/scancode-toolkit-1.2.4/src/scancode/api.py", line 44, in extract_archives
from extractcode.extract import extract
File "/home/rex/qdu_claro/yuki/tools/scancode-toolkit-1.2.4/src/extractcode/extract.py", line 37, in
from extractcode import archive
File "/home/rex/qdu_claro/yuki/tools/scancode-toolkit-1.2.4/src/extractcode/archive.py", line 47, in
from extractcode import libarchive2
File "/home/rex/qdu_claro/yuki/tools/scancode-toolkit-1.2.4/src/extractcode/libarchive2.py", line 91, in
libarchive = load_lib()
File "/home/rex/qdu_claro/yuki/tools/scancode-toolkit-1.2.4/src/extractcode/libarchive2.py", line 87, in load_lib
raise ImportError('Failed to load libarchive: %(libarchive)r' % locals())
ImportError: Failed to load libarchive: '/home/rex/qdu_claro/yuki/scancode-toolkit-1.2.4/src/extractcode/bin/linux-32/bin/libarchive.so'

There is no libarchive.so in that directory!!!
Can anyone help me? Thx...

[pombredanne]

@K-Rex first, the pre-built binaries have not been built for 32-bites Linux... Do you have access to a 64 bits Linux installation instead?

I created a new ticket to ignore .svn dirs and similar ass #34

[pombredanne]

@K-Rex which exact Linux distro do you run?

[K-Rex]

My Linux environment is : Ubuntu 12.04 i386 GNU/Linux.

Can you tell me what -e command do?

I try to use it on WIN7 64bit. It generate housekeeper.a-extract directory (included file and platform_conf.o)

But I can't understand what it do. Thx

[pombredanne]

@K-Rex Thanks for the Linux version info.

The -e or --extract option looks in the input directory and extracts any archive or compressed files found there, such as a zip file, a tarball (or in your example a static library ar archive).

In your example above, housekeeper.a is an archive, so scancode -e extracted
the housekeeper.a archive to the housekeeper.a-extract directory where you can find the files that are contained in the original housekeeper.a archive, such as platform_conf.o and posssibly other files.

If you do not have archives in the directory you want to scan or you would prefer to extract these by hand before a scan, you do not need the -e/--extract option.

[pombredanne]

@K-Rex see also scancode --examples for --extract usage examples such as:

Extract all archives found in the 'samples' directory tree:

    scancode --extract samples

Note: If an archive contains other archives, all contained archives will be
extracted recursively. Extraction is done directly in the 'samples' directory,
side-by-side with each archive. Files are extracted in a directory named after
the archive with an '-extract' suffix added to its name, created side-by-side
with the corresponding archive file.

Extract a single archive. Files are extracted in the directory
'samples/arch/zlib.tar.gz-extract/':

    scancode --extract samples/arch/zlib.tar.gz

[pombredanne]

@K-Rex the new https://github.com/nexB/scancode-toolkit/releases/tag/v1.3.0 has experimental binaries for Linux 32 bits and improved help for the --extract option. I will make this an official release when I have you testing feedback

[K-Rex]

I download "scancode-toolkit-1.3.0.tar.bz2" and install it, but it still can't work for Linux 32 bits.

error:
ImportError: Failed to load libarchive: '/home/rex/qdu_claro/yuki/tools/scancode-toolkit-1.3.0/src/extractcode/bin/linux-32/bin/libarchive.so'

[pombredanne]

@K-Rex This was another silly un-tested mistake... (I unfortunately have no 32 bits test machine for now).
Please try again with the latest from develop here:
https://github.com/nexB/scancode-toolkit/archive/develop.tar.gz

[K-Rex]

Uh.. It still can't work on Linux-32

./scancode -e ../../qdu_claro/yuki/tools/gpl/

Traceback (most recent call last):
File "/home/rex/test/scancode-toolkit-develop/bin/scancode", line 9, in
load_entry_point('scancode-toolkit==1.3.0', 'console_scripts', 'scancode')()
File "/home/rex/test/scancode-toolkit-develop/local/lib/python2.7/site-packages/click/core.py", line 664, in call
return self.main(_args, *_kwargs)
File "/home/rex/test/scancode-toolkit-develop/src/scancode/cli.py", line 230, in main
standalone_mode=standalone_mode, *_extra)
File "/home/rex/test/scancode-toolkit-develop/local/lib/python2.7/site-packages/click/core.py", line 644, in main
rv = self.invoke(ctx)
File "/home/rex/test/scancode-toolkit-develop/local/lib/python2.7/site-packages/click/core.py", line 837, in invoke
return ctx.invoke(self.callback, *_ctx.params)
File "/home/rex/test/scancode-toolkit-develop/local/lib/python2.7/site-packages/click/core.py", line 464, in invoke
return callback(_args, *_kwargs)
File "/home/rex/test/scancode-toolkit-develop/src/scancode/cli.py", line 270, in scancode
extract_with_progress(abs_input, verbose)
File "/home/rex/test/scancode-toolkit-develop/src/scancode/cli.py", line 360, in extract_with_progress
for xevent in extractions:
File "/home/rex/test/scancode-toolkit-develop/local/lib/python2.7/site-packages/click/_termui_impl.py", line 240, in next
rv = next(self.iter)
File "/home/rex/test/scancode-toolkit-develop/src/scancode/api.py", line 44, in extract_archives
from extractcode.extract import extract
File "/home/rex/test/scancode-toolkit-develop/src/extractcode/extract.py", line 37, in
from extractcode import archive
File "/home/rex/test/scancode-toolkit-develop/src/extractcode/archive.py", line 47, in
from extractcode import libarchive2
File "/home/rex/test/scancode-toolkit-develop/src/extractcode/libarchive2.py", line 91, in
libarchive = load_lib()
File "/home/rex/test/scancode-toolkit-develop/src/extractcode/libarchive2.py", line 84, in load_lib
lib = ctypes.CDLL(libarchive)
File "/usr/lib/python2.7/ctypes/init.py", line 365, in init
self._handle = _dlopen(self._name, mode)

OSError: liblzma.so.1: cannot open shared object file: No such file or directory

[pombredanne]

@K-Rex You need to install a Linux pre-requisite package that may not be present on Ubuntu 12.04.

Use this:

sudo apt-get update
sudo apt-get install xz-utils

[pombredanne]

FYI, xz-utils provides liblzma

[K-Rex]

My OS had already install xz-utils

$sudo apt-get install xz-utils
Reading package lists... Done
Building dependency tree
Reading state information... Done
xz-utils is already the newest version.
You might want to run 'apt-get -f install' to correct these:
The following packages have unmet dependencies:
bison : Depends: m4 but it is not installable
Depends: libbison-dev (= 1:2.5.dfsg-2.1) but it is not installable
gettext : Depends: libunistring0 but it is not installable
Depends: libgettextpo0 (= 0.18.1.1-5ubuntu3) but it is not installable
vim-tiny : Depends: vim-common (= 2:7.3.429-2ubuntu2.1) but 2:7.3.429-2ubuntu2 is to be installed
E: Unmet dependencies. Try 'apt-get -f install' with no packages (or specify a solution).

It still has no liblzma.so.1 QQ

[pombredanne]

@K-Rex: ok, I will install a VM to test run extactly on your OS

[K-Rex]

I try to use VM to install Ubuntu 12.04 and test develop.tar.gz and latest version 1.3.1

Error Message:
/scancode-toolkit-develop$ ./scancode --help

• Configuring ScanCode ...
• Configuring Python ...
  Traceback (most recent call last):
  File "", line 1, in
  UnicodeDecodeError: 'ascii' codec can't decode byte 0xe4 in position 10: ordinal not in range(128)
  Traceback (most recent call last):
  File "thirdparty/base/virtualenv.py", line 2363, in
  main()
  File "thirdparty/base/virtualenv.py", line 832, in main
  symlink=options.symlink)
  File "thirdparty/base/virtualenv.py", line 1004, in create_environment
  install_wheel(to_install, py_executable, search_dirs)
  File "thirdparty/base/virtualenv.py", line 969, in install_wheel
  'PIP_NO_INDEX': '1'
  File "thirdparty/base/virtualenv.py", line 910, in call_subprocess
  % (cmd_desc, proc.returncode))
  OSError

Failed to execute command:
/usr/bin/python2.7 thirdparty/base/virtualenv.py --never-download --quiet --extra-search-dir=thirdparty/dev --extra-search-dir=thirdparty/base --extra-search-dir=thirdparty/prod /home/aaa/Download/scancode-toolkit-1.3.1/scancode-toolkit-develop
./scancode: line 17: /home/aaa/Download/scancode-toolkit-1.3.1/scancode-toolkit-develop/bin/scancode: No file and no directory.

Is that because PIP?

My environment is :
Linux virtual-machine 3.13.0-32-generic #57~precise1-Ubuntu SMP Tue Jul 15 03:50:54 UTC 2014 i686 i686 i386 GNU/Linux

[pombredanne]

@K-Rex since you are trying a VM, can you try to install a 64 bits VM instead?
again the 32 bits support is really something experimental I tried for you

[pombredanne]

If possible I would like to focus on 64 bits instead

[pombredanne]

The tests are all passing fine on Ubuntu 12 64 bits FWIW: See https://travis-ci.org/nexB/scancode-toolkit/branches

[K-Rex]

Well, maybe I don't need -e command. It's ok.
Thx.

[pombredanne]

@K-Rex I made a last attempt to rebuild libarchive on 32 bits, on Ubuntu 12.
Please try again using the latest from develop: https://github.com/nexB/scancode-toolkit/archive/develop.tar.gz

If this does not work, I will instead likely provide a better error handling and disable entirely 32 bits support

[pombredanne]

But if you are using Linux in a VM, please use a 64 bits VM, rather than 32 bits

[K-Rex]

Hey, it can work on Linux-32 bit and VM-32bits, the error is because the $LANG.
I change it to en_US.UTF-8, then it can install successfully.
And --verbose and -e command can work too.
Thank you very much, maybe you can release that version for Linux-32 bits!

[pombredanne]

@K-Rex Thank you for the test!. I am not entirely surprise by that... because the last error you had was really something in a core python tool called pip.