Can't build from source; configure --dev fails

[abhi-kr-2100]

Description

According to INSTALL.rst:

Advanced installation: from source code using a git clone

• Download the Source Code or Use Git Clone::

  git clone https://github.com/nexB/scancode-toolkit.git
  cd scancode-toolkit

• Run the configure script for development usage:

  • On Linux/Mac: ./configure --dev
  • On Windows: configure --dev

If this displays the help text <https://scancode-toolkit.readthedocs.io/en/latest/cli-reference/help-text-options.html#help-text>_,
you are all set to start using ScanCode.

However, configure --dev fails with the following error:

./configure: line 251: /home/abhishek/Projects/scancode-toolkit/venv/bin/pip: No such file or directory

How To Reproduce

Follow the instructions in INSTALL.rst for building from source using git clone.

System configuration

• OS: WSL2 on Windows 11
• Latest commit: a410f6e

[abhi-kr-2100]

After running ./configure --dev a venv directory is indeed created, but it doesn't have a bin subdirectory. Instead, I see a local/bin subdirectory.

Changing CFG_BIN_DIR in the configure script fixes this for me.

[abhi-kr-2100]

Apparently, this bug affects only a few people. I have pip et. al. installed locally (~/.local/bin). That could be a reason. However, this still needs to be fixed.

[abhi-kr-2100]

To reproduce this bug if you don't have a local installation of pip, do the following:

• Obtain a clean installation of Ubuntu (as this is what was used by me) using a method suited to you:
  • WSL2 (used by me)
  • Virtual Machine
  • A physical machine (although this would be an overkill)
• Install python3-pip using sudo apt-get install python3-pip.
• Use pip3 to install pip locally: pip3 install --user --upgrade pip.
• Add ~/.local/bin to your PATH variable.
• Uninstall python3-pip package: sudo apt-get remove python3-pip && sudo apt-get autoremove.
• Install setuptools and wheel packages: pip install --user setuptools wheel
• Follow the instructions in INSTALL.rst.

[pombredanne]

@abhi-kr-2100 Thanks for this report and PR. On the surface you setup seems to be fairly customized.... Can you run this quick experiment for me?

1. open a shell, run python and paste back here the prompt that's displayed with version, arch like this:

$ python
Python 3.9.10 (main, Jan 29 2022, 10:01:49) 
[GCC 5.4.0 20160609] on linux
Type "help", "copyright", "credits" or "license" for more information.

2. run curl --location --output virtualenv.pyz https://github.com/nexB/scancode-toolkit/raw/develop/etc/thirdparty/virtualenv.pyz

3. run these:

$ mkdir venv
$ python virtualenv.pyz venv/
$ source venv/bin/activate
$ which python
$ which pip

and report the output

4. in your checkout, try running ./configure --clean and then ./configure --dev again... and paste the whole output back here.

[abhi-kr-2100]

@pombredanne Sure.

$ python3
Python 3.10.6 (main, Nov  2 2022, 18:53:38) [GCC 11.3.0] on linux
Type "help", "copyright", "credits" or "license" for more information.
>>>

2. Running these and subsequent commands inside the scancode-toolkit main directory with my modifications (is this what you meant by my checkout? If you meant the unmodified version, skip this and scroll below):

NOTE: After downloading virtualenv.pyz, I took a look inside and saw some weird characters; leaving them as is.

$ mkdir venv
$ python3 virtualenv.pyz venv/
created virtual environment CPython3.10.6.final.0-64 in 282ms
  creator CPython3Posix(dest=/home/abhishek/Projects/scancode-toolkit/venv, clear=False, no_vcs_ignore=False, global=False)
  seeder FromAppData(download=False, pip=bundle, setuptools=bundle, wheel=bundle, via=copy, app_data_dir=/home/abhishek/.local/share/virtualenv)
    added seed packages: pip==22.2.2, setuptools==63.4.1, wheel==0.37.1
  activators BashActivator,CShellActivator,FishActivator,NushellActivator,PowerShellActivator,PythonActivator
$ source venv/bin/activate
-bash: venv/bin/activate: No such file or directory
$ source venv/local/bin/activate
(venv) $ which python
/home/abhishek/Projects/scancode-toolkit/venv/local/bin/python
(venv) $ which pip
/home/abhishek/Projects/scancode-toolkit/venv/local/bin/pip

4. I ran those final commands, and scancode-toolkit was installed successfully. But as I mentioned, I ran them on my modified version.

---

1. Running the commands inside the unmodified repo on the devel branch:

$ python3
Python 3.10.6 (main, Nov  2 2022, 18:53:38) [GCC 11.3.0] on linux
Type "help", "copyright", "credits" or "license" for more information.
>>>

2. Downloaded.

$ mkdir venv
$ python3 virtualenv.pyz venv/
created virtual environment CPython3.10.6.final.0-64 in 224ms
  creator CPython3Posix(dest=/home/abhishek/Projects/scancode-toolkit/venv, clear=False, no_vcs_ignore=False, global=False)
  seeder FromAppData(download=False, pip=bundle, setuptools=bundle, wheel=bundle, via=copy, app_data_dir=/home/abhishek/.local/share/virtualenv)
    added seed packages: pip==22.2.2, setuptools==63.4.1, wheel==0.37.1
  activators BashActivator,CShellActivator,FishActivator,NushellActivator,PowerShellActivator,PythonActivator

4. Unsuccessful:

$ source venv/bin/activate
-bash: venv/bin/activate: No such file or directory
$ source venv/local/bin/activate
(venv) $ which python
/home/abhishek/Projects/scancode-toolkit/venv/local/bin/python
(venv) $ which pip
/home/abhishek/Projects/scancode-toolkit/venv/local/bin/pip
(venv) $ ./configure --clean
* Cleaning ...
(venv) $ ./configure --dev
created virtual environment CPython3.10.6.final.0-64 in 204ms
  creator CPython3Posix(dest=/home/abhishek/Projects/scancode-toolkit/venv, clear=False, no_vcs_ignore=True, global=False)
  seeder FromAppData(download=False, pip=embed, setuptools=embed, wheel=embed, via=copy, app_data_dir=/home/abhishek/.local/share/virtualenv)
    added seed packages: pip==22.2.2, setuptools==63.4.1, wheel==0.37.1
  activators BashActivator,CShellActivator,FishActivator,NushellActivator,PowerShellActivator,PythonActivator
./configure: line 251: /home/abhishek/Projects/scancode-toolkit/venv/bin/pip: No such file or directory

[pombredanne]

NOTE: After downloading virtualenv.pyz, I took a look inside and saw some weird characters; leaving them as is.

This is a zip, e.g. a binary.

$ source venv/bin/activate
-bash: venv/bin/activate: No such file or directory

This is really weird. I have never seen this before. I cannot understand why your setup would end up with venv/local/bin/pip
Is this a stock configuration or did you customize this?

[abhi-kr-2100]

I'm running a Ubuntu on WSL2, with Python 3 install from the package manager and pip installed locally inside ~/.local/bin. I tried to replicate this error in another WSL2 distro, but everything seemed to work fine. As such, right now, I'm not sure what's causing this either. The problem seems very specific to only one particular WSL2 distro I have.

[pombredanne]

@abhi-kr-2100 since this is only happening on one machine, I would like to defer making any update there. I am going to close this for now and your PR. Thank you ++ for this and please reopen if you see this occurring again.

[p-ouellette]

I ran into this. I think it's related to pypa/virtualenv#2350.

[AyanSinhaMahapatra]

@p-ouellette could you check if this is fixed in https://github.com/nexB/scancode-toolkit/releases/tag/v32.0.2? Updated to latest virtualenv there: 12fd01b

[p-ouellette]

Yes, it's fixed in v32.0.2. Thanks.

[AyanSinhaMahapatra]

Thanks for the report @p-ouellette !