incorrect installation with pip install when /tmp/ mounted with noexec option

[mszumilak]

SUMMARY

AWSCLI pip installation does not work when /tmp mounted with noexec.

OS / ENVIRONMENT

Any Linux, particularly:

Ubuntu 18.04.2 LTS

STEPS TO REPRODUCE

1. remount /tmp with noexec option:
   mount -o remount,noexec /tmp
2. create venv
3. install awscli
4. check permission on binaries

With /tmp with exec option awscli works fine:

szogun@lagman:~$ sudo mount -o remount,exec /tmp
szogun@lagman:~$ virtualenv venvs/awscli-tmp-with-exec
Using base prefix '/usr'
New python executable in /home/szogun/venvs/awscli-tmp-with-exec/bin/python3
Also creating executable in /home/szogun/venvs/awscli-tmp-with-exec/bin/python
Installing setuptools, pip, wheel...
done.
szogun@lagman:~$ . ~/venvs/awscli-tmp-with-exec/bin/activate
(awscli-tmp-with-exec) szogun@lagman:~$ pip install awscli
Collecting awscli
  Downloading https://files.pythonhosted.org/packages/f6/a5/1c99f8222fa9a57b3bda66a86c55b8e5b6d35162061c88d23aa0e3ae0fee/awscli-1.16.173-py2.py3-none-any.whl (1.6MB)
     |████████████████████████████████| 1.6MB 1.1MB/s 
Collecting PyYAML<=3.13,>=3.10 (from awscli)
Collecting rsa<=3.5.0,>=3.1.2 (from awscli)
  Using cached https://files.pythonhosted.org/packages/e1/ae/baedc9cb175552e95f3395c43055a6a5e125ae4d48a1d7a924baca83e92e/rsa-3.4.2-py2.py3-none-any.whl
Collecting colorama<=0.3.9,>=0.2.5 (from awscli)
  Using cached https://files.pythonhosted.org/packages/db/c8/7dcf9dbcb22429512708fe3a547f8b6101c0d02137acbd892505aee57adf/colorama-0.3.9-py2.py3-none-any.whl
Collecting s3transfer<0.3.0,>=0.2.0 (from awscli)
  Using cached https://files.pythonhosted.org/packages/16/8a/1fc3dba0c4923c2a76e1ff0d52b305c44606da63f718d14d3231e21c51b0/s3transfer-0.2.1-py2.py3-none-any.whl
Collecting docutils>=0.10 (from awscli)
  Using cached https://files.pythonhosted.org/packages/36/fa/08e9e6e0e3cbd1d362c3bbee8d01d0aedb2155c4ac112b19ef3cae8eed8d/docutils-0.14-py3-none-any.whl
Collecting botocore==1.12.163 (from awscli)
  Downloading https://files.pythonhosted.org/packages/6a/42/f601129ebb1c08249b3ea3e1db1074cd4f1b6e4e00eed7f6cebe5b2432d6/botocore-1.12.163-py2.py3-none-any.whl (5.5MB)
     |████████████████████████████████| 5.5MB 6.2MB/s 
Collecting pyasn1>=0.1.3 (from rsa<=3.5.0,>=3.1.2->awscli)
  Using cached https://files.pythonhosted.org/packages/7b/7c/c9386b82a25115cccf1903441bba3cbadcfae7b678a20167347fa8ded34c/pyasn1-0.4.5-py2.py3-none-any.whl
Collecting urllib3<1.26,>=1.20; python_version >= "3.4" (from botocore==1.12.163->awscli)
  Using cached https://files.pythonhosted.org/packages/e6/60/247f23a7121ae632d62811ba7f273d0e58972d75e58a94d329d51550a47d/urllib3-1.25.3-py2.py3-none-any.whl
Collecting python-dateutil<3.0.0,>=2.1; python_version >= "2.7" (from botocore==1.12.163->awscli)
  Using cached https://files.pythonhosted.org/packages/41/17/c62faccbfbd163c7f57f3844689e3a78bae1f403648a6afb1d0866d87fbb/python_dateutil-2.8.0-py2.py3-none-any.whl
Collecting jmespath<1.0.0,>=0.7.1 (from botocore==1.12.163->awscli)
  Using cached https://files.pythonhosted.org/packages/83/94/7179c3832a6d45b266ddb2aac329e101367fbdb11f425f13771d27f225bb/jmespath-0.9.4-py2.py3-none-any.whl
Collecting six>=1.5 (from python-dateutil<3.0.0,>=2.1; python_version >= "2.7"->botocore==1.12.163->awscli)
  Using cached https://files.pythonhosted.org/packages/73/fb/00a976f728d0d1fecfe898238ce23f502a721c0ac0ecfedb80e0d88c64e9/six-1.12.0-py2.py3-none-any.whl
Installing collected packages: PyYAML, pyasn1, rsa, colorama, docutils, urllib3, six, python-dateutil, jmespath, botocore, s3transfer, awscli
Successfully installed PyYAML-3.13 awscli-1.16.173 botocore-1.12.163 colorama-0.3.9 docutils-0.14 jmespath-0.9.4 pyasn1-0.4.5 python-dateutil-2.8.0 rsa-3.4.2 s3transfer-0.2.1 six-1.12.0 urllib3-1.25.3
(awscli-tmp-with-exec) szogun@lagman:~$

With noexec it does not:

szogun@lagman:~$ sudo mount -o remount,noexec /tmp
szogun@lagman:~$ virtualenv venvs/awscli-tmp-with-noexec
Using base prefix '/usr'
New python executable in /home/szogun/venvs/awscli-tmp-with-noexec/bin/python3
Also creating executable in /home/szogun/venvs/awscli-tmp-with-npexec/bin/python
Installing setuptools, pip, wheel...
done.
szogun@lagman:~$ . ~/venvs/awscli-tmp-with-noexec/bin/activate
(awscli-tmp-with-noexec) szogun@lagman:~$ pip install awscli
Collecting awscli
  Downloading https://files.pythonhosted.org/packages/f6/a5/1c99f8222fa9a57b3bda66a86c55b8e5b6d35162061c88d23aa0e3ae0fee/awscli-1.16.173-py2.py3-none-any.whl (1.6MB)
     |████████████████████████████████| 1.6MB 1.1MB/s 
Collecting PyYAML<=3.13,>=3.10 (from awscli)
Collecting rsa<=3.5.0,>=3.1.2 (from awscli)
  Using cached https://files.pythonhosted.org/packages/e1/ae/baedc9cb175552e95f3395c43055a6a5e125ae4d48a1d7a924baca83e92e/rsa-3.4.2-py2.py3-none-any.whl
Collecting colorama<=0.3.9,>=0.2.5 (from awscli)
  Using cached https://files.pythonhosted.org/packages/db/c8/7dcf9dbcb22429512708fe3a547f8b6101c0d02137acbd892505aee57adf/colorama-0.3.9-py2.py3-none-any.whl
Collecting s3transfer<0.3.0,>=0.2.0 (from awscli)
  Using cached https://files.pythonhosted.org/packages/16/8a/1fc3dba0c4923c2a76e1ff0d52b305c44606da63f718d14d3231e21c51b0/s3transfer-0.2.1-py2.py3-none-any.whl
Collecting docutils>=0.10 (from awscli)
  Using cached https://files.pythonhosted.org/packages/36/fa/08e9e6e0e3cbd1d362c3bbee8d01d0aedb2155c4ac112b19ef3cae8eed8d/docutils-0.14-py3-none-any.whl
Collecting botocore==1.12.163 (from awscli)
  Downloading https://files.pythonhosted.org/packages/6a/42/f601129ebb1c08249b3ea3e1db1074cd4f1b6e4e00eed7f6cebe5b2432d6/botocore-1.12.163-py2.py3-none-any.whl (5.5MB)
     |████████████████████████████████| 5.5MB 6.2MB/s 
Collecting pyasn1>=0.1.3 (from rsa<=3.5.0,>=3.1.2->awscli)
  Using cached https://files.pythonhosted.org/packages/7b/7c/c9386b82a25115cccf1903441bba3cbadcfae7b678a20167347fa8ded34c/pyasn1-0.4.5-py2.py3-none-any.whl
Collecting urllib3<1.26,>=1.20; python_version >= "3.4" (from botocore==1.12.163->awscli)
  Using cached https://files.pythonhosted.org/packages/e6/60/247f23a7121ae632d62811ba7f273d0e58972d75e58a94d329d51550a47d/urllib3-1.25.3-py2.py3-none-any.whl
Collecting python-dateutil<3.0.0,>=2.1; python_version >= "2.7" (from botocore==1.12.163->awscli)
  Using cached https://files.pythonhosted.org/packages/41/17/c62faccbfbd163c7f57f3844689e3a78bae1f403648a6afb1d0866d87fbb/python_dateutil-2.8.0-py2.py3-none-any.whl
Collecting jmespath<1.0.0,>=0.7.1 (from botocore==1.12.163->awscli)
  Using cached https://files.pythonhosted.org/packages/83/94/7179c3832a6d45b266ddb2aac329e101367fbdb11f425f13771d27f225bb/jmespath-0.9.4-py2.py3-none-any.whl
Collecting six>=1.5 (from python-dateutil<3.0.0,>=2.1; python_version >= "2.7"->botocore==1.12.163->awscli)
  Using cached https://files.pythonhosted.org/packages/73/fb/00a976f728d0d1fecfe898238ce23f502a721c0ac0ecfedb80e0d88c64e9/six-1.12.0-py2.py3-none-any.whl
Installing collected packages: PyYAML, pyasn1, rsa, colorama, docutils, urllib3, six, python-dateutil, jmespath, botocore, s3transfer, awscli
Successfully installed PyYAML-3.13 awscli-1.16.173 botocore-1.12.163 colorama-0.3.9 docutils-0.14 jmespath-0.9.4 pyasn1-0.4.5 python-dateutil-2.8.0 rsa-3.4.2 s3transfer-0.2.1 six-1.12.0 urllib3-1.25.3
(awscli-tmp-with-noexec) szogun@lagman:~$ aws
bash: /home/szogun/venvs/awscli-tmp-with-noexec/bin/aws: Permission denied
(awscli-tmp-with-noexec) szogun@lagman:~$ ls -l /home/szogun/venvs/awscli-tmp-with-noexec/bin/aws
-rw-rw-r-- 1 szogun szogun 840 Jun  7 17:10 /home/szogun/venvs/awscli-tmp-with-noexec/bin/aws

EXPECTED RESULTS

Package should handle setting proper permission while being installed. There are some packages not affected by this issue.

[justnance]

@mszumilak - Thank you for posting this issue and the production steps. I was able to reproduce and confirm the behavior.

(awscli-tmp-with-noexec) ubuntu@ip-x-x-x-x:~$ aws
-bash: /home/ubuntu/venvs/awscli-tmp-with-noexec/bin/aws: Permission denied

This appears to be an issue with the pip because some the files in the /bin directory do not have the execution bit set. After doing some digging I found a related issue in the pypa/pip repo with more details. I suggest +1 the issue.

It looks like there are a couple of work arounds mentioned in #2830, on Stack Overflow, and in our online documentation.

Let us know if there are any questions.

[mszumilak]

Issue on PIP was opened in March and not fixed yet. I believe proper chmod settings can be set in setup.py for awscli. Chmod is handled in similar way for youtube-dl for example.

[justnance]

@mszumilak - Thanks for your feedback. I am marking this issue as an Enhancement as it may be reasonable to implement in CLI V2 which we are tracking under #3587.

[mszumilak]

Does os.chmod() in the for loop have to wait for v2?

[kyleknap]

@mszumilak I think we can address it in v1 possibly. Can you point to how youtube-dl works around this? I was poking around their setup.py file and did not really see where they do the workaround. In general, I think I'd prefer to see this get fixed in pip, but I want to see the options here.

In the meantime, I'd recommend installing via the bundled installer or just updating the permissions on the executable that pip installs.

[no-response]

This issue has been automatically closed because there has been no response to our request for more information from the original author. With only the information that is currently in the issue, we don't have enough information to take action. Please reach out if you have or find the answers we need so that we can investigate further.