Skip to content

Problem with pip cache permissions #62

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom
Closed
RasterBurn opened this issue Aug 22, 2017 · 24 comments · Fixed by #64
Closed

Problem with pip cache permissions #62

RasterBurn opened this issue Aug 22, 2017 · 24 comments · Fixed by #64

Comments

@RasterBurn
Copy link

I'm getting some issues with the docker based build and file permissions inside of the container. Any ideas?

Here are the configuration options:

custom:
  pythonRequirements:
    zip: true
    dockerizePip: true

And here is the error message I'm getting:

Serverless: Removing Python requirements helper...
Serverless: Adding Python requirements helper...
Serverless: Parsing Python requirements.txt
Serverless: Installing required Python packages for runtime python2.7...
Serverless: Docker Image: lambci/lambda:build-python2.7

  Error --------------------------------------------------

  The directory '/root/.cache/http' or its parent directory is not owned by the current user and the cache has been disabled. Please check the permissions and owner of that directory. If executing pip with sudo, you may want sudo's -H flag.
The directory '/root/.cache' or its parent directory is not owned by the current user and caching wheels has been disabled. check the permissions and owner of that directory. If executing pip with sudo, you may want sudo's -H flag.
Exception:
Traceback (most recent call last):
  File "/usr/local/lib/python2.7/site-packages/pip/basecommand.py", line 215, in main
    status = self.run(options, args)
  File "/usr/local/lib/python2.7/site-packages/pip/commands/install.py", line 324, in run
    requirement_set.prepare_files(finder)
  File "/usr/local/lib/python2.7/site-packages/pip/req/req_set.py", line 380, in prepare_files
    ignore_dependencies=self.ignore_dependencies))
  File "/usr/local/lib/python2.7/site-packages/pip/req/req_set.py", line 554, in _prepare_file
    require_hashes
  File "/usr/local/lib/python2.7/site-packages/pip/req/req_install.py", line 281, in populate_link
    self.link = self._wheel_cache.cached_wheel(self.link, self.name)
  File "/usr/local/lib/python2.7/site-packages/pip/wheel.py", line 68, in cached_wheel
    self._cache_dir, link, self._format_control, package_name)
  File "/usr/local/lib/python2.7/site-packages/pip/wheel.py", line 129, in cached_wheel
    wheel_names = os.listdir(root)
OSError: [Errno 13] Permission denied: '/root/.cache/wheels/44/56/6f/d5ac6dce9c887d96df2cd7b27ed0a6dd9357ef82b951ebb777'


     For debugging logs, run again after setting the "SLS_DEBUG=*" environment variable.

  Get Support --------------------------------------------
     Docs:          docs.serverless.com
     Bugs:          github.com/serverless/serverless/issues
     Forums:        forum.serverless.com
     Chat:          gitter.im/serverless/serverless

  Your Environment Information -----------------------------
     OS:                     darwin
     Node Version:           8.4.0
     Serverless Version:     1.20.2
@RasterBurn
Copy link
Author

If I take the -u uid:gid option off of the docker call it works. It seems my uid (501) is not in the docker image, so it barfed when trying to write to /tmp.

Also I had to upgrade go because I was getting a "split stack overflow", but that may just be a docker issue.

Is it permissible to remove the -u option from the docker command?

@dschep
Copy link
Contributor

dschep commented Aug 22, 2017

Is it permissible to remove the -u option from the docker command?

Yeah, that's reasonable. Maybe a noDockerUser option? or dockerUID which if set to a string uses it as the -u option instead of the default and if set to null omits the option.

@AndrewFarley
Copy link
Contributor

AndrewFarley commented Aug 23, 2017 •
edited
Loading

+1 been having this problem months ago on OS-X. I started manually packaging that project, but a new project I wanted to use this and found this bug report just now. Removing the -u option fixed this for me. I wouldn't mind creating a MR to help fix this... but I'm not sure the right way to fix this. I can do a noDockerUser as suggested above, or perhaps detect the host OS and skip the -u option altogether if on OS-X.

Thoughts/ideas/feedback from other contributors?

@dschep
Copy link
Contributor

dschep commented Aug 23, 2017 •
edited
Loading

I'm ok with detecting OS. If you check for win32 also you'd fix #21 too.

dschep added a commit that referenced this issue Aug 23, 2017
@dschep
Only pass UID option to Docker on Linux
817860d 
closes #62 and  closes #21
@dschep dschep mentioned this issue Aug 23, 2017
Merged
dschep added a commit that referenced this issue Aug 23, 2017
@dschep
Only pass UID option to Docker on Linux
9a7cda0 
closes #62 and  closes #21
@dschep
Copy link
Contributor

dschep commented Aug 24, 2017

I've released this in version 3.0.0-beta. It's a major version because the no deploy change is a breaking change and it's a beta because @unitedincome is nearing a public launch so I don't wanna pull a bunch of changes into our release.

@AndrewFarley
Copy link
Contributor

Thanks guys... just tried the patch locally also, works great on OS-X. Cheers!

@fernando-mc
Copy link

@AndrewFarley and @dschep I'm seeing this issue on Ubuntu. Is the fix here supposed to work on other OSes? Or is there an alternative workaround suggested?

@dschep
Copy link
Contributor

dschep commented Mar 20, 2018

@fernando-mc checkout the v4 beta release, I just merged #145 which may help with this. (npm install serverless-python-requirements@beta)

@fernando-mc
Copy link

fernando-mc commented Mar 21, 2018 •
edited
Loading

@dschep this still seems to be failing for me (I turned on the debug logs if it helps):

Serverless: Load command config
Serverless: Load command config:credentials
Serverless: Load command create
Serverless: Load command install
Serverless: Load command package
Serverless: Load command deploy
Serverless: Load command deploy:function
Serverless: Load command deploy:list
Serverless: Load command deploy:list:functions
Serverless: Load command invoke
Serverless: Load command invoke:local
Serverless: Load command info
Serverless: Load command logs
Serverless: Load command login
Serverless: Load command logout
Serverless: Load command metrics
Serverless: Load command print
Serverless: Load command remove
Serverless: Load command rollback
Serverless: Load command rollback:function
Serverless: Load command slstats
Serverless: Load command plugin
Serverless: Load command plugin
Serverless: Load command plugin:install
Serverless: Load command plugin
Serverless: Load command plugin:uninstall
Serverless: Load command plugin
Serverless: Load command plugin:list
Serverless: Load command plugin
Serverless: Load command plugin:search
Serverless: Load command emit
Serverless: Load command config
Serverless: Load command config:credentials
Serverless: Load command rollback
Serverless: Load command rollback:function
Serverless: Load command requirements
Serverless: Load command requirements:clean
Serverless: Load command requirements:install
Serverless: Load command wsgi
Serverless: Load command wsgi:serve
Serverless: Load command wsgi:clean
Serverless: Load command create_domain
Serverless: Load command delete_domain
Serverless: Invoke deploy
Serverless: Invoke package
Serverless: Invoke aws:common:validate
Serverless: Invoke aws:common:cleanupTempDir
Serverless: Installing requirements of requirements.txt in .serverless...
Serverless: Docker Image: lambci/lambda:build-python3.6
 
  Error --------------------------------------------------
 
  The directory '/.cache/pip/http' or its parent directory is not owned by the current user and the cache has been disabled. Please check the permissions and owner of that directory. If executing pip with sudo, you may want sudo's -H flag.
The directory '/.cache/pip' or its parent directory is not owned by the current user and caching wheels has been disabled. check the permissions and owner of that directory. If executing pip with sudo, you may want sudo's -H flag.
  Could not find a version that satisfies the requirement pkg-resources (from -r .serverless/requirements.txt (line 14)) (from versions: )
No matching distribution found for pkg-resources (from -r .serverless/requirements.txt (line 14))
You are using pip version 9.0.1, however version 9.0.2 is available.
You should consider upgrading via the 'pip install --upgrade pip' command.

 
     For debugging logs, run again after setting the "SLS_DEBUG=*" environment variable.
 
  Stack Trace --------------------------------------------
 
Error: The directory '/.cache/pip/http' or its parent directory is not owned by the current user and the cache has been disabled. Please check the permissions and owner of that directory. If executing pip with sudo, you may want sudo's -H flag.
The directory '/.cache/pip' or its parent directory is not owned by the current user and caching wheels has been disabled. check the permissions and owner of that directory. If executing pip with sudo, you may want sudo's -H flag.
  Could not find a version that satisfies the requirement pkg-resources (from -r .serverless/requirements.txt (line 14)) (from versions: )
No matching distribution found for pkg-resources (from -r .serverless/requirements.txt (line 14))
You are using pip version 9.0.1, however version 9.0.2 is available.
You should consider upgrading via the 'pip install --upgrade pip' command.

Here's the stack trace:

    at installRequirements (/home/fernando/Documents/code/myappdir/node_modules/serverless-python-requirements/lib/pip.js:114:11)
    at ServerlessPythonRequirements.installAllRequirements (/home/fernando/Documents/code/myappdir/node_modules/serverless-python-requirements/lib/pip.js:202:5)
From previous event:
    at PluginManager.invoke (/home/fernando/.nvm/versions/node/v6.10.3/lib/node_modules/serverless/lib/classes/PluginManager.js:372:22)
    at PluginManager.spawn (/home/fernando/.nvm/versions/node/v6.10.3/lib/node_modules/serverless/lib/classes/PluginManager.js:390:17)
    at Deploy.BbPromise.bind.then.then (/home/fernando/.nvm/versions/node/v6.10.3/lib/node_modules/serverless/lib/plugins/deploy/deploy.js:123:50)
From previous event:
    at Object.before:deploy:deploy [as hook] (/home/fernando/.nvm/versions/node/v6.10.3/lib/node_modules/serverless/lib/plugins/deploy/deploy.js:113:10)
    at BbPromise.reduce (/home/fernando/.nvm/versions/node/v6.10.3/lib/node_modules/serverless/lib/classes/PluginManager.js:372:55)
From previous event:
    at PluginManager.invoke (/home/fernando/.nvm/versions/node/v6.10.3/lib/node_modules/serverless/lib/classes/PluginManager.js:372:22)
    at PluginManager.run (/home/fernando/.nvm/versions/node/v6.10.3/lib/node_modules/serverless/lib/classes/PluginManager.js:403:17)
    at variables.populateService.then (/home/fernando/.nvm/versions/node/v6.10.3/lib/node_modules/serverless/lib/Serverless.js:102:33)
    at runCallback (timers.js:672:20)
    at tryOnImmediate (timers.js:645:5)
    at processImmediate [as _immediateCallback] (timers.js:617:5)
From previous event:
    at Serverless.run (/home/fernando/.nvm/versions/node/v6.10.3/lib/node_modules/serverless/lib/Serverless.js:89:74)
    at serverless.init.then (/home/fernando/.nvm/versions/node/v6.10.3/lib/node_modules/serverless/bin/serverless:42:50)

And my environment info:

 Your Environment Information -----------------------------
     OS:                     linux
     Node Version:           6.10.3
     Serverless Version:     1.26.1

Any suggestions?

@fernando-mc
Copy link

@dschep I think this might actually be related to either #158 or a bug with Ubuntu. https://bugs.launchpad.net/ubuntu/+source/python-pip/+bug/1635463

When I removed the pkg-resources line in my requirements.txt this worked fine.

@dschep
Copy link
Contributor

dschep commented Mar 21, 2018

Thanks for the report @fernando-mc. Was the pkg-resources line version 0.0.0 like that bug in the ubuntu launchpad?

@fernando-mc
Copy link

@dschep yep! Just removing the line resolved this. Not sure it will break anything on other OSes doing this but it resolved the issue on Ubuntu 16.04.

@danbovey
Copy link

danbovey commented Apr 13, 2018 •
edited
Loading

I think I'm seeing the same issue on Ubuntu 16.04, running serverless-python-requirements@4.0.0.

The directory '/.cache/pip/http' or its parent directory is not owned by the current user and the cache has been disabled. Please check the permissions and owner of that directory. If executing pip with sudo, you may want sudo's -H flag.
The directory '/.cache/pip' or its parent directory is not owned by the current user and caching wheels has been disabled. check the permissions and owner of that directory. If executing pip with sudo, you may want sudo's -H flag.
Command "python setup.py egg_info" failed with error code 1 in /tmp/pip-build-HoBoR1/psutil/

I don't have pkg-resources in my requirements.txt, so it's a slightly different permissions error from what @fernando-mc had.

Is this the same issue of docker not running pip as the correct user, or an error with this wheels package?

@dschep
Copy link
Contributor

dschep commented Apr 17, 2018

Hmm. are you using a custom docker image?

@danbovey
Copy link

@dschep No I'm using the default lambci/lambda image.

@simlu
Copy link

simlu commented May 23, 2018

Same error/problem here. @dschep How are you starting the docker image?

@AndrewFarley
Copy link
Contributor

AndrewFarley commented May 23, 2018 •
edited
Loading

Hey @simlu and @danbovey and anyone else with this caching issue still, can you guys do me/us a favor and try my PR to improve caching? See if this solves the problem? In the PR we are making caching an actual feature, not something you "kinda hack in poorly" with a pip option. Additionally, the PR significantly changes (and limits) the scope of what docker does, which should eliminate all permissions issues. To do this...

rm -Rf node_modules/serverless-python-requirements 
npm i UnitedIncome/serverless-python-requirements#pull/165/head

Then in your serverless.yml enable the cache with...

custom: 
  pythonRequirements:
    dockerizePip: true
    useDownloadCache: true

Then try a few full deploys... see if that works. The first time might be slow (normal speed, filling the download cache), but the second should be faster.

If that works for you, please also then try the following to enable the "static" cache (which caches the outputs of pip, not just the downloads). To do this...

custom: 
  pythonRequirements:
    dockerizePip: true
    useDownloadCache: true
    useStaticCache: true

And if that works, you'll see your builds go super-fast after the first time as long as your requirements.txt doesn't change (which it rarely does). Let me know how that works, and if that solves your problem. Thanks!

@simlu
Copy link

simlu commented May 23, 2018

For now I've fixed this by using

    pipCmdExtraArgs:
      - --no-cache-dir

I'll try your branch next

@simlu
Copy link

simlu commented May 24, 2018

Getting an error when trying to install as you described:

npm ERR! code 1
npm ERR! Command failed: /usr/bin/git checkout pull/165/head
npm ERR! error: pathspec 'pull/165/head' did not match any file(s) known to git.
npm ERR!

@simlu
Copy link

simlu commented May 24, 2018

Tried with npm i github:andrewfarley/serverless-python-requirements#master as per #165

but still getting the same permission error. I should highlight that we use our own docker container and then run sls deploy inside it. So we have dockerizePip set to false.

@AndrewFarley
Copy link
Contributor

AndrewFarley commented May 24, 2018 •
edited
Loading

@simlu Ahh... alright, well, your issue is a bit different than most of the rest above whose problem revolves around this plugin using docker. You've got a bit of a custom thing going... that sounds like you just need to add a chown to one of your steps in your Dockerfile to fix that. Or if your Docker container mounts volumes containing the parent code and that's where you're running serverless from, then that would explain things.

@simlu
Copy link

simlu commented May 25, 2018

I tried that initially and couldn't get it to work. So I thought this might be a nice and easy workaround. Why does your approach not work for our use case then?

@AndrewFarley
Copy link
Contributor

Because my fork still tries to use and pip cache inside the users home directory (sort of) but I bet your dockerfile and the files in your docker file are mounted via a volume which gives them an invalid/null owner which freaks pip out. If you don’t mind sharing your dockerfile and the script and or command you use to deploy your serverless stacks I’d gladly look at it and either advise you what to change or submit a patch to this software. The problem you are facing is specific to pip, not this plugin. :)

@simlu
Copy link

simlu commented May 25, 2018

Well, not really. It's using the user coming from the OS so files are accessed/written with the same permissions.

For sure, it's from py-gardener which is published under MIT:

We run . manage.sh to start the docker container (Ubuntu 16.04 / 18.04)

Thank you very much for helping!

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees
No one assigned
Labels
None yet
Projects
None yet
Milestone
No milestone
Development

Successfully merging a pull request may close this issue.

Only pass UID option to Docker on Linux serverless/serverless-python-requirements
6 participants
@RasterBurn @AndrewFarley @dschep @danbovey @simlu @fernando-mc