Problem with pip cache permissions

[RasterBurn]

I'm getting some issues with the docker based build and file permissions inside of the container. Any ideas?

Here are the configuration options:

custom:
  pythonRequirements:
    zip: true
    dockerizePip: true

And here is the error message I'm getting:

Serverless: Removing Python requirements helper...
Serverless: Adding Python requirements helper...
Serverless: Parsing Python requirements.txt
Serverless: Installing required Python packages for runtime python2.7...
Serverless: Docker Image: lambci/lambda:build-python2.7

  Error --------------------------------------------------

  The directory '/root/.cache/http' or its parent directory is not owned by the current user and the cache has been disabled. Please check the permissions and owner of that directory. If executing pip with sudo, you may want sudo's -H flag.
The directory '/root/.cache' or its parent directory is not owned by the current user and caching wheels has been disabled. check the permissions and owner of that directory. If executing pip with sudo, you may want sudo's -H flag.
Exception:
Traceback (most recent call last):
  File "/usr/local/lib/python2.7/site-packages/pip/basecommand.py", line 215, in main
    status = self.run(options, args)
  File "/usr/local/lib/python2.7/site-packages/pip/commands/install.py", line 324, in run
    requirement_set.prepare_files(finder)
  File "/usr/local/lib/python2.7/site-packages/pip/req/req_set.py", line 380, in prepare_files
    ignore_dependencies=self.ignore_dependencies))
  File "/usr/local/lib/python2.7/site-packages/pip/req/req_set.py", line 554, in _prepare_file
    require_hashes
  File "/usr/local/lib/python2.7/site-packages/pip/req/req_install.py", line 281, in populate_link
    self.link = self._wheel_cache.cached_wheel(self.link, self.name)
  File "/usr/local/lib/python2.7/site-packages/pip/wheel.py", line 68, in cached_wheel
    self._cache_dir, link, self._format_control, package_name)
  File "/usr/local/lib/python2.7/site-packages/pip/wheel.py", line 129, in cached_wheel
    wheel_names = os.listdir(root)
OSError: [Errno 13] Permission denied: '/root/.cache/wheels/44/56/6f/d5ac6dce9c887d96df2cd7b27ed0a6dd9357ef82b951ebb777'


     For debugging logs, run again after setting the "SLS_DEBUG=*" environment variable.

  Get Support --------------------------------------------
     Docs:          docs.serverless.com
     Bugs:          github.com/serverless/serverless/issues
     Forums:        forum.serverless.com
     Chat:          gitter.im/serverless/serverless

  Your Environment Information -----------------------------
     OS:                     darwin
     Node Version:           8.4.0
     Serverless Version:     1.20.2

[RasterBurn]

If I take the -u uid:gid option off of the docker call it works. It seems my uid (501) is not in the docker image, so it barfed when trying to write to /tmp.

Also I had to upgrade go because I was getting a "split stack overflow", but that may just be a docker issue.

Is it permissible to remove the -u option from the docker command?

[dschep]

Is it permissible to remove the -u option from the docker command?

Yeah, that's reasonable. Maybe a noDockerUser option? or dockerUID which if set to a string uses it as the -u option instead of the default and if set to null omits the option.

[AndrewFarley]

+1 been having this problem months ago on OS-X. I started manually packaging that project, but a new project I wanted to use this and found this bug report just now. Removing the -u option fixed this for me. I wouldn't mind creating a MR to help fix this... but I'm not sure the right way to fix this. I can do a noDockerUser as suggested above, or perhaps detect the host OS and skip the -u option altogether if on OS-X.

Thoughts/ideas/feedback from other contributors?

[dschep]

I'm ok with detecting OS. If you check for win32 also you'd fix #21 too.

[dschep]

I've released this in version 3.0.0-beta. It's a major version because the no deploy change is a breaking change and it's a beta because @unitedincome is nearing a public launch so I don't wanna pull a bunch of changes into our release.

[AndrewFarley]

Thanks guys... just tried the patch locally also, works great on OS-X. Cheers!

[fernando-mc]

@AndrewFarley and @dschep I'm seeing this issue on Ubuntu. Is the fix here supposed to work on other OSes? Or is there an alternative workaround suggested?

[dschep]

@fernando-mc checkout the v4 beta release, I just merged #145 which may help with this. (npm install serverless-python-requirements@beta)

[fernando-mc]

@dschep this still seems to be failing for me (I turned on the debug logs if it helps):

Serverless: Load command config
Serverless: Load command config:credentials
Serverless: Load command create
Serverless: Load command install
Serverless: Load command package
Serverless: Load command deploy
Serverless: Load command deploy:function
Serverless: Load command deploy:list
Serverless: Load command deploy:list:functions
Serverless: Load command invoke
Serverless: Load command invoke:local
Serverless: Load command info
Serverless: Load command logs
Serverless: Load command login
Serverless: Load command logout
Serverless: Load command metrics
Serverless: Load command print
Serverless: Load command remove
Serverless: Load command rollback
Serverless: Load command rollback:function
Serverless: Load command slstats
Serverless: Load command plugin
Serverless: Load command plugin
Serverless: Load command plugin:install
Serverless: Load command plugin
Serverless: Load command plugin:uninstall
Serverless: Load command plugin
Serverless: Load command plugin:list
Serverless: Load command plugin
Serverless: Load command plugin:search
Serverless: Load command emit
Serverless: Load command config
Serverless: Load command config:credentials
Serverless: Load command rollback
Serverless: Load command rollback:function
Serverless: Load command requirements
Serverless: Load command requirements:clean
Serverless: Load command requirements:install
Serverless: Load command wsgi
Serverless: Load command wsgi:serve
Serverless: Load command wsgi:clean
Serverless: Load command create_domain
Serverless: Load command delete_domain
Serverless: Invoke deploy
Serverless: Invoke package
Serverless: Invoke aws:common:validate
Serverless: Invoke aws:common:cleanupTempDir
Serverless: Installing requirements of requirements.txt in .serverless...
Serverless: Docker Image: lambci/lambda:build-python3.6
 
  Error --------------------------------------------------
 
  The directory '/.cache/pip/http' or its parent directory is not owned by the current user and the cache has been disabled. Please check the permissions and owner of that directory. If executing pip with sudo, you may want sudo's -H flag.
The directory '/.cache/pip' or its parent directory is not owned by the current user and caching wheels has been disabled. check the permissions and owner of that directory. If executing pip with sudo, you may want sudo's -H flag.
  Could not find a version that satisfies the requirement pkg-resources (from -r .serverless/requirements.txt (line 14)) (from versions: )
No matching distribution found for pkg-resources (from -r .serverless/requirements.txt (line 14))
You are using pip version 9.0.1, however version 9.0.2 is available.
You should consider upgrading via the 'pip install --upgrade pip' command.

 
     For debugging logs, run again after setting the "SLS_DEBUG=*" environment variable.
 
  Stack Trace --------------------------------------------
 
Error: The directory '/.cache/pip/http' or its parent directory is not owned by the current user and the cache has been disabled. Please check the permissions and owner of that directory. If executing pip with sudo, you may want sudo's -H flag.
The directory '/.cache/pip' or its parent directory is not owned by the current user and caching wheels has been disabled. check the permissions and owner of that directory. If executing pip with sudo, you may want sudo's -H flag.
  Could not find a version that satisfies the requirement pkg-resources (from -r .serverless/requirements.txt (line 14)) (from versions: )
No matching distribution found for pkg-resources (from -r .serverless/requirements.txt (line 14))
You are using pip version 9.0.1, however version 9.0.2 is available.
You should consider upgrading via the 'pip install --upgrade pip' command.

Here's the stack trace:

    at installRequirements (/home/fernando/Documents/code/myappdir/node_modules/serverless-python-requirements/lib/pip.js:114:11)
    at ServerlessPythonRequirements.installAllRequirements (/home/fernando/Documents/code/myappdir/node_modules/serverless-python-requirements/lib/pip.js:202:5)
From previous event:
    at PluginManager.invoke (/home/fernando/.nvm/versions/node/v6.10.3/lib/node_modules/serverless/lib/classes/PluginManager.js:372:22)
    at PluginManager.spawn (/home/fernando/.nvm/versions/node/v6.10.3/lib/node_modules/serverless/lib/classes/PluginManager.js:390:17)
    at Deploy.BbPromise.bind.then.then (/home/fernando/.nvm/versions/node/v6.10.3/lib/node_modules/serverless/lib/plugins/deploy/deploy.js:123:50)
From previous event:
    at Object.before:deploy:deploy [as hook] (/home/fernando/.nvm/versions/node/v6.10.3/lib/node_modules/serverless/lib/plugins/deploy/deploy.js:113:10)
    at BbPromise.reduce (/home/fernando/.nvm/versions/node/v6.10.3/lib/node_modules/serverless/lib/classes/PluginManager.js:372:55)
From previous event:
    at PluginManager.invoke (/home/fernando/.nvm/versions/node/v6.10.3/lib/node_modules/serverless/lib/classes/PluginManager.js:372:22)
    at PluginManager.run (/home/fernando/.nvm/versions/node/v6.10.3/lib/node_modules/serverless/lib/classes/PluginManager.js:403:17)
    at variables.populateService.then (/home/fernando/.nvm/versions/node/v6.10.3/lib/node_modules/serverless/lib/Serverless.js:102:33)
    at runCallback (timers.js:672:20)
    at tryOnImmediate (timers.js:645:5)
    at processImmediate [as _immediateCallback] (timers.js:617:5)
From previous event:
    at Serverless.run (/home/fernando/.nvm/versions/node/v6.10.3/lib/node_modules/serverless/lib/Serverless.js:89:74)
    at serverless.init.then (/home/fernando/.nvm/versions/node/v6.10.3/lib/node_modules/serverless/bin/serverless:42:50)

And my environment info:

 Your Environment Information -----------------------------
     OS:                     linux
     Node Version:           6.10.3
     Serverless Version:     1.26.1

Any suggestions?

[fernando-mc]

@dschep I think this might actually be related to either #158 or a bug with Ubuntu. https://bugs.launchpad.net/ubuntu/+source/python-pip/+bug/1635463

When I removed the pkg-resources line in my requirements.txt this worked fine.

[dschep]

Thanks for the report @fernando-mc. Was the pkg-resources line version 0.0.0 like that bug in the ubuntu launchpad?