pip-compile doesn't work when home directory isn't writeable

[adamchainz]

I'm using pip-compile on AWS Lambda to automate spotting when my requirements file needs updating. It always tries to write the cache and there's no way to disable it. It thus ends up dying

Steps to replicate

1. Run pip-compile on a system where the home directory isn't writeable.

Expected result

It works, but maybe reports that the home directory isn't writeable so caching is broken

Actual result

  File "/var/task/main.py", line 58, in get_requirements_diff
    pip_compile(args=['-n', '-r', '-U'], standalone_mode=False)
  File "/var/task/click/core.py", line 716, in __call__
    return self.main(*args, **kwargs)
  File "/var/task/click/core.py", line 696, in main
    rv = self.invoke(ctx)
  File "/var/task/click/core.py", line 889, in invoke
    return ctx.invoke(self.callback, **ctx.params)
  File "/var/task/click/core.py", line 534, in invoke
    return callback(*args, **kwargs)
  File "/var/task/piptools/scripts/compile.py", line 160, in cli
    clear_caches=rebuild)
  File "/var/task/piptools/resolver.py", line 66, in __init__
    cache = DependencyCache()  # pragma: no cover
  File "/var/task/piptools/cache.py", line 55, in __init__
    os.makedirs(cache_dir)
  File "/usr/lib64/python2.7/os.py", line 150, in makedirs
    makedirs(head, mode)
  File "/usr/lib64/python2.7/os.py", line 150, in makedirs
    makedirs(head, mode)
  File "/usr/lib64/python2.7/os.py", line 157, in makedirs
    mkdir(name, mode)
OSError: [Errno 30] Read-only file system: '/home/sbx_user1056'

I'm working around this by doing the following, as /tmp is writeable on AWS Lambda, but it's not a general solution

from piptools import cache
from piptools.repositories import pypi


cache.CACHE_DIR = '/tmp/piptools-cache'  # workaround
pypi.CACHE_DIR = '/tmp/piptools-cache'  # workaround

[adamchainz]

FWIW pip itself successfully disables the cache and emits a warning if it has problems writing: https://github.com/pypa/pip/blob/7954523ce3174ee1677f1fdc6bcca8efb2b5e95a/pip/download.py#L260

[jedie]

Maybe i rise into the same Problem with pip-tools==2.0.1:

Traceback (most recent call last):
  File "/usr/local/bin/pip-compile", line 11, in <module>
    sys.exit(cli())
  File "/usr/local/lib/python3.5/dist-packages/click/core.py", line 722, in __call__
    return self.main(*args, **kwargs)
  File "/usr/local/lib/python3.5/dist-packages/click/core.py", line 697, in main
    rv = self.invoke(ctx)
  File "/usr/local/lib/python3.5/dist-packages/click/core.py", line 895, in invoke
    return ctx.invoke(self.callback, **ctx.params)
  File "/usr/local/lib/python3.5/dist-packages/click/core.py", line 535, in invoke
    return callback(*args, **kwargs)
  File "/usr/local/lib/python3.5/dist-packages/piptools/scripts/compile.py", line 243, in cli
    allow_unsafe=allow_unsafe)
  File "/usr/local/lib/python3.5/dist-packages/piptools/writer.py", line 123, in write
    f = stack.enter_context(AtomicSaver(self.dst_file))
  File "/usr/lib/python3.5/contextlib.py", line 304, in enter_context
    result = _cm_type.__enter__(cm)
  File "/usr/local/lib/python3.5/dist-packages/piptools/io.py", line 432, in __enter__
    self.setup()
  File "/usr/local/lib/python3.5/dist-packages/piptools/io.py", line 428, in setup
    self._open_part_file()
  File "/usr/local/lib/python3.5/dist-packages/piptools/io.py", line 393, in _open_part_file
    fd = os.open(self.part_path, self.open_flags, file_perms)
OSError: [Errno 30] Read-only file system: 'requirements.txt.part'

I'm not sure where piptools will create the file requirements.txt.part. Is it the cwd ? This directory is write able for the current user.

[atugushev]

Cache dir is customizable since 4.4.0.

$ pip-compile --help | grep cache-dir -A 2
  --cache-dir DIRECTORY           Store the cache data in DIRECTORY.  [env
                                  var: PIP_TOOLS_CACHE_DIR; default:
                                  /root/.cache/pip-tools]