OSError: : [Errno 2] No such file or directory

[objmagic]

Hi,

Pex is used to generate Python executable files in twitter/heron. However, we sometimes can see the following error message that makes build fail:

    sys.exit(main())
  File "/home/travis/.cache/bazel/_bazel_travis/030a6cc9b27b34f2fe0e137b942f59bf/heron/bazel-out/local_linux-fastbuild/bin/third_party/pex/_pex.runfiles/__main__/third_party/pex/_pex.py", line 159, in main
    resolver_options_builder, interpreter=interpreter)
  File "/home/travis/.cache/bazel/_bazel_travis/030a6cc9b27b34f2fe0e137b942f59bf/heron/bazel-out/local_linux-fastbuild/bin/third_party/pex/_pex.runfiles/__main__/third_party/pex/pex/bin/pex.py", line 472, in build_pex
    resolveds = resolver.resolve(resolvables)
  File "/home/travis/.cache/bazel/_bazel_travis/030a6cc9b27b34f2fe0e137b942f59bf/heron/bazel-out/local_linux-fastbuild/bin/third_party/pex/_pex.runfiles/__main__/third_party/pex/pex/resolver.py", line 191, in resolve
    dist = self.build(package, resolvable.options)
  File "/home/travis/.cache/bazel/_bazel_travis/030a6cc9b27b34f2fe0e137b942f59bf/heron/bazel-out/local_linux-fastbuild/bin/third_party/pex/_pex.runfiles/__main__/third_party/pex/pex/resolver.py", line 254, in build
    os.rename(target + '~', target)
OSError: [Errno 2] No such file or directory

This error happens intermittently. I thought #271 fixed the problem and upgrade the Pex, but seems that the problem is still there.

Here is a sample CI log: https://travis-ci.org/twitter/heron/builds/146219942

/cc: @kwlzn

[kwlzn]

which version of pex? can you provide a repro?

afaict, this wouldn't be related to #271 but more likely related to bazel's use of pex.

[objmagic]

we put the Pex source under third_party/pex and we have written a file called _pex.py, which serves as a customized entry to the actual Pex program. We use py_binary rule of Bazel to build it. Now I think maybe our _pex.py has done something wrong... I'll investigate problems on our side now...

[kwlzn]

seems like this boils down to this block of code: https://github.com/pantsbuild/pex/blob/master/pex/resolver.py#L259-L264

which is related to distribution caching for resolves. I wonder if in the bazel execution environment (which is presumably sandboxed), it even has access to write to the pex resolve cache (usually in ~/.pex)?

if not, you might consider passing --disable-cache to avoid attempted use of caching altogether which would skip this codepath.

beyond that, I'd suspect this has something to do with how file access works in the context of bazel's execution of pex. should be trivial to debug with a repro + a few checks and print statements.

[objmagic]

thank you so much! I will follow the hints.

[objmagic]

It seems that if multiple Travis CI job are running, this problem can happen. I am wondering if we can set PEX home directory so that we can change ~/.pex to something else for each CI job.

[kwlzn]

yes, you can control this at runtime using the PEX_ROOT environment variable:

$ PEX_ROOT=/some/alternate/path/to/.pex ./the.pex

[billonahill]

@objmagic +1 for this approach. I feel like I also got burned with what I thought was potential shared cache issues at one point, causing me to wipe out the pex cache dir before each run. I looked and I can't find where I had done this though. I think it would be good to put this under the users home and clean it before each build.

[jsirois]

I'm going to close this as answered. Setting PEX_ROOT as @kwlzn pointed out should do the trick here.