Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

OSError: : [Errno 2] No such file or directory #293

Closed
objmagic opened this issue Jul 21, 2016 · 8 comments
Closed

OSError: : [Errno 2] No such file or directory #293

objmagic opened this issue Jul 21, 2016 · 8 comments
Labels

Comments

@objmagic
Copy link

objmagic commented Jul 21, 2016

Hi,

Pex is used to generate Python executable files in twitter/heron. However, we sometimes can see the following error message that makes build fail:

    sys.exit(main())
  File "/home/travis/.cache/bazel/_bazel_travis/030a6cc9b27b34f2fe0e137b942f59bf/heron/bazel-out/local_linux-fastbuild/bin/third_party/pex/_pex.runfiles/__main__/third_party/pex/_pex.py", line 159, in main
    resolver_options_builder, interpreter=interpreter)
  File "/home/travis/.cache/bazel/_bazel_travis/030a6cc9b27b34f2fe0e137b942f59bf/heron/bazel-out/local_linux-fastbuild/bin/third_party/pex/_pex.runfiles/__main__/third_party/pex/pex/bin/pex.py", line 472, in build_pex
    resolveds = resolver.resolve(resolvables)
  File "/home/travis/.cache/bazel/_bazel_travis/030a6cc9b27b34f2fe0e137b942f59bf/heron/bazel-out/local_linux-fastbuild/bin/third_party/pex/_pex.runfiles/__main__/third_party/pex/pex/resolver.py", line 191, in resolve
    dist = self.build(package, resolvable.options)
  File "/home/travis/.cache/bazel/_bazel_travis/030a6cc9b27b34f2fe0e137b942f59bf/heron/bazel-out/local_linux-fastbuild/bin/third_party/pex/_pex.runfiles/__main__/third_party/pex/pex/resolver.py", line 254, in build
    os.rename(target + '~', target)
OSError: [Errno 2] No such file or directory

This error happens intermittently. I thought #271 fixed the problem and upgrade the Pex, but seems that the problem is still there.

Here is a sample CI log: https://travis-ci.org/twitter/heron/builds/146219942

/cc: @kwlzn

@kwlzn
Copy link
Contributor

kwlzn commented Jul 21, 2016

which version of pex? can you provide a repro?

afaict, this wouldn't be related to #271 but more likely related to bazel's use of pex.

@objmagic
Copy link
Author

objmagic commented Jul 21, 2016

we put the Pex source under third_party/pex and we have written a file called _pex.py, which serves as a customized entry to the actual Pex program. We use py_binary rule of Bazel to build it. Now I think maybe our _pex.py has done something wrong... I'll investigate problems on our side now...

@kwlzn
Copy link
Contributor

kwlzn commented Jul 21, 2016

seems like this boils down to this block of code: https://github.com/pantsbuild/pex/blob/master/pex/resolver.py#L259-L264

which is related to distribution caching for resolves. I wonder if in the bazel execution environment (which is presumably sandboxed), it even has access to write to the pex resolve cache (usually in ~/.pex)?

if not, you might consider passing --disable-cache to avoid attempted use of caching altogether which would skip this codepath.

beyond that, I'd suspect this has something to do with how file access works in the context of bazel's execution of pex. should be trivial to debug with a repro + a few checks and print statements.

@objmagic
Copy link
Author

thank you so much! I will follow the hints.

@objmagic
Copy link
Author

It seems that if multiple Travis CI job are running, this problem can happen. I am wondering if we can set PEX home directory so that we can change ~/.pex to something else for each CI job.

@kwlzn
Copy link
Contributor

kwlzn commented Jul 27, 2016

yes, you can control this at runtime using the PEX_ROOT environment variable:

$ PEX_ROOT=/some/alternate/path/to/.pex ./the.pex

@billonahill
Copy link

@objmagic +1 for this approach. I feel like I also got burned with what I thought was potential shared cache issues at one point, causing me to wipe out the pex cache dir before each run. I looked and I can't find where I had done this though. I think it would be good to put this under the users home and clean it before each build.

@jsirois
Copy link
Member

jsirois commented Aug 6, 2019

I'm going to close this as answered. Setting PEX_ROOT as @kwlzn pointed out should do the trick here.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
Projects
None yet
Development

No branches or pull requests

4 participants