Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

fix: always ignore .pyc.NNNN files from the hermetic runtime tree #1541

Merged
merged 1 commit into from
Nov 7, 2023

Conversation

rickeylev
Copy link
Contributor

@rickeylev rickeylev commented Nov 6, 2023

Part of the pyc compilation process is to create a temporary file named <name>.pyc.NNNN, where NNNN is a timestamp. Once the pyc is entirely written, this file is renamed to the regular pyc file name. These files only exist for brief periods of time, but its possible for different threads/processes to see the temporary files when computing the glob() values. Later, since the file is gone, an error is raised about the file missing.

PR #1266 mostly fixed this issue, except that the exclude for the .pyc.NNNN files for an
interpreter runtime's files was behind the ignore_root_user_error flag, which meant it
wasn't always applied. This changes it to always be applied, which should eliminate the
failures due to the missing NNNN files.

Fixes #1261

Work towards #1520

Part of the pyc compilation process is to create a temporary file named
`<name>.pyc.NNNN`, where `NNNN` is a timestamp. Once the pyc is entirely
written, this file is renamed to the regular pyc file name. These files
only exist for brief periods of time, but its possible for different
threads/processes to see the temporary files when computing the glob()
values. Later, since the file is gone, an error is raised about the file
missing.

PR bazelbuild#1266 mostly fixed this issue, except that the glob exclude for an
interpreter runtime's files was behind the `ignore_root_user_error`
flag, which meant it wasn't always applied. This changes it to always be
applied, which should eliminate the failures.

Fixes bazelbuild#1261

Work towards bazelbuild#1520
@aignas
Copy link
Collaborator

aignas commented Nov 6, 2023

Not sure why we are moving some of the files in the ignore_root_user_error but not all. Could you please add more context to the description of the PR please?

@rickeylev
Copy link
Contributor Author

PR #1266 only added the *.pyc.* ignore to the glob. The other excludes, such as *.pyc (no trailing star) were already present. So this is just moving that one line affected. I updated the description to better describe its just the NNNN pattern from #1266 being moved.

I'm not entirely sure as to why the pycs are only excluded as part of the ignore_root_user_error condition. It looks like it originates in #907. I can't tell why it is behind that guard, though. The conversation seems to go in the direction that it shouldn't be behind that guard, but then it ended up being commited behind the guard (which a follow up #1038 fixing a typo in the original PR).

Perhaps python-build-standalone changed something so the pyc files aren't invalidated anymore? Though I guess if that was true, there wouldn't be any NNNN files being created. Also, this pyc.NNNN failure seems to only show up on windows now; so perhaps this is a no-op for linux, but something about python-built-standalone on Windows still needs it.

@rickeylev rickeylev added this pull request to the merge queue Nov 7, 2023
Merged via the queue into bazelbuild:main with commit 955da69 Nov 7, 2023
3 checks passed
@aignas
Copy link
Collaborator

aignas commented Nov 7, 2023 via email

@rickeylev rickeylev deleted the ignore.tmp.pyc.files branch November 7, 2023 22:43
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

flaky test: bzlmod integration test on MacOS
2 participants