Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Obtain status file lock when updating full status #319

Merged
merged 1 commit into from
May 19, 2021

Conversation

shanemcd
Copy link
Member

@shanemcd shanemcd commented May 12, 2021

We are randomly seeing errors that look like this:

2021-05-03 07:59:09,461 ERROR    [a40f6c03679040dc990455ce2422c80c] awx.main.tasks job 111 (running) Exception occurred while running task
Traceback (most recent call last):
  File "/var/lib/awx/venv/awx/lib64/python3.8/site-packages/awx/main/tasks.py", line 1427, in run
    res = receptor_job.run()
  File "/var/lib/awx/venv/awx/lib64/python3.8/site-packages/awx/main/tasks.py", line 2969, in run
    receptor_ctl.simple_command(f"work release {self.unit_id}")
  File "/var/lib/awx/venv/awx/lib64/python3.8/site-packages/receptorctl/socket_interface.py", line 72, in simple_command
    return self.read_and_parse_json()
  File "/var/lib/awx/venv/awx/lib64/python3.8/site-packages/receptorctl/socket_interface.py", line 51, in read_and_parse_json
    raise RuntimeError(text[7:])
RuntimeError: unlinkat /tmp/receptor/example.com/S4kPB890: directory not empty

Copy link
Contributor

@ghjm ghjm left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Before merging this I would want to go through the codebase for the dozen or so callers of this function, and make sure none of them touch the status lock inside their update function.

@shanemcd
Copy link
Member Author

I was wrong. This does not fix the issue.

@shanemcd shanemcd closed this May 13, 2021
shanemcd added a commit to shanemcd/receptor that referenced this pull request May 13, 2021
shanemcd added a commit to shanemcd/receptor that referenced this pull request May 13, 2021
shanemcd added a commit to shanemcd/receptor that referenced this pull request May 13, 2021
@shanemcd shanemcd mentioned this pull request May 15, 2021
@shanemcd shanemcd reopened this May 19, 2021
@shanemcd
Copy link
Member Author

I think we actually want / need this now.

Co-authored-by: Seth Foster <fosterseth@users.noreply.github.com>
@shanemcd shanemcd force-pushed the add-missing-status-lock branch from 3f7156f to 2eb2243 Compare May 19, 2021 15:36
@shanemcd shanemcd enabled auto-merge (rebase) May 19, 2021 15:37
@shanemcd shanemcd merged commit 14dc61f into ansible:devel May 19, 2021
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

3 participants