Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

When adding a new tombstone record, delete all the older ones #2533

Merged
merged 1 commit into from
Jun 29, 2024

Conversation

jpalermo
Copy link
Member

Noticed that these are never cleaned up currently. There may be a performance hit when the first new tombstone record is added and thousands of old ones are destroyed but it shouldn't be terrible.

@jpalermo jpalermo requested review from a team, ramonskie and anshrupani and removed request for a team June 27, 2024 14:49
Copy link
Member

@aramprice aramprice left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

lgtm

@beyhan
Copy link
Member

beyhan commented Jun 28, 2024

cool finding! 👍

@jpalermo jpalermo merged commit 6410f73 into main Jun 29, 2024
4 checks passed
@jpalermo jpalermo deleted the pr_remove_tombstone_local_dns_records branch June 29, 2024 22:39
@aramprice
Copy link
Member

This PR seems to be causing consistent Deadlock errors in the MySQL integration specs.

Reformatted output from CI job above

bosh  --tty -n -e https://127.0.0.1:61702 -d simple \
--config /tmp/build/04d13fd0/bosh-src/src/tmp/integration-tests-workspace/pid-16878/client-sandbox/bosh_config.yml \
--ca-cert /tmp/build/04d13fd0/bosh-src/src/bosh-dev/assets/sandbox/ca/certs/rootCA.pem delete-deployment 

Using environment 'https://127.0.0.1:61702' as client 'production_team'

Using deployment 'simple'

Task 4

Task 4 | 03:19:29 | Deleting instances: fake-name1/a977324b-4e5a-4cbb-97ee-cc76fcd736f7 (0)
Task 4 | 03:19:29 | Deleting instances: fake-name1/479efb85-a860-4d4a-855a-db0f7fa40c40 (2)
Task 4 | 03:19:29 | Deleting instances: fake-name1/6ad77256-ae09-4650-8297-4516716d4a2a (1)
Task 4 | 03:19:31 | Deleting instances: fake-name1/479efb85-a860-4d4a-855a-db0f7fa40c40 (2) (00:00:02)
Task 4 | 03:19:31 | Deleting instances: fake-name1/6ad77256-ae09-4650-8297-4516716d4a2a (1) (00:00:02)
                  L Error: Mysql2::Error: Deadlock found when trying to get lock; try restarting transaction
Task 4 | 03:19:31 | Deleting instances: fake-name1/a977324b-4e5a-4cbb-97ee-cc76fcd736f7 (0) (00:00:02)
Task 4 | 03:19:31 | Error: Mysql2::Error: Deadlock found when trying to get lock; try restarting transaction

Task 4 Started  Sun Jun 30 03:19:29 UTC 2024
Task 4 Finished Sun Jun 30 03:19:31 UTC 2024
Task 4 Duration 00:00:02
Task 4 error

Deleting deployment 'simple':
  Expected task '4' to succeed but state is 'error'

Exit code 1

@beyhan
Copy link
Member

beyhan commented Jul 11, 2024

@aramprice @jpalermo should we open an issue for the above mentioned deadlocks or it is addressed already?

jpalermo added a commit that referenced this pull request Jul 11, 2024
The tombstone creation happens in a larger transaction, so if you both insert and clean tombstone records at the
same time you can end up in a deadlock state if that is happening in multiple threads at the same time.

This commit moves the tombstone cleanup into a scheduled job that runs once a day instead.
@jpalermo
Copy link
Member Author

New PR to move it to a scheduled job

@beyhan
Copy link
Member

beyhan commented Jul 11, 2024

Thanks! @jpalermo this was faster than the light :-)

@jpalermo
Copy link
Member Author

Convenient timing 😀

aramprice pushed a commit that referenced this pull request Jul 11, 2024
The tombstone creation happens in a larger transaction, so if you both insert and clean tombstone records at the
same time you can end up in a deadlock state if that is happening in multiple threads at the same time.

This commit moves the tombstone cleanup into a scheduled job that runs once a day instead.
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
Development

Successfully merging this pull request may close these issues.

4 participants