Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Creating new deployment and reset sagas leads to high load and accessibility problems with independent deployments #2160

Open
pavsenin opened this issue Dec 1, 2021 · 0 comments
Labels
bug Issues related to defects and incorrect/unexpected feature behavior severity: major Issues with high impact on the product, e.g., new packages, installation issues, etc.

Comments

@pavsenin
Copy link

pavsenin commented Dec 1, 2021

Describe the bug
We have a task to automate our release. It does the following steps:

  1. Create new deployment (resolve-cloud deploy)
  2. Clone old event store to backup it (resolve-cloud clone)
  3. Link new deployment old event store (resolve-cloud link)
  4. Reset read models and sagas for new deployment (resolve-cloud read-models reset all & sagas reset-all)
  5. Manually create about 5 projects, 1 team, remove 3 projects (about 20 events).

Yesterday we run this script twice in the afternoon.
After that our independent main production deployment had the following issues:

  • 20-40 sec delays for any api requests
  • a lot of 503 errors

In new deployments event store adds about 5000 events in hour without any activity.
In a 5 hours the event store had 25000 events

After two hours the situation became better and we tried to reproduce the issue and run the steps above one more time, exactly after that the issue with production deployment returned.
After that we decided to delete all three new deployments and in a minute the load became zero and any issues were gone.

  • Three new deployment ids: tthtfa cyixfh menhr6, event store id l5pcpg
  • ReSolve Version [e.g. 0.32.0]
@MrCheater MrCheater added bug Issues related to defects and incorrect/unexpected feature behavior severity: major Issues with high impact on the product, e.g., new packages, installation issues, etc. labels Dec 1, 2021
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
bug Issues related to defects and incorrect/unexpected feature behavior severity: major Issues with high impact on the product, e.g., new packages, installation issues, etc.
Projects
None yet
Development

No branches or pull requests

2 participants