KeyDB segmentation faults on 6.3.4 #7238
Labels
bug
Something isn't working
Comments
|
Yep, there was a lot of pain with new KeyDB version :( |
3 tasks
azhavoro
pushed a commit
that referenced
this issue
Dec 11, 2023
<!-- Raise an issue to propose your change (https://github.com/opencv/cvat/issues). It helps to avoid duplication of efforts from multiple independent contributors. Discuss your ideas with maintainers to be sure that changes will be approved and merged. Read the [Contribution guide](https://opencv.github.io/cvat/docs/contributing/). --> <!-- Provide a general summary of your changes in the Title above --> ### Motivation and context <!-- Why is this change required? What problem does it solve? If it fixes an open issue, please link to the issue here. Describe your changes in detail, add screenshots. --> That KeyDB version turned out rather unstable in practice, with multiple crashes and freezes observed in production. Resolved #7238 This reverts commit 118cc72. ### How has this been tested? <!-- Please describe in detail how you tested your changes. Include details of your testing environment, and the tests you ran to see how your change affects other areas of the code, etc. --> ### Checklist <!-- Go over all the following points, and put an `x` in all the boxes that apply. If an item isn't applicable for some reason, then ~~explicitly strikethrough~~ the whole line. If you don't do that, GitHub will show incorrect progress for the pull request. If you're unsure about any of these, don't hesitate to ask. We're here to help! --> - [x] I submit my changes into the `develop` branch - [ ] I have created a changelog fragment <!-- see top comment in CHANGELOG.md --> - ~~[ ] I have updated the documentation accordingly~~ - ~~[ ] I have added tests to cover my changes~~ - ~~[ ] I have linked related issues (see [GitHub docs]( https://help.github.com/en/github/managing-your-work-on-github/linking-a-pull-request-to-an-issue#linking-a-pull-request-to-an-issue-using-a-keyword))~~ - ~~[ ] I have increased versions of npm packages if it is necessary ([cvat-canvas](https://github.com/opencv/cvat/tree/develop/cvat-canvas#versioning), [cvat-core](https://github.com/opencv/cvat/tree/develop/cvat-core#versioning), [cvat-data](https://github.com/opencv/cvat/tree/develop/cvat-data#versioning) and [cvat-ui](https://github.com/opencv/cvat/tree/develop/cvat-ui#versioning))~~ ### License - [x] I submit _my code changes_ under the same [MIT License]( https://github.com/opencv/cvat/blob/develop/LICENSE) that covers the project. Feel free to contact the maintainers if that's a concern.
amjadsaadeh
pushed a commit
to amjadsaadeh/cvat
that referenced
this issue
Dec 14, 2023
<!-- Raise an issue to propose your change (https://github.com/opencv/cvat/issues). It helps to avoid duplication of efforts from multiple independent contributors. Discuss your ideas with maintainers to be sure that changes will be approved and merged. Read the [Contribution guide](https://opencv.github.io/cvat/docs/contributing/). --> <!-- Provide a general summary of your changes in the Title above --> ### Motivation and context <!-- Why is this change required? What problem does it solve? If it fixes an open issue, please link to the issue here. Describe your changes in detail, add screenshots. --> That KeyDB version turned out rather unstable in practice, with multiple crashes and freezes observed in production. Resolved cvat-ai#7238 This reverts commit 118cc72. ### How has this been tested? <!-- Please describe in detail how you tested your changes. Include details of your testing environment, and the tests you ran to see how your change affects other areas of the code, etc. --> ### Checklist <!-- Go over all the following points, and put an `x` in all the boxes that apply. If an item isn't applicable for some reason, then ~~explicitly strikethrough~~ the whole line. If you don't do that, GitHub will show incorrect progress for the pull request. If you're unsure about any of these, don't hesitate to ask. We're here to help! --> - [x] I submit my changes into the `develop` branch - [ ] I have created a changelog fragment <!-- see top comment in CHANGELOG.md --> - ~~[ ] I have updated the documentation accordingly~~ - ~~[ ] I have added tests to cover my changes~~ - ~~[ ] I have linked related issues (see [GitHub docs]( https://help.github.com/en/github/managing-your-work-on-github/linking-a-pull-request-to-an-issue#linking-a-pull-request-to-an-issue-using-a-keyword))~~ - ~~[ ] I have increased versions of npm packages if it is necessary ([cvat-canvas](https://github.com/opencv/cvat/tree/develop/cvat-canvas#versioning), [cvat-core](https://github.com/opencv/cvat/tree/develop/cvat-core#versioning), [cvat-data](https://github.com/opencv/cvat/tree/develop/cvat-data#versioning) and [cvat-ui](https://github.com/opencv/cvat/tree/develop/cvat-ui#versioning))~~ ### License - [x] I submit _my code changes_ under the same [MIT License]( https://github.com/opencv/cvat/blob/develop/LICENSE) that covers the project. Feel free to contact the maintainers if that's a concern.
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Actions before raising this issue
Steps to Reproduce
We noticed abnormally high CPU usage by the KeyDB instance (process name was
keydb-bgsave). It used 100% of 1 core, where it would normally use much less than that (before upgrading to the latest CVAT version 2.9.1). Upon further investigation we found that the KeyDB instance would segfault, in the dmesg output of my server I would get entries such as:There would also be a lot of logs in the
cvat_rediscontainer, such as:It seems to happen each time when it was testing the memory. Another error we got:
This is difficult to reproduce, but with KeyDB 6.3.4 it happened within a few hours, each time, even when clearing the
cvat_cache_dbdocker volume each time.Expected Behavior
KeyDB should not crash, and should not consume excessive CPU cycles.
Note: The CVAT system as a whole continued to perform as expected, CVAT was usable.
Possible Solution
We migrated back to KeyDB v 6.3.2 and the issue seems to have gone away. Likely caused by some changes in KeyDB 6.3.2 -> 6.3.4.
Solution for now would be to change the version of KeyDB used in docker-compose.yml from 6.3.4 back to 6.3.2.
Excerpt of docker-compose.yml:
KeyDB was upgraded to 6.3.4 in #7118. There seems to be a PR already out to fix this #7236.
Context
No response
Environment
No response
The text was updated successfully, but these errors were encountered: