Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Improve vttablet memory consumption with ceph backups #4804

Closed
deepthi opened this issue Apr 12, 2019 · 2 comments
Closed

Improve vttablet memory consumption with ceph backups #4804

deepthi opened this issue Apr 12, 2019 · 2 comments

Comments

@deepthi
Copy link
Member

deepthi commented Apr 12, 2019

Reported by @swallowstalker
While taking a backup using ceph storage, vttablet memory consumption goes super high.
Setup details:
vttablet is in one pod with two containers,
200 GB memory for mysqld container
4 GB memory for vttablet container
the database size is 80-ish GB
vttablet memory usage peaks around 9 GB
Ceph backup 2019-01-22

Backup of the same DB with file storage consumes much less memory (< 500MB)
file backup 2019-01-29

After upgrading the minio-go library to a newer version in #4598, the memory consumption for a ceph backup decreased somewhat, but is still significantly higher than the file storage mode

ceph backup 2019-03-28

@deepthi
Copy link
Member Author

deepthi commented Apr 12, 2019

The fix for this might be as simple as detecting the file size and passing it correctly to the minio-go API instead of passing it as -1.
https://github.com/vitessio/vitess/blob/master/go/vt/mysqlctl/cephbackupstorage/ceph.go#L91

For context on similar issues: minio/minio-go#848

@deepthi
Copy link
Member Author

deepthi commented Dec 3, 2019

Fixed by #5369

@deepthi deepthi closed this as completed Dec 3, 2019
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
None yet
Development

No branches or pull requests

2 participants