Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Sync data very slowly when count exceeds 280 millions #731

Open
Songyue-ke opened this issue Sep 3, 2024 · 1 comment
Open

Sync data very slowly when count exceeds 280 millions #731

Songyue-ke opened this issue Sep 3, 2024 · 1 comment

Comments

@Songyue-ke
Copy link

Songyue-ke commented Sep 3, 2024

Monstache config:
version: 6.7.17
all the config in .toml are default except the mongo url and elasticsearch url config
the machine specification: 4G 4U

ES specification
1.4 TB storage 32GB RAM 5vCPU

MongoDB:
1500GB storage 256GB RAM
the count of the collection that want to sync to ES is 365millions

When the data was first synchronized, it was very fast, 17 million data is synchronized every hour. However, with more and more data import, the speed is getting slower and slower, and when the data exceeds 280 million, it is only tens of thousands of synchronization per hour.
image

@Songyue-ke
Copy link
Author

Is there any performance enhancement solution?

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

No branches or pull requests

1 participant