Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Import of big data sets #398

Open
dmitry-ratushnyy opened this issue Apr 12, 2024 · 2 comments
Open

Import of big data sets #398

dmitry-ratushnyy opened this issue Apr 12, 2024 · 2 comments
Labels
bug Something isn't working

Comments

@dmitry-ratushnyy
Copy link
Contributor

Steps to reproduce

  1. Deploy MongoDB cluster, for example on LXD
  2. Obtain connection URI
  3. Install mongosh on host machine in case of LXD or copy your data set into juju unit
  4. Using "charmed-mongodb.mongoimport" (from inside of juju unit) or mongoimport (from host machine) start import data
  5. Periodically check free space on disk.

Expected behavior

Import of 70 GB of data should not consume 500+GB of disk space

Actual behavior

Disk space is being consumed very quickly. This most likely happens due to audit logs.
Log rotation and allowing users to opt out from audit logging should solve the problem

Versions

stable/164

Operating system:

Juju CLI:

Juju agent:

Charm revision:

LXD:

Log output

Juju debug log:

Additional context
@dmitry-ratushnyy dmitry-ratushnyy added the bug Something isn't working label Apr 12, 2024
@github-actions GitHub Actions
Copy link

https://warthogs.atlassian.net/browse/DPE-4058

@dmitry-ratushnyy
Copy link
Contributor Author

We are working on resolving this issue

@dmitry-ratushnyy dmitry-ratushnyy mentioned this issue Apr 23, 2024
Merged
Mehdi-Bendriss pushed a commit that referenced this issue Apr 29, 2024
@dmitry-ratushnyy
Add log rotation with "logrotate" (#409)
34dba61 
## Issue
[ISSUE-398](#398)
At the current moment Charmed MongoDB logs, can grow very fast,
especially audit log.
For example when a user imports data sets.

## Solution
Support log rotation to avoid consuming too much space with log files.
Log will be rotated if they grow bigger than 100M, the check for log
files size will be performed every minute

## Result of log rotation
<img width="635" alt="Screenshot 2024-04-23 at 18 59 47"
src="https://github.com/canonical/mongodb-operator/assets/132273757/c1ccbd81-ca9d-4678-9423-0f2374f7226d">
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees
No one assigned
Labels
bug Something isn't working
Projects
None yet
Milestone
No milestone
Development

No branches or pull requests
1 participant
@dmitry-ratushnyy