Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

OOM killer: onedb purge-done eat too much RAM #3269

Closed
7 tasks
baby-gnu opened this issue Apr 24, 2019 · 3 comments
Closed
7 tasks

OOM killer: onedb purge-done eat too much RAM #3269

baby-gnu opened this issue Apr 24, 2019 · 3 comments
Assignees
@al3xhh
Labels
Category: Core & System Community Priority: Normal Status: Accepted Type: Bug
Milestone
Release 5.8.3

Comments

@baby-gnu
Copy link
Contributor

Description

After upgrading to 5.8.1, I wanted to shrink a little my MySQL database (12GB on disk for 5.6GB uncompressed SQL file).

I tried the onedb purge-done command but it was killed by the kernel, causing an Out Of Memory Killer run.

To Reproduce

We have 211 768 VMs in our MySQL database since 2014, I just ran onedb purge-done.

Expected behavior

The command should not eat all the memory.

Details

  • Affected Component: Database
  • Hypervisor: KVM
  • Version: 5.8.1

Additional context

@xorel told me to use the --start and --end option, so I did a purge 3 months by 3 months. It take around 6GB of RAM on our 12GB frontend to run.

Progress Status

  • Branch created
  • Code committed to development branch
  • Testing - QA
  • Documentation
  • Release notes - resolved issues, compatibility, known issues
  • Code committed to upstream release/hotfix branches
  • Documentation committed to upstream release/hotfix branches
@baby-gnu
Copy link
Contributor Author

I did not have the same issue with onedb purge-history.

@al3xhh al3xhh self-assigned this Apr 29, 2019
@al3xhh al3xhh added this to the Release 5.8.2 milestone Apr 29, 2019
al3xhh pushed a commit to al3xhh/one that referenced this issue May 27, 2019
B OpenNebula#3269: onedb purge-done eat too much RAM
26e6448 
    * Divide the function between delete VMs and history records
    * Delete in group of 1000 VMs
al3xhh pushed a commit to al3xhh/one that referenced this issue May 27, 2019
B OpenNebula#3269: onedb purge-done eat too much RAM
560bf14
al3xhh pushed a commit to al3xhh/one that referenced this issue May 28, 2019
B OpenNebula#3269: onedb purge-done eat too much RAM
bf4e4a6 
    * Process done VMs in pages, instead of all together
    * Add new method to iterate over pages
    * Add new CLI paramter to specify pages size
@al3xhh
Copy link
Contributor

al3xhh commented May 28, 2019

PRs to merge in master and 5.8:

Code: #3376
Docs: OpenNebula/docs#635

al3xhh pushed a commit to al3xhh/one that referenced this issue May 28, 2019
B OpenNebula#3269: onedb purge-done eat too much RAM
9d03d16 
    * Process done VMs in pages, instead of all together
    * Add new method to iterate over pages
    * Add new CLI paramter to specify pages size
al3xhh pushed a commit to al3xhh/one that referenced this issue May 28, 2019
B OpenNebula#3269: onedb purge-done eat too much RAM
74082bd 
    * Process done VMs in pages, instead of all together
    * Add new method to iterate over pages
    * Add new CLI paramter to specify pages size
rsmontero pushed a commit that referenced this issue May 28, 2019
B #3269: onedb purge-done eat too much RAM (#3376)
b386ae4 
    * Process done VMs in pages, instead of all together
    * Add new method to iterate over pages
    * Add new CLI paramter to specify pages size
rsmontero pushed a commit that referenced this issue May 28, 2019
@rsmontero
B #3269: onedb purge-done eat too much RAM (#3376)
ef8fe76 
    * Process done VMs in pages, instead of all together
    * Add new method to iterate over pages
    * Add new CLI paramter to specify pages size

(cherry picked from commit b386ae4)
rsmontero pushed a commit to OpenNebula/docs that referenced this issue May 28, 2019
B OpenNebula/one#3269: add bug to resolved issues (#635)
7ee5175
@al3xhh
Copy link
Contributor

al3xhh commented May 28, 2019

Hello @baby-gnu

I have made an improvement on this, could you please try it and let me know if it works for you?

Thanks!

@al3xhh al3xhh closed this as completed May 28, 2019
tinova added a commit to OpenNebula/docs that referenced this issue Jun 19, 2019
@tinova
Squashed commit of the following:
bc7c142 
commit 9e339df
Author: Tino Vazquez <tinova@opennebula.org>
Date:   Mon Jun 17 16:58:49 2019 +0200

    M #-: RNs for 5.8.3

commit 1c5c4da
Author: Tino Vazquez <tinova@opennebula.org>
Date:   Mon Jun 17 13:05:46 2019 +0200

    M #-: Reduce number of upgrade guides

    Two new categories, upgrading from 4.x.x and from 3.x.x

commit 5834470
Author: Christian González <cgonzalez@opennebula.systems>
Date:   Mon Jun 17 09:59:09 2019 +0200

    F #3393: Add references to DB maintenance section (#665)

    (cherry picked from commit 8ba5fde)

commit 807f50a
Author: Jorge Lobo <47326048+jloboescalona2@users.noreply.github.com>
Date:   Fri Jun 14 18:13:05 2019 +0200

    B #3430: fix bug lost icons VNC in federation (#663)

    Signed-off-by: Jorge Lobo <jlobo@opennebula.systems>

commit 2ac9001
Author: Christian González <cgonzalez@opennebula.systems>
Date:   Fri Jun 14 15:55:10 2019 +0200

    F #3393: Add information about DB maintenance (#655)

commit d655ea7
Author: Christian González <cgonzalez@opennebula.systems>
Date:   Fri Jun 14 15:42:37 2019 +0200

    F #1324: Option to not store monitoring data to the DB (#656)

commit d7b5cea
Author: Angel Luis Moya Gonzalez <amoya@opennebula.systems>
Date:   Fri Jun 14 13:37:10 2019 +0200

    B #3225: Add IDE management explanation

commit ea703be
Author: Tino Vázquez <cvazquez@opennebula.systems>
Date:   Fri Jun 14 13:32:57 2019 +0200

    B #3430: Add resolved issues reference

commit 67f06f9
Author: Angel Luis Moya Gonzalez <amoya@opennebula.systems>
Date:   Fri Jun 14 13:05:46 2019 +0200

    Update resolved_issues_583.rst (#661)

commit d961a6c
Author: Ruben S. Montero <rsmontero@opennebula.org>
Date:   Wed Jun 12 17:49:41 2019 +0200

    B OpenNebula/one#3393: Reduce DB disk space

commit 9258381
Author: Daniel Clavijo Coca <dclavijo@opennebula.systems>
Date:   Tue Jun 11 04:49:41 2019 -0500

    F #3406: Update resolved issues (#653)

commit 08781b9
Author: Daniel Clavijo Coca <dclavijo@opennebula.systems>
Date:   Tue Jun 11 04:47:05 2019 -0500

    B #3390: Update resolved issues (#654)

commit aacb7e4
Author: Alejandro Huertas Herrero <ahuertas@opennebula.systems>
Date:   Mon Jun 10 12:25:55 2019 +0200

    F OpenNebula/one#3378: add link to the new onezone.yaml (#652)

commit 1c8d842
Author: Alejandro Huertas Herrero <ahuertas@opennebula.systems>
Date:   Mon Jun 10 12:00:59 2019 +0200

    F OpenNebula/one#3378: add feature to resolved issues (#651)

    * F OpenNebula/one#3378: add feature to resolved issues

    * Update resolved_issues_583.rst

commit bf7473a
Author: Ruben S. Montero <rsmontero@opennebula.org>
Date:   Fri Jun 7 16:59:21 2019 +0200

    Update resolved_issues_583.rst

commit 977557f
Author: Alejandro Huertas Herrero <ahuertas@opennebula.systems>
Date:   Tue Jun 4 10:09:18 2019 +0200

    B OpenNebula/one#1710: add bug to resolved issues (#641)

commit 015f924
Author: Jorge Lobo <47326048+jloboescalona2@users.noreply.github.com>
Date:   Mon Jun 3 17:23:33 2019 +0200

    B # OpenNebula/one#3057: add resolved_issue (#640)

    Signed-off-by: Jorge Lobo <jlobo@opennebula.systems>

commit 89ac977
Author: Tino Vazquez <tinova@opennebula.org>
Date:   Mon Jun 3 13:44:57 2019 +0200

    F #3366: Add feature to release notes

    Now a group base can be different from user base

commit cebc55c
Author: Steven Armstrong <steven-github@armstrong.cc>
Date:   Mon Jun 3 13:40:52 2019 +0200

    M #: document new :group_base ldap config (#638)

    implemented in OpenNebula/one#3379

    Signed-off-by: Steven Armstrong <steven.armstrong@id.ethz.ch>
    (cherry picked from commit 33d8a0d)

commit 9c7225b
Author: Alejandro Huertas Herrero <ahuertas@opennebula.systems>
Date:   Mon Jun 3 13:03:52 2019 +0200

    B OpenNebula/one#2047: add bug to resolved issues (#642)

commit b4daae2
Author: Jorge Lobo <47326048+jloboescalona2@users.noreply.github.com>
Date:   Thu May 30 18:24:35 2019 +0200

    B # OpenNebula/one#3368: add bug (#637)

    Signed-off-by: Jorge Lobo <jlobo@opennebula.systems>

commit e4d232d
Author: Daniel Clavijo Coca <dclavijo@opennebula.systems>
Date:   Thu May 30 03:35:24 2019 -0500

    M #: doc required one restart (#636)

commit 7ee5175
Author: Alejandro Huertas Herrero <ahuertas@opennebula.systems>
Date:   Tue May 28 17:26:52 2019 +0200

    B OpenNebula/one#3269: add bug to resolved issues (#635)

commit d10efbb
Author: Ruben S. Montero <rsmontero@opennebula.systems>
Date:   Mon May 27 13:05:45 2019 +0200

    Update resolved_issues_583.rst

commit 808b6bd
Author: Tino Vazquez <tinova@opennebula.org>
Date:   Fri May 24 12:44:55 2019 +0200

    F #964: Remove FILES in vCenter context limitation

commit be2c062
Author: Tino Vazquez <tinova@opennebula.org>
Date:   Fri May 24 12:30:49 2019 +0200

    F #964: Add note in RNs

commit 8fcdf2e
Author: Alejandro Huertas Herrero <ahuertas@opennebula.systems>
Date:   Thu May 23 11:44:48 2019 +0200

    B OpenNebula/one#3357: add bug to resolved issues (#633)

commit 6f86bb4
Author: Tino Vazquez <tinova@opennebula.org>
Date:   Tue May 21 11:16:14 2019 +0200

    B #3335: Add note on Sunstone vector attributes

    (cherry picked from commit eb194c8)

commit 52359cc
Author: Tino Vazquez <tinova@opennebula.org>
Date:   Tue May 21 11:12:50 2019 +0200

    B #3335: Add user template Sunstone confs

commit 840edcb
Author: sergiojvg <svega@opennebula.systems>
Date:   Mon May 20 11:43:25 2019 -0500

    F #3337: Resolved issues (#630)

commit bca1a32
Author: Daniel Clavijo Coca <dclavijo@opennebula.systems>
Date:   Mon May 20 04:53:00 2019 -0500

    F OpenNebula#3222: updated resolved issues (#626)

commit 5bd33c4
Author: Tino Vazquez <tinova@opennebula.org>
Date:   Fri May 17 18:29:00 2019 +0200

    B #3331: Support token behind proxy

commit 501a028
Author: jmdelafe <13242860+jmdelafe@users.noreply.github.com>
Date:   Fri May 17 11:51:17 2019 -0400

    B OpenNebula/one#3280 and B OpenNebula/one#3281 fixed (#629)

    * Update resolved_issues_583.rst

    * Update resolved_issues_583.rst

    * Update resolved_issues_583.rst

commit 1d1b563
Author: Vlastimil Holer <vholer@opennebula.systems>
Date:   Tue May 14 17:21:44 2019 +0200

    M #-: Bump version to 5.8.3, scaffolding (#625)

commit 8dfe0c1
Author: Tino Vazquez <tinova@opennebula.org>
Date:   Tue May 14 17:03:18 2019 +0200

    M #-: Fix hotfix box for 5.8.2

    (cherry picked from commit 51d5a82)
rsmontero pushed a commit that referenced this issue Oct 29, 2024
@vichansson @tinova
B #6751: Fix raw data validation handling (#3269)
5333608 
Signed-off-by: Victor Hansson <vhansson@opennebula.io>
Co-authored-by: Tino Vázquez <cvazquez@opennebula.io>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees

@al3xhh al3xhh

Labels
Category: Core & System Community Priority: Normal Status: Accepted Type: Bug
Projects
None yet
Milestone
Release 5.8.3
Development

No branches or pull requests
3 participants
@rsmontero @baby-gnu @al3xhh