Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

T6484: Smoketest: Increase KVM memory limit #656

Merged
merged 1 commit into from
Jun 14, 2024

Conversation

MattKobayashi
Copy link
Contributor

Change Summary

Types of changes

  • Bug fix (non-breaking change which fixes an issue)
  • New feature (non-breaking change which adds functionality)
  • Code style update (formatting, renaming)
  • Refactoring (no functional changes)
  • Migration from an old Vyatta component to vyos-1x, please link to related PR inside obsoleted component
  • Other (please describe):

Related Task(s)

https://vyos.dev/T6484

Component(s) name

smoketest

Proposed changes

When running smoketests in the vyos-build container, an error message similar to the following appears:

DEBUG - test_fastnetmon (__main__.TestServiceIDS.test_fastnetmon) ... [ 4213.579433] Out of memory: Killed process 543005 (fastnetmon) total-vm:2301048kB, anon-rss:7432kB, file-rss:1812608kB, shmem-rss:0kB, UID:0 pgtables:3804kB oom_score_adj:0

Further investigation reveals that the KVM created for the smoketests is limited to 3GB of memory. Increasing this to 4GB allows the smoketests to complete and pass.

How to test

Run smoketests with make test after building ISO. When the KVM memory limit is set to 3GB, the smoketests will fail as fastnetmon is OOM killed. When it is set to 4GB, the smoketests pass.

Checklist:

  • I have read the CONTRIBUTING document
  • I have linked this PR to one or more Phabricator Task(s)
  • My commit headlines contain a valid Task id
  • My change requires a change to the documentation
  • I have updated the documentation accordingly

@MattKobayashi MattKobayashi requested a review from a team as a code owner June 14, 2024 00:44
@MattKobayashi MattKobayashi requested review from dmbaturin, sarthurdev, zdc, jestabro, sever-sever and c-po and removed request for a team June 14, 2024 00:44
Copy link
Member

@dmbaturin dmbaturin left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Are there specific tests that may fail due to insufficient memory? I have nothing against the change, but if possible, we should document the motivation for it in a comment.

@dd010101
Copy link

The snippet of OP has this information already included - test_fastnetmon in the message and that's the failing test - part of the test_service_ids_ddos-protection.py specifically - https://github.com/vyos/vyos-1x/blob/current/smoketest/scripts/cli/test_service_ids_ddos-protection.py#L52

Reproduced via Smoketest running via check-qemu-install - test VM 6 calculated cores, 3 GB RAM and the fastnetmon test fails 100% of the time due to the internal OOM killer at this exact test. Works fine with less assigned cores to the Smoketest VM but the check-qemu-install always assign 1/2 available cores thus machine with many cores causes the memory overflow over 3GB.

DEBUG - test_api_configure (__main__.TestHTTPSService.test_api_configure) ... ok
DEBUG - test_api_generate (__main__.TestHTTPSService.test_api_generate) ... ok
DEBUG - test_api_incomplete_key (__main__.TestHTTPSService.test_api_incomplete_key) ... ok
DEBUG - test_api_missing_keys (__main__.TestHTTPSService.test_api_missing_keys) ... ok
DEBUG - test_api_reset (__main__.TestHTTPSService.test_api_reset) ... ok
DEBUG - test_api_show (__main__.TestHTTPSService.test_api_show) ... ok
DEBUG - test_certificate (__main__.TestHTTPSService.test_certificate) ... ok
DEBUG -
DEBUG - ----------------------------------------------------------------------
DEBUG - Ran 11 tests in 89.177s
DEBUG -
DEBUG - OK
DEBUG - Running Testcase: /usr/libexec/vyos/tests/smoke/cli/test_service_ids_ddos-protection.py
DEBUG - test_fastnetmon (__main__.TestServiceIDS.test_fastnetmon) ... [ 5346.802073] Out of memory: Killed process 488433 (fastnetmon) total-vm:1580152kB, anon-rss:7504kB, file-rss:1286272kB, shmem-rss:0kB, UID:0 pgtables:2768kB oom_score_adj:0
DEBUG - FAIL
DEBUG -
DEBUG - ======================================================================
DEBUG - FAIL: test_fastnetmon (__main__.TestServiceIDS.test_fastnetmon)
DEBUG - ----------------------------------------------------------------------
DEBUG - Traceback (most recent call last):
DEBUG -   File "/usr/libexec/vyos/tests/smoke/cli/test_service_ids_ddos-protection

@c-po
Copy link
Member

c-po commented Jun 14, 2024

@Mergifyio backport sagitta

Copy link

mergify bot commented Jun 14, 2024

backport sagitta

✅ Backports have been created

@c-po c-po merged commit 710351e into vyos:current Jun 14, 2024
7 of 8 checks passed
c-po added a commit that referenced this pull request Jun 27, 2024
T6484: Smoketest: Increase KVM memory limit (backport #656)
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Development

Successfully merging this pull request may close these issues.

4 participants