-
Notifications
You must be signed in to change notification settings - Fork 616
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Talos not working with Proxmox. Memory overloads on boot and does not release for a long time. #3054
Comments
Also, doesn't work in Virtualbox. Any other os releases the memory back, but Talos, is holding on to the ram, or flooding it with something that doesn't get garbage collected. When you turn the vm off the memory comes back. |
Virtual Box just crashed out my mac. If I give the VM 3 gigs. Talos will Boot. If I give it 16 Gig. Even though I have 8 gig of 16 Gig used. It will freeze up my computer. If I do the same thing with Ubuntu. It will not freeze up my computer. |
auto-init clearing system memory may take some time. |
10.1.10.21: kern: info: [2021-01-14T16:15:59.280740804Z]: Memory: 15938488K/16416224K available (24599K kernel code, 3524K rwdata, 6052K rodata, 1756K init, 952K bss, 477476K reserved, 0K cma-reserved) |
10.1.10.21: kern: info: [2021-01-14T16:15:33.157948425Z]: mem auto-init: clearing system memory may take some time... |
Mind, This is the Kernel dump which must be coming from within the VM, although the memory issue is sitting on the Baremetal itself. Because Talos Dashboard reported back more free memory than was actualy available. As if it was reading from the VM and what the vm was saying is available. Compared with what it knows it has. The memory it was using was not compared with the memory that was actually consumed on the baremetal appliction in Htop. /////////////////////////////// 10.1.10.21: kern: notice: [2021-01-14T16:15:32.559459804Z]: Linux version 5.10.1-talos (@buildkitsandbox) (gcc (GCC) 10.2.0, GNU ld (GNU Binutils) 2.35) #1 SMP Fri Dec 18 13:31:56 UTC 2020 |
I wonder if this is something caused by xz decompression. |
https://zoomadmin.com/HowToLinux/LinuxCommand/xz Potential garbage collection issue and issue with not setting max memory limit on decompression. |
This is when the ram ramps up. brandon 13 minutes ago Andrew Rynhard 13 minutes ago brandon 13 minutes ago brandon 12 minutes ago brandon 12 minutes ago brandon 12 minutes ago brandon 11 minutes ago brandon 9 minutes ago brandon 9 minutes ago brandon 8 minutes ago brandon 8 minutes ago brandon 8 minutes ago brandon 7 minutes ago brandon < 1 minute ago brandon < 1 minute ago |
It is believed to be related to the KSPP default security settings of Talos, specifically one has to disable two options by adding two kernel args: |
This should have been already solved in Talos 0.9-0.10. |
This might be related to why Talos is not working with Devspace.sh, but it might a different issue. I'm not sure. There is something wrong with the way memory is being handled in the boot kernel. When you install Talos, or even reboot an install, it floods the memory. You can not see this from Proxmox, or Kubernetes. You have to go into the terminal of Proxmox and run something like Atop, or Htop to see the memory usage is being consumed. This will not report correctly from Talos dashboard either. It reports as memory consumption available. But this is not true. It's just showing how much you allocated, Such as 50 GIG, and how much is being used currently 2 Gig. But on the backend Talos has flooded the memory on boot. If you load 3 machines with Balooning and over provision. An install of Ubuntu will only use what memory is available. But Talos locks that memory and slowly releases it back to the VM.
Bug Report
Description
Logs
Environment
talosctl version --nodes <problematic nodes>
]kubectl version --short
]The text was updated successfully, but these errors were encountered: