This repository has been archived by the owner on Dec 7, 2023. It is now read-only.
extremely slow vm networking? #827
Comments
|
That's really peculiar. With just a small number of VM's on my WSL2 ignite host, I'm seeing effectively native bandwidth on my gigabit uplink using the CNI bridge. Maybe @bboreham or CNI bridge-plugin maintainer would know how extensive usage could cause the host kernel to slow down WRT bridge networking? |
|
Came back to this because I saw the issue again. There are a great many files in There are only 15 VMs running on this system. VMs are terminating normally using This issue seems to be stopping simple things like |
|
Inside VM: Outside VM: |
|
Any ideas @bboreham? Hi btw :-) 👋 |
|
#442 (comment) indicates that iptables rules were once cleaned up, but I'm seeing them not being cleaned up on |
|
same behaviour with newer CNI as well: |
|
to see what was going on, I moved I am seeing both ADD and DEL commands: |
|
this seems to be operating correctly, so my assumption is now that ignite is doing something with iptables rules itself that it's failing to clean up. I'm not sure though. I'm not sure why the CNI bridge plugin doesn't release the IP addresses, very many files in |
|
possibly related, i am running |
|
i guess actually we are using the firewall plugin in CNI to create the iptables rules that aren't being cleared up? and host-local plugin for IPAM? |
|
adding instrumentation to |
|
for reference: |
|
I've worked around this for now by writing my own code which interacts with |
|
I'm not sure if ignite forces docker runtime to use CNI (it's not trivial) but wouldn't it make sense to use |
|
--runtime docker together with --network-plugin docker-bridge worked when I
ran it manually but mysteriously failed (couldn't ping VMs) when running it
from the code which wraps ignite in our project
…On Fri, May 7, 2021 at 9:56 AM Michael Kashin ***@***.***> wrote:
I'm not sure if ignite forces docker runtime to use CNI (it's not trivial)
but wouldn't it make sense to use --runtime docker together with --network-plugin
docker-bridge? In this case I don't see any stale entries in my iptables
—
You are receiving this because you authored the thread.
Reply to this email directly, view it on GitHub
<#827 (comment)>,
or unsubscribe
<https://github.com/notifications/unsubscribe-auth/AACATUROGL37MN62IADHIITTMOTLRANCNFSM43VHEY6Q>
.
--
lukemarsden.net
***@***.***
@lmarsden <https://twitter.com/lmarsden>
|
|
not sure why you'd have a problem doing this via API, it should work the same way. But as for the IPT leaking with docker runtime, I think I've found the issue -- IPT rules are setup by the CNI plugin using proper docker container ID as the "id" in IPT rule comments, however, when they are being removed, the call to - if err = removeNetworking(vm.PrefixedID(), vm.Spec.Network.Ports...); err != nil {
+ if err = removeNetworking(vm.Status.Runtime.ID, vm.Spec.Network.Ports...); err != nil { |
|
thanks @networkop good spot. Any chance we could get this fix into a release please? |
|
I guess this made it into https://github.com/weaveworks/ignite/releases/tag/v0.10.0? |
|
@lukemarsden yes, it did 2f840ad . |
Sign up for free
to subscribe to this conversation on GitHub.
Already have an account?
Sign in.
Has anyone seen VM networking being very slow and flaky? This is with ignite 0.8.0.
Speed test (python speedtest-cli package) outside the VM is 1gig+
Speed test inside the VM is like 14mbit...
Possibly related, there are 100k+ files in
/var/lib/cni. But I'm seeing networking flakiness and slowness even when I clean out/var/lib/cni. Starting VMs does speed up again when/var/lib/cniis cleared out though.The text was updated successfully, but these errors were encountered: