Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Activate virtio devices via VMM thread #2032

Merged
merged 8 commits into from
Dec 17, 2020
Merged

Activate virtio devices via VMM thread #2032

merged 8 commits into from
Dec 17, 2020

Conversation

rbradford
Copy link
Member

If a virtio device needs activation (because the device status field has been updated) signal to the VMM thread that there is a device that can be activated via an EventFd. Since the device must be ready when the write to the configuration register is completed use a barrier so that the CPU thread will wait until the VMM thread activates the device. This ensures that from the guest perspective the device is activated synchronously by the write that it made to the status register.

Fixes: #1863

This is based on an earlier PR: #1953 that didn't correctly handle the required synchronous behaviour.

@rbradford rbradford force-pushed the 2020-12-04-activate-virtio-on-vmm-thread branch from af89bc8 to 3b46616 Compare December 4, 2020 11:39
@rbradford
Copy link
Member Author

rbradford commented Dec 4, 2020 •
edited
Loading

@sameo This PR shows up a Rust (nightly) compiler bug...but otherwise works dandy!

@rbradford rbradford requested a review from sameo December 4, 2020 12:14
@rbradford
Copy link
Member Author

Here is the nightly failure: rust-lang/rust#79661

@Aaron1011 Aaron1011 mentioned this pull request Dec 4, 2020
Closed
@bryteise
Copy link
Contributor

bryteise commented Dec 6, 2020

LGTM

@rbradford rbradford force-pushed the 2020-12-04-activate-virtio-on-vmm-thread branch from 3b46616 to ef4fa3f Compare December 7, 2020 16:57
@rbradford
devices: acpi: Remove empty BusDevice::write implementation
c0700df 
The trait already includes an empty implementation of this function.

Signed-off-by: Rob Bradford <robert.bradford@intel.com>
@rbradford
misc: Make Bus::write() return an Option<Arc<Barrier>>
3901fa5 
This can be uses to indicate to the caller that it should wait on the
barrier before returning as there is some asynchronous activity
triggered by the write which requires the KVM exit to block until it's
completed.

This is useful for having vCPU thread wait for the VMM thread to proceed
to activate the virtio devices.

See cloud-hypervisor#1863

Signed-off-by: Rob Bradford <robert.bradford@intel.com>
@rbradford
vm-device, vmm: Wait for barrier if one is returned
11aee6c 
Wait for the barrier if one is provided by the result of the MMIO and
PIO write.

Signed-off-by: Rob Bradford <robert.bradford@intel.com>
@rbradford
pci, virtio-devices: Extend barrier returning through PCI code
51cc441 
We need to be able to return the barrier from the code that prepares to
activate the virtio device. This triggered by a write to the
configuration fields stored in the PCI BAR. Since bars can be accessed
by both memory mapping and through PCI config I/O several prototypes
must be changed.

Signed-off-by: Rob Bradford <robert.bradford@intel.com>
@rbradford
virtio-devices: pci_device: Split out device activation
f4b4c1b 
This can then be used to activate the device from a separate thread.

Signed-off-by: Rob Bradford <robert.bradford@intel.com>
@rbradford
vmm: Activate virtio device from VMM thread
fef246f 
When a device is ready to be activated signal to the VMM thread via an
EventFd that there is a device to be activated. When the VMM receives a
notification on the EventFd that there is a device to be activated
notify the device manager to attempt to activate any devices that have
not been activated.

As a side effect the VMM thread will create the virtio device threads.

Fixes: cloud-hypervisor#1863

Signed-off-by: Rob Bradford <robert.bradford@intel.com>
@rbradford
tests: Bump boot default timeout
01fe93f 
Starting the virtio device threads from the VMM thread has slowed down
the start of the VM when running on a highly contested system like the
CI.

Signed-off-by: Rob Bradford <robert.bradford@intel.com>
@rbradford rbradford force-pushed the 2020-12-04-activate-virtio-on-vmm-thread branch from ef4fa3f to 01fe93f Compare December 10, 2020 10:39
@rbradford
Copy link
Member Author

rbradford commented Dec 10, 2020 •
edited
Loading

I was interested in if this had any impact on boot time:

target/release/cloud-hypervisor --kernel ~/src/linux/vmlinux --disk path=~/workloads/focal.raw --cpus boot=4 --memory size=512M --cmdline "root=/dev/vda1 console=hvc0 quiet"

  master 2020-12-04-activate-virtio-on-vmm-thread
  0.210741 0.175107
  0.203597 0.173539
  0.226752 0.195774
  0.215518 0.190128
  0.205798 0.17194
  0.224076 0.184733
  0.217794 0.173808
  0.17458 0.182993
  0.176529 0.187618
  0.190063 0.195202
     
Mean 0.204545 0.1830842
Min 0.17458 0.17194
Max 0.226752 0.195774
Standard deviation 0.017618 0.008635723

target/release/cloud-hypervisor --kernel ~/src/linux/vmlinux --disk path=~/workloads/focal.raw --cpus boot=1 --memory size=512M --cmdline "root=/dev/vda1 console=hvc0 quiet"

  master 2020-12-04-activate-virtio-on-vmm-thread
  0.142239 0.148566
  0.158944 0.142539
  0.168101 0.154837
  0.138282 0.188731
  0.161172 0.160799
  0.146613 0.14604
  0.142624 0.154742
  0.140008 0.152569
  0.15988 0.158853
  0.149579 0.156416
     
Mean 0.150744 0.1564092
Min 0.138282 0.142539
Max 0.168101 0.188731
Standard deviation 0.00994 0.012047234

@rbradford
tests: integration: Retry epoll if we receive -EINTR or -EAGAIN
99c2014 
On the CI we are seeing that sometimes the epoll is receing these errors
which do not indicate a failure but that we should retry.

Signed-off-by: Rob Bradford <robert.bradford@intel.com>
@rbradford
Copy link
Member Author

I saw test_api_pause_resume fail and it looks like it was due to the barrier failing to be released:

cloud-hypervisor: 13.698136888s: INFO:vmm/src/vm.rs:413 -- Waiting for barrier
cloud-hypervisor: 13.698219693s: INFO:virtio-devices/src/block.rs:382 -- Changing cache mode to writeback
cloud-hypervisor: 13.698389304s: INFO:vmm/src/vm.rs:413 -- Waiting for barrier
cloud-hypervisor: 13.698423407s: INFO:virtio-devices/src/transport/pci_device.rs:671 -- Waiting for barrier
cloud-hypervisor: 13.698460909s: INFO:virtio-devices/src/transport/pci_device.rs:673 -- Barrier released
cloud-hypervisor: 13.69846961s: INFO:vmm/src/vm.rs:415 -- Barrier released

@sameo
Copy link
Member

sameo commented Dec 17, 2020

@rbradford I see CI is green now. Did you update the PR to fix the barrier not being released issue?

@rbradford
Copy link
Member Author

It could either be just general flakiness, or it could be that this change makes the tests more flaky.

@sameo
Copy link
Member

sameo commented Dec 17, 2020

It could either be just general flakiness, or it could be that this change makes the tests more f

I'd say let's merge that one.

@rbradford rbradford merged commit 3ee5611 into cloud-hypervisor:master Dec 17, 2020
@rbradford rbradford deleted the 2020-12-04-activate-virtio-on-vmm-thread branch January 24, 2021 17:04
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Reviewers

@sameo sameo sameo approved these changes

Assignees
No one assigned
Labels
None yet
Projects
None yet
Milestone
No milestone
Development

Successfully merging this pull request may close these issues.

virtio thread creation could be handled by vmm instead of vcpu0
3 participants
@rbradford @bryteise @sameo