Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

interrupt: Initial interrupt manager traits #3

Open
wants to merge 1 commit into
base: master
Choose a base branch
from
Open
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
2 changes: 1 addition & 1 deletion rust-vmm-ci
208 changes: 208 additions & 0 deletions src/interrupt/mod.rs
Original file line number Diff line number Diff line change
@@ -0,0 +1,208 @@
// Copyright (C) 2019 Alibaba Cloud. All rights reserved.
// Copyright 2019 Amazon.com, Inc. or its affiliates. All Rights Reserved.
// Copyright © 2019 Intel Corporation
//
// SPDX-License-Identifier: Apache-2.0 OR BSD-3-Clause

//! Traits and Structs to manage interrupt sources for devices.
//!
//! In system programming, an interrupt is a signal to the processor emitted by hardware or
//! software indicating an event that needs immediate attention. An interrupt alerts the processor
//! to a high-priority condition requiring the interruption of the current code the processor is
//! executing. The processor responds by suspending its current activities, saving its state, and
//! executing a function called an interrupt handler (or an interrupt service routine, ISR) to deal
//! with the event. This interruption is temporary, and, after the interrupt handler finishes,
//! unless handling the interrupt has emitted a fatal error, the processor resumes normal
//! activities.
//!
//! Hardware interrupts are used by devices to communicate that they require attention from the
//! operating system, or a bare-metal program running on the CPU if there are no OSes. The act of
//! initiating a hardware interrupt is referred to as an interrupt request (IRQ). Different devices
//! are usually associated with different interrupts using a unique value associated with each
//! interrupt. This makes it possible to know which hardware device caused which interrupts.
//! These interrupt values are often called IRQ lines, or just interrupt lines.
//!
//! Nowadays, IRQ lines is not the only mechanism to deliver device interrupts to processors.
//! MSI [(Message Signaled Interrupt)](https://en.wikipedia.org/wiki/Message_Signaled_Interrupts)
//! is another commonly used alternative in-band method of signaling an interrupt, using special
//! in-band messages to replace traditional out-of-band assertion of dedicated interrupt lines.
//! While more complex to implement in a device, message signaled interrupts have some significant
//! advantages over pin-based out-of-band interrupt signaling. Message signaled interrupts are
//! supported in PCI bus since its version 2.2, and in later available PCI Express bus. Some
//! non-PCI architectures also use message signaled interrupts.
//!
//! While IRQ is a term commonly used by Operating Systems when dealing with hardware
//! interrupts, the IRQ numbers managed by OSes are independent of the ones managed by VMM.
//! For simplicity sake, the term `Interrupt Source` is used instead of IRQ to represent both
//! pin-based interrupts and MSI interrupts.
//!
//! A device may support multiple types of interrupts, and each type of interrupt may support one
//! or multiple interrupt sources. For example, a PCI device may support:
//! * Legacy Irq: exactly one interrupt source.
//! * PCI MSI Irq: 1,2,4,8,16,32 interrupt sources.
//! * PCI MSIx Irq: 2^n(n=0-11) interrupt sources.
//!
//! A distinct Interrupt Source Identifier (ISID) will be assigned to each interrupt source.
//! An ID allocator will be used to allocate and free Interrupt Source Identifiers for devices.
//! To decouple the vm-device crate from the ID allocator, the vm-device crate doesn't take the
//! responsibility to allocate/free Interrupt Source IDs but only makes use of assigned IDs.
//!
//! The overall flow to deal with interrupts is:
//! * The VMM creates an interrupt manager
//! * The VMM creates a device manager, passing on an reference to the interrupt manager
//! * The device manager passes on an reference to the interrupt manager to all registered devices
//! * The guest kernel loads drivers for virtual devices
//! * The guest device driver determines the type and number of interrupts needed, and update the
//! device configuration
//! * The virtual device backend requests the interrupt manager to create an interrupt group
//! according to guest configuration information

use std::fs::File;
use std::sync::Arc;

/// Reuse std::io::Result to simplify interoperability among crates.
pub type Result<T> = std::io::Result<T>;

/// Data type to store an interrupt source identifier.
pub type InterruptIndex = u32;

/// Configuration data for legacy, pin based interrupt groups.
///
/// A legacy interrupt group only takes one irq number as its configuration.
#[derive(Copy, Clone, Debug)]
pub struct LegacyIrqGroupConfig {
/// Legacy irq number.
pub irq: InterruptIndex,
}

/// Configuration data for MSI/MSI-X interrupt groups
///
/// MSI/MSI-X interrupt groups are basically a set of vectors.
#[derive(Copy, Clone, Debug)]
pub struct MsiIrqGroupConfig {
/// First index of the MSI/MSI-X interrupt vectors
pub base: InterruptIndex,
/// Number of vectors in the MSI/MSI-X group.
pub count: InterruptIndex,
}

/// Trait to manage interrupt sources for virtual device backends.
///
/// The InterruptManager implementations should protect itself from concurrent accesses internally,
/// so it could be invoked from multi-threaded context.
pub trait InterruptManager {
type GroupConfig;
Copy link

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

We should remove "type GroupConfig" and reverts to "base: InterruptIndex, count: InterruptIndex".

Introduction of "type GroupConfig" causes troubles, it limits the InterruptManager could only create one type of InterruptSourceGroup, either LegacyIrq or MsiIrq. It works with Cloud Hypervisor because CLH handles LegacyIrq by userspace IOAPIC and handles MSI in kernel emulated LAPIC.

We use kvm emulated ioapic/lapic to support LegacyIrq, MsiIrq and VfioIrq.
So the GroupConfig breaks our design.

One the hand, revertint to base, count should work with CLH too.

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

base and count don't really make sense for all interrupts out there. In Firecracker we don't really need this, and our group will always have one interrupt, which means that our GroupConfig will probably be empty. I think it makes more sense to enable all usecases by using GroupConfig.

Copy link

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

base is always needed, the count is always 1 for firecracker so could be skipped:)


/// Create an [InterruptSourceGroup](trait.InterruptSourceGroup.html) object to manage
/// interrupt sources for a virtual device
///
/// An [InterruptSourceGroup](trait.InterruptSourceGroup.html) object manages all interrupt
/// sources of the same type for a virtual device.
///
/// # Arguments
/// * config: The interrupt group configuration
fn create_group(&self, config: Self::GroupConfig)
-> Result<Arc<Box<dyn InterruptSourceGroup>>>;

/// Destroy an [InterruptSourceGroup](trait.InterruptSourceGroup.html) object created by
/// [create_group()](trait.InterruptManager.html#tymethod.create_group).
///
/// Assume the caller takes the responsibility to disable all interrupt sources of the group
/// before calling destroy_group(). This assumption helps to simplify InterruptSourceGroup
/// implementations.
fn destroy_group(&mut self, group: Arc<Box<dyn InterruptSourceGroup>>) -> Result<()>;
}

/// Configuration data for legacy interrupts.
///
/// On x86 platforms, legacy interrupts means those interrupts routed through PICs or IOAPICs.
#[derive(Copy, Clone, Debug)]
pub struct LegacyIrqSourceConfig {}

/// Configuration data for MSI/MSI-X interrupts.
///
/// On x86 platforms, these interrupts are vectors delivered directly to the LAPIC.
#[derive(Copy, Clone, Debug, Default)]
pub struct MsiIrqSourceConfig {
/// High address to delivery message signaled interrupt.
pub high_addr: u32,
/// Low address to delivery message signaled interrupt.
pub low_addr: u32,
/// Data to write to delivery message signaled interrupt.
pub data: u32,
}

/// Configuration data for an interrupt source.
#[derive(Copy, Clone, Debug)]
pub enum InterruptSourceConfig {
/// Configuration data for Legacy interrupts.
LegacyIrq(LegacyIrqSourceConfig),
/// Configuration data for PciMsi, PciMsix and generic MSI interrupts.
MsiIrq(MsiIrqSourceConfig),
}

pub trait InterruptSourceGroup: Send + Sync {
/// Enable the interrupt sources in the group to generate interrupts.
fn enable(&mut self) -> Result<()> {
Copy link

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Suggest to revert enable() to:
fn enable(&self, configs: &[InterruptSourceConfig]) -> Result<()>;

The proposed work flow for virtio devices is:

  1. guest kernel writes MSI configuration registers, the vmm caches those configuration information.
  2. guest kernel writes virtio status register to start virtio device
    2.1) the vmm applies cached msi configuration as a batch and commit them to kvm routing table.
    2.2) invoke virtio device's activate()
  3. guest kernel writes MSI registers to update MSI configuration. Because the virtio device has been activated, the vmm calls InterruptSourceGroup::update() to directly update kvm routing table.

So we need to pass the cached MSI configuration information when calling InterruptSourceGroup::enable().

And as Paolo suggested, it would be better for enable()/disable() to take "&self" instead of "&mut self". BTW, what's the scenario needing a "&mut self" for enable()/disable()?

// Not all interrupt sources can be enabled.
// To accommodate this, we can have a no-op here.
Ok(())
}

/// Disable the interrupt sources in the group to generate interrupts.
fn disable(&mut self) -> Result<()> {
// Not all interrupt sources can be disabled.
// To accommodate this, we can have a no-op here.
Ok(())
}

/// Inject an interrupt from this interrupt source into the guest.
///
/// # Arguments
/// * index: sub-index into the group.
fn trigger(&self, index: InterruptIndex) -> Result<()>;

/// Returns an interrupt notifier from this interrupt.
///
/// An interrupt notifier allows for external components and processes
/// to inject interrupts into a guest, by writing to the file returned
/// by this method.
#[allow(unused_variables)]
fn notifier(&self, index: InterruptIndex) -> Option<File> {
// One use case of the notifier is to implement vhost user backends.
// For all other implementations we can just return None here.
None
}

/// Update the interrupt source group configuration.
///
/// # Arguments
/// * index: sub-index into the group.
/// * config: configuration data for the interrupt source.
fn update(&self, _index: InterruptIndex, _config: InterruptSourceConfig) -> Result<()> {
// Interrupt source group update is a typical MSI
// related operation. Most legacy interrupts don't
// support that.
Ok(())
}

/// Mask an interrupt from this interrupt source group.
///
/// # Arguments
/// * index: sub-index into the group.
fn mask(&self, _index: InterruptIndex) -> Result<()> {
// Not all interrupt sources can be disabled.
// To accommodate this, we can have a no-op here.
Ok(())
}

/// Unmask an interrupt from this interrupt source group.
///
/// # Arguments
/// * index: sub-index into the group.
fn unmask(&self, _index: InterruptIndex) -> Result<()> {
// Not all interrupt sources can be disabled.
// To accommodate this, we can have a no-op here.
Ok(())
}
}
1 change: 1 addition & 0 deletions src/lib.rs
Original file line number Diff line number Diff line change
Expand Up @@ -6,6 +6,7 @@
use std::cmp::{Ord, Ordering, PartialOrd};

pub mod device_manager;
pub mod interrupt;
pub mod resources;

// IO Size.
Expand Down