Tianocore EDK2 UEFI firmware support

[X547]

Reference: https://github.com/tianocore/edk2/tree/master/OvmfPkg/RiscVVirt

Note that EDK 2 support both ACPI and FDT hardware definitions, so existing FDT generation should be fine. Some NVRAM emulation may be needed.

[LekKit]

Afaik pstore is CFI flash device, which I've looked into but never finished it, I'll check on that.

Did you try building and running a QEMU virt firmware?

[LekKit]

Okay I've built the firmware. It seems that RISCV_VIRT_CODE.fd should be loaded as S-mode kernel, and RISCV_VIRT_VARS.fd should be in rw pstore flash device.

I've tried to run RISCV_VIRT_CODE.fd as -kernel in RVVM, but it simply hangs. Note the same thing happens under QEMU if we replicate the approach, so apparently we really need that pstore thingy.

Here is a QEMU cmdline that actually works:

qemu-system-riscv64 -M virt,pflash0=pflash0,pflash1=pflash1,acpi=off -nographic -blockdev node-name=pflash0,driver=file,read-only=on,filename=RISCV_VIRT_CODE.fd -blockdev node-name=pflash1,driver=file,filename=RISCV_VIRT_VARS.fd

Here is my build from tianocore/edk2@c0dfe3e: riscv_edk2.zip

[X547]

Did you try building and running a QEMU virt firmware?

Yes, I built it and managed to run Haiku in QEMU with it. It builds surprisingly easy in Haiku host using Clang cross compiler.

I used pflash for specifying EDK2 firmware ROM, other options did not work.

[LekKit]

Question: Does it only support UEFI or some way to simply load a kernel like with U-Boot scripts is also present?

[X547]

I am not completely confident, but I think that EDK2 supports running EFI PE binaries only and no scripts are supported. Some Linux distros such as Fedora provide EFI boot loader for riscv64 target. GRUB also probably works.

[LekKit]

For now I figured that EDK2 isn't loaded at usual 0x80200000 kernel payload address, but at some different location and QEMU is using fw_dynamic instead of fw_jump to specify jump address.

Is there some kind of spec/documentation to all of this?

[X547]

I suspect that it use dynamic address allocation. It should just work after implementing requirements such as pflash without need to know all inner workings.

[LekKit]

I am able to get EDK2 output even when I use RVVM device tree with QEMU (or any tiny device tree with plic, clint, uart only). It crashes, but it at least outputs a crash so that I know it tried to do something.

!!!! RISCV64 Exception Type - 000000000000000D(EXCEPT_RISCV_LOAD_ACCESS_PAGE_FAULT) !!!!
     t0 = 0x00000000000000000        t1 = 0x000000000836788C8
     t2 = 0x00000000000001000        t3 = 0x00000000000000010
     t4 = 0x00000000080026DDA        t5 = 0x0000000000000000F
     t6 = 0x00000000000000027        s0 = 0x00000000083FFFAF0
     s1 = 0x00000000000000001        s2 = 0x00000000000000000
     s3 = 0x00000000020000000        s4 = 0x00000000000000000
     s5 = 0x00000000087E00000        s6 = 0x08000000A00006800
     s7 = 0x00000000080020040        s8 = 0x00000000000002000
     s9 = 0x000000000800226E0       s10 = 0x00000000000000000
    s11 = 0x00000000000000000        a0 = 0x00000000087FA2018
     a1 = 0x00000000022000000        a2 = 0x00000000000040000
     a3 = 0x00000000000040000        a4 = 0x00000000022000000
     a5 = 0x0000000002203FFF8        a6 = 0x00000000000000001
     a7 = 0x00000000083FFFA64      zero = 0x00000000000000000
     ra = 0x00000000087EAE242        sp = 0x00000000083684BD4
     gp = 0x00000000000000000        tp = 0x00000000080027000
   sepc = 0x00000000087EAE53E   sstatus = 0x08000000200006120
  stval = 0x0000000002203FFF8

On the other hand when fully running on RVVM and passing EDK2 as -kernel payload, nothing is output, so I want to tackle that first (because it feels like we are loading it incorrectly to begin with)

[X547]

Some information about fw_dynamic: https://github.com/riscv-software-src/opensbi/blob/master/docs/firmware/fw_dynamic.md

https://github.com/riscv-software-src/opensbi/blob/d4d2582eef7aac442076f955e4024403f8ff3d96/include/sbi/fw_dynamic.h#L49

[LekKit]

I know about fw_dynamic. I want to figure out how exactly edk2 wants to be loaded/jumped into because it seems that RISCV_VIRT_CODE.fd has some custom layout as opposed to directly containing the S-mode payload

[LekKit]

Okay so apparently it jumps into 0x20000000 which is a base address of the CFI flash:

	flash@20000000 {
		bank-width = <0x04>;
		reg = <0x00 0x20000000 0x00 0x2000000 0x00 0x22000000 0x00 0x2000000>;
		compatible = "cfi-flash";
	}

And it is likely not relocatable which would explains everything

[LekKit]

I also would like to ask a few unrelated/barely related questions that I had for a while:

• Should RVVM have ACPI support at any point in future? I honestly don't like ACPI, but it's part of RVA server specification.
• I am prototyping RVVM KVM support in the meantime, but it won't allow me to run M-mode firmware inside it due to how RISC-V KVM is designed. I can probably trap-and-emulate M-mode features to make KVM guests function the same way as emulated ones, or I can simply let the S-mode run atop the pre-supplied host SBI (I don't believe this is safe nor versatile, but it's what QEMU does).
• I also wanted to be able to implement near-native virtualization on RISC-V boards without H-ext. It is feasible, but I need some help from the host kernel, namely being able to manually create/run/pause U-mode contexts, and manipulate their registers & manually map memory pages into them. Maybe this concept could be tested on Haiku?

[X547]

Should RVVM have ACPI support at any point in future?

For now I do not see cases where ACPI support is really required for RISC-V operating systems. I also do not know any real ACPI RISC-V hardware. But ACPI support may become needed if Microsoft decides to port Windows to RISC-V, Windows support ACPI only. For Haiku both ACPI and FDT support is planned. I managed to run my Haiku development branch on QEMU with ACPI after fixing minor QEMU bug (qemu/qemu@3018765).

Maybe this concept could be tested on Haiku?

I think it can if it do not need too invasive and global changes to Haiku kernel. First some detailed discussion is needed about necessary kernel features and kernel <-> VM communication protocol.

[LekKit]

I think it can if it do not need too invasive and global changes to Haiku kernel. First some detailed discussion is needed about necessary kernel features and kernel <-> VM communication protocol.

Basic idea is:

• RVVM sets up emulation as usual (JIT is not needed, thus only interpreter is used)
• For each vCPU, RVVM creates a U-mode context that is entirely for it's disposal. This context should not have access to Haiku syscalls nor have anything mapped into it's address space (even in the upper half).
• Upon running the vCPU, RVVM copies the U-mode CPU state (int/fp registers and fcsr) from rvvm_hart_t struct into it's respective U-mode "runner" context on the host
• On any pagefault, ecall or instruction exception the "runner" context stops and returns control to RVVM vCPU thread. RVVM copies back vCPU state into rvvm_hart_t and interprets the faulting instruction as usual.
• On TLB refill, RVVM actually shares a host page mapping of the guest physical memory into the "runner" context address space. TLB flushes flush the "runner" context address space mappings, partially/fully. Ideally ASID support should minimize the amount of misses to this shadow pagetable.
• Afterwards execution returns into this "runner" context and we are at step 3 again

This approach generally allows fully native execution, with the following downsides compared to full H-ext virtualization:

• Shadow pagetable is used which will have an observable perf penalty for memory access & context switches. However, this cost is much less than a soft TLB approach used right now, and can be further improved with proper ASID support.
• Any privileged instruction will be still interpreted

My initial idea of VM<->kernel interface (Pseudocode, this can really be implemented as a syscall or ioctl or whatever):

// Creates a shadow pagetable, destroy with close()
int vm_pgt_create();

// Shares memory from a usual process into the shadow pagetable
// Can be replaced with mmap-like interface that accepts a FD if that makes things simpler, but this approach seems more extensible
int vcpu_map_pages(int pgt_fd, size_t va, void* uptr, size_t size);

int vcpu_unmap_pages(int pgt_fd, size_t va, size_t size);

struct vcpu_runner_state {
    size_t pc;
    size_t gpr[31];
    double fpr[32];
    size_t fcsr;
};

// Accepts initial vCPU U-mode state & shadow pagetable handle for address space
// Runs the vCPU until any kind of exception, modifies vcpu_state post execution
// This design allows vCPU entry/exit & state manipulation in a single syscall
int vcpu_run(int pgt_fd, struct vcpu_runner_state* vcpu_state);

[LekKit]

Ah, almost forgot about interrupts.
This API is pretty similar to Linux KVM API but simplified a lot (And based on a notion that a lot of work is moved into userspace). In KVM they send a signal to a vCPU thread to interrupt vcpu_run() and force it to return immediately.
We could either adopt this approach, or add some kind of vCPU runner handle and a vcpu_kick(int vcpu_fd) kernel interface (Because signals are ugly IMO).

[LekKit]

About EDK2: Maybe it makes more sense to ask them for an S-mode payload without cfi-flash requirement?

If they won't agree then we can implement RVVM support anyways, but it probably makes more sense to have a more generic payload that may actually run on real hardware.

The biggest problem with EDK2 riscv support currently is the requirement to use fw_dynamic OpenSBI variant to jump into flash ROM for S-mode payload. I imagine no hardware currently has means to actually do that.

[LekKit]

Asked for a generic S-mode payload in tianocore/edk2#5815