Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

--enable-modernity results in instacrash for xen unikernels #70

Closed
yomimono opened this issue Jun 22, 2015 · 13 comments
Closed

--enable-modernity results in instacrash for xen unikernels #70

yomimono opened this issue Jun 22, 2015 · 13 comments

Comments

@yomimono
Copy link

I'm building unikernels with mirage-seal version 0.3.1. I have the following pins set:

$ opam pin list
nocrypto.0.4.0 path /home/dothraki/ocaml-nocrypto
tls.0.5.0 git https://github.com/mirleft/ocaml-tls#naming-updates
x509.0.3.1 git https://github.com/mirleft/ocaml-x509#naming

The local FS pin of nocrypto is master as of 61023ea . Running mirage-seal with this configuration results in a unikernel which crashes on boot (the results of xl dump-core for such a run are available if helpful):

$ sudo xl create seal.xl -c
Parsing config from seal.xl
Xen Minimal OS!
  start_info: 00000000003e6000(VA)
    nr_pages: 0x2000
  shared_inf: 0xd9678000(MA)
     pt_base: 00000000003e9000(VA)
nr_pt_frames: 0x7
    mfn_list: 00000000003d6000(VA)
   mod_start: 0x0(VA)
     mod_len: 0
       flags: 0x0
    cmd_line: 
       stack: 00000000003b5820-00000000003d5820
MM: Init
      _text: 0000000000000000(VA)
     _etext: 000000000020fe1f(VA)
   _erodata: 0000000000272000(VA)
     _edata: 000000000037bc20(VA)
stack start: 00000000003b5820(VA)
       _end: 00000000003d5820(VA)
  start_pfn: 3f3
    max_pfn: 2000
Mapping memory range 0x800000 - 0x2000000
setting 0000000000000000-0000000000272000 readonly
skipped 1000
MM: Initialise page allocator for 3ff000(3ff000)-2000000(2000000)
MM: done
Demand map pfns at 2001000-0000002002001000.
Initialising timer interface
Initialising console ... done.
gnttab_table mapped at 0000000002001000.
getenv(OCAMLRUNPARAM) -> null
getenv(CAMLRUNPARAM) -> null
getenv(PATH) -> null
Unsupported function lseek called in Mini-OS kernel
Unsupported function lseek called in Mini-OS kernel
Unsupported function lseek called in Mini-OS kernel
getenv(OCAMLRUNPARAM) -> null
getenv(CAMLRUNPARAM) -> null
getenv(TMPDIR) -> null
getenv(TEMP) -> null
GPF rip: 197eec, error_code=0
RIP: e030:[<0000000000197eec>] 
RSP: e02b:00000000003cfe38  EFLAGS: 00010046
RAX: 0000000000412b48 RBX: 0000000000000001 RCX: 000000000000000e
RDX: 0000000000fbcab0 RSI: 0000000000412b08 RDI: 0000000000fbcae8
RBP: 000000000032b918 R08: 0000000000000008 R09: 0000000000000010
R10: 0000000000000001 R11: 000000000037f500 R12: 0000000000141268
R13: 000000000035abf8 R14: 00000000003cfee0 R15: 000000000090d470
base is 0x32b918 caller is 0x7
base is 0x6e00 caller is 0x86e6058d4830ef83
base is 0x49c7894808ec8348 GPF rip: 20f797, error_code=0
RIP: e030:[<000000000020f797>] 
RSP: e02b:00000000003cfd58  EFLAGS: 00010016
RAX: 000000000000001b RBX: 49c7894808ec8348 RCX: 0000000000000713
RDX: 0000000000000713 RSI: 00000000003cfba8 RDI: 0000000000000004
RBP: 00000000003cfd88 R08: 0000000000000712 R09: 0000000000000020
R10: 000000000000001b R11: 0000000000000000 R12: 0000000000141268
R13: 000000000035abf8 R14: 00000000003cfee0 R15: 000000000090d470
base is 0x3cfd88 caller is 0x3cfee0
base is 0x90d470 caller is 0x1e1
base is 0x400 Page fault at linear address 408, rip 20f797, regs 00000000003cfbc8, sp 3cfc78, our_sp 00000000003cfb88, code 0
RIP: e030:[<000000000020f797>] 
RSP: e02b:00000000003cfc78  EFLAGS: 00010012
RAX: 000000000000000e RBX: 0000000000000400 RCX: 0000000000000925
RDX: 0000000000000925 RSI: 00000000003cfad8 RDI: 0000000000000004
RBP: 00000000003cfca8 R08: 0000000000000124 R09: 0000000000000020
R10: 000000000000000e R11: 0000000000000000 R12: 0000000000141268
R13: 000000000035abf8 R14: 00000000003cfee0 R15: 000000000090d470
base is 0x3cfca8 caller is 0x3cfee0
base is 0x90d470 caller is 0x1e1
base is 0x400 Page fault in pagetable walk (access to invalid memory?).

Rebuilding against a nocrypto built with with --disable-modernity results in a normally booting unikernel.

Please let me know if there's more information I can provide, or any other way I can help.

@yomimono
Copy link
Author

This is on x86_64; haven't yet tested ARM but will do so if helpful.

@pqwy
Copy link
Contributor

pqwy commented Jun 23, 2015

Ouch, ouch and ouch.

This means it is crashing in the new SSE/AES-NI code that --enable-modernity activates.

What's the behavior if you use the same flags on the same machine but with Unix underneath? Alternatively, what flags do you see in /proc/cpuinfo? What's the CPU model?

Explanation 1: Your CPU does not know AES-NI. That would be OK, I could rename the flag to something less presumptuous, and maybe disable it by default. Dynamic detection is yet to be done.

Explanation 2: SSE registers are unavailable on Xen/MiniOS. This is the stuff of nightmares.

Re: ARM, modernity currently does nothing for ARM.

@yomimono
Copy link
Author

No problem running on Unix; process runs and serves files over HTTPS just fine.

Here's the relevant-looking stuff from /proc/cpuinfo:

vendor_id   : GenuineIntel
cpu family  : 6
model       : 58
model name  : Intel(R) Core(TM) i7-3630QM CPU @ 2.40GHz
flags       : fpu de tsc msr pae mce cx8 apic sep mca cmov pat clflush acpi mmx fxsr sse sse2 ss ht syscall nx lm constant_tsc rep_good nopl nonstop_tsc eagerfpu pni pclmulqdq monitor est ssse3 cx16 sse4_1 sse4_2 popcnt tsc_deadline_timer aes xsave avx f16c rdrand hypervisor lahf_lm ida arat epb xsaveopt pln pts dtherm fsgsbase erms

@hannesm
Copy link
Member

hannesm commented Jun 23, 2015

Is it xen (does AES-NI work in a Linux domU?)? or is it minios/mirage running in supervisor mode?

@avsm
Copy link

avsm commented Jun 23, 2015

SSE is available but stack alignment must be preserved. This should just work though, because builtins like memcpy use SSE instructions from gcc these days and so crashes tend to happen early.

@yomimono
Copy link
Author

@hannesm When I say "no problem running on Unix" I mean in dom0; I haven't got a *nix guest set up but re-reading @pqwy's comment that's probably what was asked for? Can do that this afternoon.

@hannesm
Copy link
Member

hannesm commented Jun 23, 2015

@yomimono both are of interest, and I understood you tested in dom0. domU should work (from @avsm comment)

@pqwy
Copy link
Contributor

pqwy commented Jun 23, 2015

@yomimono Thank you! I just wanted to check if it's the CPU feature set. It is not. The next thing that would be helpful is to add printfs to the beginning of these two functions, because I think the first brush with this code is in the early RNG seeding. Another helpful printf-case is confirming that the start function of the unikernel is never even reached.

In more detail, for the purposes of my question, it makes no difference whatsoever whether you ran the Unix test without Xen in sight, or in dom0, or domU. It especially does not matter if it didn't explode, because we can immediately rule out the CPU limitation on your own computer.

Xen itself obviously "supports" SSE in that these are not privileged instructions, are not trapped in HVM and are not hindered by having no access to ring0 in PVM. There is no difference between dom0 and domU in this respect, as both are actually virtualized. dom0 might be allowed access to memory regions peripherals are mapped to, but runs under the same regime of CPU virtualization. And as @avsm alluded to, actual lack of support for SSE under Xen would make Linux bomb out immediately.

SSE requires OS cooperation at least in order to preserve the extended register set on context-switch. I suspect, but at this point without knowing for a fact, that the OS also needs to enable them, and that MiniOS never does it. I also think there is no other SSE code running in MiniOS as it doesn't use the standard libc, so this is the first time MiniOS was prodded into running SSE.

@avsm It's specifically using SSE2 unaligned-access MOVDQU via _mm_loadu_si128 to move data from memory into the SSE registers. Do you think alignment matters anyway?

Off to untar my Xen-debugging rig...

@yomimono
Copy link
Author

@pqwy - thanks for the explanation. printfs in the suggested places don't produce any output before the VM crashes.

Happy untaring...

@pqwy
Copy link
Contributor

pqwy commented Jun 23, 2015

Ok, I have a repro. But no clue so far.

@pqwy
Copy link
Contributor

pqwy commented Jun 24, 2015

It's alignment! Spot on @avsm ! 🎊

Most traffic between memory and SSE registers goes via unaligned load/store, except when it doesn't -- AES round keys are just referenced directly, by mistake. I managed to remain completely oblivious to this because on Unix the allocator returns 16-byte aligned bigarrays. Not on Xen.

mirage/ocaml-cstruct#38 would probably help for a nice, long-term solution.

pqwy added a commit that referenced this issue Jun 24, 2015
See #70.

RKs are not explicitly loaded and stored so the pointers need to be aligned.

A better solution than allocating a little more and aligning on access would be
to make a utility to allocate them aligned in the first place. The best solution
would be to have a reusable one.
@pqwy
Copy link
Contributor

pqwy commented Jun 24, 2015

@yomimono Does it work for you with fb96934?

@yomimono
Copy link
Author

@pqwy - Yes, it boots and serves nicely in Xen with that patch. Hooray!

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Development

No branches or pull requests

4 participants