Activate VM-Exit I/O port 0x5658 crash vmtoolsd.exe

[foxever]

Description

I test I/O interception for vmware backdoor instruction in 0x5658, however when I activate VM-Exit I/O port 0x5658 crash vmtoolsd.exe on VMWare workstation 12

Expected behavior

no crash and log instruction IN 0x5658

Actual behavior

always crash vmtools when load driver

Steps to reproduce the problem

modify code in VmpBuildIoBitmaps, add one line RtlSetBits(&bitmap_a_header, 0x5658, 0x1);
build and load HyperPlatform dirver
crash...

Specifications

• OS version:
  host Windows 7 x64
  guest Windows 7 x64
• Hardware:
  VMware 12
• Anything else:

some log found vmware-vmsvc.log

[Sep 03 15:23:11.720] [ message] [vmsvc] backtrace[00] frame 0x01a7f3f8 IP 0x01a7f440 params 0x4f4c4354 0x80000000 0x7fef8fdaf42 0x338 [no module data] ???
[Sep 03 15:23:11.720] [ message] [vmsvc] backtrace[01] frame 0x01a7f400 IP 0x0096c2f0 params 0x80000000 0x7fef8fdaf42 0x338 0 [no module data] ???
[Sep 03 15:23:11.720] [ message] [vmsvc] backtrace[02] frame 0x01a7f408 IP 0x4f4c4354 params 0x7fef8fdaf42 0x338 0 0x1a7f628 [no module data] ???
[Sep 03 15:23:11.720] [ message] [vmsvc] backtrace[03] frame 0x01a7f410 IP 0x80000000 params 0x338 0 0x1a7f628 0x1 [no module data] ???
[Sep 03 15:23:11.736] [ message] [vmsvc] backtrace[04] frame 0x01a7f418 IP 0x7fef8fdaf42 params 0 0x1a7f628 0x1 0x564d5868 [C:\Program Files\VMware\VMware Tools\vmtools.dll base 0x000007fef8f80000 0x0001:0x0000000000059f42] GuestApp_GetConfPath

when decompile vmtools.dll crash in address 0x7fef8fdaf42

.text:000007FEF8FDAF38                 mov     [r11-38h], r14w
.text:000007FEF8FDAF3D                 call    Backdoor
.text:000007FEF8FDAF42                 test    byte ptr [rsp+68h+var_36], 1

.text:000007FEF8FE1A00 Backdoor        proc near               ; CODE XREF: sub_7FEF8FDAEF0+4D�p
.text:000007FEF8FE1A00                                         ; sub_7FEF8FDAEF0+7D�p ...
.text:000007FEF8FE1A00                 mov     eax, 5658h
.text:000007FEF8FE1A05                 mov     dword ptr [rcx], 564D5868h
.text:000007FEF8FE1A0B                 mov     [rcx+18h], ax
.text:000007FEF8FE1A0F                 jmp     sub_7FEF8FE1A60
.text:000007FEF8FE1A0F Backdoor        endp

.text:000007FEF8FE1A60 sub_7FEF8FE1A60 proc near               ; CODE XREF: Backdoor+F�j
.text:000007FEF8FE1A60                                         ; DATA XREF: .pdata:000007FEF903F868�o
.text:000007FEF8FE1A60
.text:000007FEF8FE1A60 var_20          = qword ptr -20h
.text:000007FEF8FE1A60
.text:000007FEF8FE1A60                 push    rbx
.text:000007FEF8FE1A62                 push    rsi
.text:000007FEF8FE1A63                 push    rdi
.text:000007FEF8FE1A64                 mov     rax, rcx
.text:000007FEF8FE1A67                 push    rax
.text:000007FEF8FE1A68                 mov     rdi, [rax+28h]
.text:000007FEF8FE1A6C                 mov     rsi, [rax+20h]
.text:000007FEF8FE1A70                 mov     rdx, [rax+18h]
.text:000007FEF8FE1A74                 mov     rcx, [rax+10h]
.text:000007FEF8FE1A78                 mov     rbx, [rax+8]
.text:000007FEF8FE1A7C                 mov     rax, [rax]
.text:000007FEF8FE1A7F                 in      eax, dx
.text:000007FEF8FE1A80                 xchg    rax, [rsp+20h+var_20]
.text:000007FEF8FE1A84                 mov     [rax+28h], rdi
.text:000007FEF8FE1A88                 mov     [rax+20h], rsi
.text:000007FEF8FE1A8C                 mov     [rax+18h], rdx
.text:000007FEF8FE1A90                 mov     [rax+10h], rcx
.text:000007FEF8FE1A94                 mov     [rax+8], rbx
.text:000007FEF8FE1A98                 pop     qword ptr [rax]
.text:000007FEF8FE1A9A                 pop     rdi
.text:000007FEF8FE1A9B                 pop     rsi
.text:000007FEF8FE1A9C                 pop     rbx
.text:000007FEF8FE1A9D                 retn
.text:000007FEF8FE1A9D sub_7FEF8FE1A60 endp

[tandasat]

Impact of this issue seems relatively low, so it may take some time until I find time to take a look at it, but still good to be aware of. Thanks for detailed report!

[wbenny]

The issue here is that VMWare's backdoor (operating at port 0x5658, both via in/out instructions) has somewhat its own "calling convention". in/out instructions usually operate with rdi/rsi/rax/rdx registers only - but VMWare's backdoor expects parameters in RAX, RBX, RCX, RDX and uses the same set of registers for the output. Unfortunatelly, the only option here is either:

• write custom assembly function, which takes RAX, RBX, RCX, RDX from the guest save area, calls in or out, and then write those registers back to the save area
• ignore 0x5658 port completely

I'm making this dead thread alive because I'm dealing with the exact same problem right now, and at the same time I want to give a hint to others who happen to stuck here :)

[wbenny]

I've researched this bit more and... boy, does it behave weirdly!

First of all, setting up custom I/O VM-exit instruction handler doesn't solve it. Why? Because VMWare Tools execute I/O instructions in user mode. And what does Intel Manual say about exception priorities? Thats right, #GP due to I/O executed in CPL > 0 has priority over VM-exit, even if unconditional I/O exiting is enabled.

So in the moment when you start intercepting 0x5658 and 0x5659 I/O ports, you won't actually catch them in your handler at all, because #GPs will be suddenly raised (no matter if you catching GP via exception bitmap or not).

BUT when you write a custom handler for #GP and check if the guest RIP points to any I/O instruction (which can be done via really simple function - no complex disassembler is needed), you can emulate the I/O instruction yourself (note that this I/O must be emulated with [ RAX, RBX, RCX, RDX, RSI, RDI, RFLAGS.DF ] set according to the guest - __in*/__out* intrinsics won't help you here, you have to write custom ASM function for this). Also, you must not forget to write back those mentioned registers back to the guest state (except the RFLAGS.DF), because VMWare uses them for both input and output.
After that, you MUST skip the current instruction AND NOT reinject the #GP. This is obvious - you want to continue execution after the in/out instruction like nothing unusual happened. I'm just typing this just to be super-clear.

When you do this, VMWare Tools continue to work.