govc vm.ls kills VIC appliance VM

[gigawhitlocks]

So I've hit a really bizarre issue in govc/govmomi.
Steps to reproduce:

Install a VIC VCH on ESXi with vic-machine in the standard way, then, after updating your environment to point docker at your VCH:

docker create --name foo busybox
cat some-archive.tar.gz | docker cp - foo:/
govc vm.ls $VCH_NAME /tmp

govc will hang. The VCH will stop responding. govc vm.ip will hang. govc vm.info will show the endpoint VM as powered on, with no IP address. Prior to calling govc vm.ls, everything works normally in the VCH.

I was also able to procure a stack trace from the toolbox code in vic-init:

Nov 22 2017 19:16:27.360Z DEBUG op=189.1: logrotate finished successfully
Nov 22 2017 19:16:27.363Z DEBUG Inspecting children with status change
Nov 22 2017 19:16:27.366Z DEBUG No more child processes to reap
panic: runtime error: slice bounds out of range

goroutine 10 [running]:
github.com/vmware/vic/vendor/github.com/vmware/govmomi/toolbox.(*CommandServer).ListFiles(0xc420258cc0, 0x5d00d0001, 0x3300000062, 0xd00000022, 0xb10000000d, 0xffffffff00000000, 0xb1, 0x100000000, 0xc4203a564c, 0x2f, ...)
    /go/src/github.com/vmware/vic/vendor/github.com/vmware/govmomi/toolbox/command.go:515 +0x442
github.com/vmware/vic/vendor/github.com/vmware/govmomi/toolbox.(*CommandServer).ListFiles-fm(0x5d00d0001, 0x3300000062, 0xd00000022, 0xb10000000d, 0xffffffff00000000, 0xb1, 0x100000000, 0xc4203a564c, 0x2f, 0x2f, ...)
    /go/src/github.com/vmware/vic/vendor/github.com/vmware/govmomi/toolbox/command.go:72 +0x90
github.com/vmware/vic/vendor/github.com/vmware/govmomi/toolbox.(*CommandServer).Dispatch(0xc420258cc0, 0xc4203a5607, 0x74, 0x74, 0x8b, 0x3, 0xc4203a55f0, 0xc420154dd0, 0x8b)
    /go/src/github.com/vmware/vic/vendor/github.com/vmware/govmomi/toolbox/command.go:166 +0x440
github.com/vmware/vic/vendor/github.com/vmware/govmomi/toolbox.(*CommandServer).Dispatch-fm(0xc4203a5606, 0x75, 0x75, 0x15, 0xc420154e70, 0xc4203a5501, 0x15, 0x8b)
    /go/src/github.com/vmware/vic/vendor/github.com/vmware/govmomi/toolbox/command.go:81 +0x48
github.com/vmware/vic/vendor/github.com/vmware/govmomi/toolbox.(*Service).Dispatch(0xc4201c3340, 0xc4203a55f0, 0x8b, 0x8b, 0x0, 0x0, 0xc0)
    /go/src/github.com/vmware/vic/vendor/github.com/vmware/govmomi/toolbox/service.go:236 +0x177
github.com/vmware/vic/vendor/github.com/vmware/govmomi/toolbox.(*Service).Start.func1(0xc4201c3340, 0xc42025b840)
    /go/src/github.com/vmware/vic/vendor/github.com/vmware/govmomi/toolbox/service.go:185 +0x2ce
created by github.com/vmware/vic/vendor/github.com/vmware/govmomi/toolbox.(*Service).Start
    /go/src/github.com/vmware/vic/vendor/github.com/vmware/govmomi/toolbox/service.go:193 +0xd8

[dougm]

The code assumes offset is still valid, which it shouldn't (will fix). With the tether/toolbox panic, none of the guest operations/queries will work until tether has restarted.

[gigawhitlocks]

Do you know what causes the offset to become invalid @dougm ? I'm triggering this error case in the PR I'm currently working on and I'm trying to understand what's causing it. (PR for reference is vmware/vic#6808 )

[dougm]

I think if files have been removed in between requests. I'm going to write a test to reproduce before fixing.

[gigawhitlocks]

Ping me on Slack if you need more info. You may have to build off of my branch to reproduce.