proposal: support GOOS=none GOARCH=amd64 for running Go programs without an OS

[eliasnaur]

The proposal itself is simple: allow

GOOS=none GOARCH=amd64 go build ...

to output a virtual machine image capable of booting and running
directly on a hypervisor such as KVM.

As a motivating example, consider the single command

GOOS=none GOARCH=amd64 go build golang.org/x/tools/cmd/godoc

which will produce a virtual machine image that implements godoc.org.

The benefits are the usual isolation and security benefits of virtual
machines, with the added simplicity and performance benefits of no
underlying operating system.

That the underlying operating system gets in the way of the Go runtime
is not a theroretical concern; see #35777 for a recent example.

Implementation

Without an operating system, the Go runtime must implement functionality normally
provided by the OS, the most important being CPU scheduling and memory
management.

Note that the Go runtime already implements scheduling of
goroutines onto operating system threads. In addition, the runtime
also manages and garbage collects operating system provided memory.

I claim that the runtime need relatively few changes to schedule
goroutines onto virtual CPUs and few changes to not allocate memory
from the OS but directly from the available virtual memory.

Futher, any interesting Go program needs I/O, most often network and
file I/O. Fortunately, the virtio specification standardizes virtual machine access
to networking, console and file systems devices. I expect the standard
library packages net and os to be implemented as Go drivers for virtio devices.

If the proposal is accepted and no-one more experience than myself claims it, I plan to seek sponsorships for implementing it for Go 1.15 along with the builder for build.golang.org.

[rsc]

This doesn't sound like "GOOS=none". It sounds like "GOOS=kvm", especially if you depend on virtio and other virtual devices only provided by some VMMs.

[eliasnaur]

The GOOS=none name seems right to me because there's no reason a GOOS=none binary couldn't boot and run on bare metal just as a linux kernel bzImage can run on bare metal as well as on a hypervisor. There is certainly no dependency on a particular hypervisor such as kvm.

The virtio devices are not a hard dependency: a program that doesn't use the net or os packages can run without them. In fact, that's what I plan to do if this proposal i accepted and implemented: implement a bare-metal hypervisor where hardware drivers are implemented as virtual machine(s) on top.

I could even imagine a mechanism for registering network and file system drivers with the Go runtime to make package net and os useful even on real hardware.

Finally, virtio devices are not necessarily virtual. In fact, one of the touted features of the virtio 1.1 specification is easier implementation in hardware.

My proposal focuses on a virtual machine environment because it is easier to target initially and also seems more generally useful.

[bcmills]

Even if kvm is not the right GOOS, what you're describing still seems to imply a GOOS other than none, in the sense that the GOOS determines the system call ABI for the runtime and standard library. Perhaps GOOS=virtio would be appropriate?

(A GOOS=none in the precise sense seems like it would require either memory-mapped I/O or some sort of pluggable driver layer, but a “pluggable driver layer” is, in some sense, a GOOS.)

[eliasnaur]

Even if kvm is not the right GOOS, what you're describing still seems to imply a GOOS other than none, in the sense that the GOOS determines the system call ABI for the runtime and standard library. Perhaps GOOS=virtio would be appropriate?

(A GOOS=none in the precise sense seems like it would require either memory-mapped I/O or some sort of pluggable driver layer, but a “pluggable driver layer” is, in some sense, a GOOS.)

I think GOOS=metal or "noos" as suggested on golang-dev is better than "virtio". There is nothing special about virtio devices other than being common in virtual environments and particularly easy to write drivers for. Virtio devices are still discovered and used from the kernel through the same mechanisms as actual hardware devices.

I suppose an interesting question is: does GOOS=virtio still seem appropriate for running Go on bare metal where I wouldn't use virtio?

Perhaps GOOS is the wrong selector and there should be a GODRIVER=... for everything else than what your strict interpretation of GOOS=none implies.

[rsc]

GOOS=kernel?

[rsc]

Bikeshedding is fun but also distracting from the bigger question: should we do this at all?

We used to have a GOOS=tiny for bare hardware, which we removed (see 434f1e3). Having a port in the main tree requires a lot of effort. You may be willing to do the initial port, but the runtime/compiler team will ultimately be responsible for debugging problems, fixing bugs, and so on. That seems like a significant burden to take on (and is part of why we deleted tiny).

Also, Go depends pretty heavily on the OS for things like a file system, network access, TCP/IP stack, time, and so on. Tiny had none of those. Even if we were to assume Go sitting on top of KVM, nearly all of those things would still be missing and would have to be provided by Go itself. That's a huge amount of new code that would have to be added to the runtime and maintained.

I'm very skeptical about GOOS=none GOARCH=amd64 go build golang.org/x/tools/cmd/godoc implementing godoc.org. That implies a TCP/IP stack, DHCP, a DNS config from somewhere, a TLS root set from somewhere, maybe NTP, a file system to hold the Go sources whose documentation is being served, probably more. I'm pretty confident that Go should not be in the business of having its own version of these things. Making that command-line work would be an enormous undertaking with significant ongoing maintenance costs.

Maybe I'm wrong, but I just don't see the benefits here outweighing the costs.

[eliasnaur]

I like the color of GOOS=kernel.

I agree that adding the necessary code for I/O to the Go standard library is too much. However, I'd still like for Go to enable programs running on bare metal or virtual machines. There are similar efforts for "not so small" microcontrollers, announced here:

https://groups.google.com/forum/#!topic/golang-dev/pAgZ64nnpJ4

Another related project is TinyGo, although they're also trying to avoid the large binary sizes of Go programs.

So a follow-up to Russ' objections is the question: would a GOOS=kernel port be interesting even if it didn't support package os, net and dependent packages? In other words, GOOS=kernel would implement enough of the runtime to support the Go language (goroutines, timers, memory allocation), but not the Go standard library packages for network and file system access.

[eliasnaur]

Another related project:

https://github.com/inversepath/tamago

"bare metal Go for ARM SoCs". I believe the project is motivated by running Go on the USB [Armory device}(https://github.com/inversepath/usbarmory/wiki).

And of course the now dormant gopher-os implementating of a kernel in Go:

https://github.com/gopher-os/gopher-os

The point is that there is enough interest in running Go on bare-metal or without an OS that the significant work of porting the Go runtime and toolchain has been done several times. I would like to combine those efforts into an upstream GOOS target so the work won't need to be repeated over and over.

FWIW, Rust runs on bare metal, which lead to https://www.redox-os.org/.

[rsc]

would a GOOS=kernel port be interesting even if it didn't support package os, net and dependent packages?

Personally, I don't think so, no. That's very far afield from Go's focus right now. If we put it in the main repo it will not get the attention and time it really needs to be done well.

I think it is fine for that to live in a separate place than the main Go repo, just like TinyGo does. If you are concerned about duplication of effort, the thing to do would be to make sure there's just one separate place instead of many.

If I were trying to make a "standalone" godoc binary then I think I would write a tiny kernel entirely separate from Go that supported the necessary system calls to run a Linux ELF binary (a bit like gVisor does) and just build a linux/amd64 binary and then cat kernel gobinary >image and run that. That makes a lot more sense to me than making the kernel piece be part of the Go runtime, and it could support non-Go programs too, potentially.

[rsc]

For the reasons already outlined on the thread, it seems pretty clear that this is far too much work for the main team to take on in the core repo. We had such a hard time tracking down the Linux 5.3 signal handler bug, the last thing I want is to be tracking down chip errata too. Let's leave turning Go programs into kernels for a different project.

It looks like these projects already exist. Here is one: https://nanovms.com/dev/tutorials/running-go-unikernels.

This seems like a likely decline.

Leaving open for a week for final comments.

[mjl-]

fyi: https://github.com/mjl-/vmgo/tree/solo5test adds GOOS=solo5hvt (work in progress). it compiles to a binary/image that can be run by the solo5 hypervisor (which aims to be small and secure).

it uses netstack (pure go tcp/ip stack) from the gvisor project for networking. also see the "netstack" branch in that repo, which can be used to compile regular linux/amd64 binaries that use netstack instead of the kernel's.

i figured an approach like this would not (soon) be included in go. it probably wouldn't be used enough to justify the complexity it adds to the runtime. i'm doing it to learn. posting it as an example of what the original poster might have in mind.

[rsc]

Thanks for the link @mjl-.

No change in consensus here, so declining.

[eliasnaur]

FWIW, I've started to implement Russ' proposal (implementing a wrapper around the Go program with a minimal kernel). You don't get the performance and flexibility from a tight integration, but maintain the important ability to run unmodified Go programs. I have a prototype that can run simple Go programs that I intend to open source when it's sufficiently useful; let me know if anyone is interested in collaborating.

[abarisani]

Hello all, I started a similar discussion here, the thread directed me here as well.

I think there is great interest for bare metal support as long as it can cleanly maintained.

We targeted arm and SoCs as we think this is an attainable goal, in fact our example application happily emulates Ethernet over USB, has a TCP/IP stack and can serve SSH, HTTPS and so on in pure Go.

Please check out the links in the thread, I just thought to comment here to show that freestanding Go can actually do meaningful I/O and implement drivers without actually polluting the compiler itself (if you check the separation we have in our architecture).

[amscanne]

I just found this bug, and wanted to add a thought. I've played with a few versions of this over the last couple of years. It makes sense that anything in-tree would probably rot quickly.

What I always wanted upstream is a mechanism that lets you build for a given OS, but "hijack" critical runtime bits and allow you to provide your own implementation. This would, of course, totally void all warranties. No assumptions about these private APIs, exactly like go:linkname. This is the approach taken by the Gopher-OS project [1], where they even adopt the standard looking "//go:redirect-from" function tag, though it does all the rewriting after the fact. So this could totally support a unikernel/bare metal system via a simple 'import "turn-into-an-os"' and some build flags, assuming your target is loading ELF binaries.

Of course, random imported packages could abuse this. But they can already abuse unsafe and raw system calls arbitrarily, so I'm not sure it actually makes anything more dangerous.

[1] https://github.com/gopher-os/gopher-os/blob/1a7aca464ed24473d1a01d1b6a7c42ebabcfc6d0/src/gopheros/kernel/goruntime/bootstrap.go#L40
[2] https://github.com/gopher-os/gopher-os/blob/master/tools/redirects/redirects.go

[eliasnaur]

@amscanne, this is either a weird coincidence or you found this bug through my unikernel post, here:

https://groups.google.com/forum/#!topic/golang-nuts/4cDIL5Vr_es

Unik implements a very basic kernel that mimics the Linux kernel system calls, so that a GOOS=linux GOARCH=amd64 static Go binary can run in virtual machines (and in theory bare-metal). The project requires no modification to Go itself.

[amscanne]

Yes, I saw the post. The approach makes sense. When I did sth similar in the past, I thought it would be nice to avoid boilerplate and optimize some paths by avoiding the trap and pipeline flush. (E.g. you could just remap futexwakeup and futexsleep directly.)

//go:redirect-from would have other uses too, such as the netstack example (with an unmodified toolchain. Or low-level dependency injection for tests. Maybe worth a separate proposal someday.