proposal: refactor platform support for easier ports.

[beoran]

Proposal

I propose to refactor the support for platforms, architectures and operation systems in the Go run time and compiler so it becomes easier to write a new port, also for third parties, and simpler to maintain current ports.

Currently, support for platforms is spread out all over the code base of Go, using build tags or target specific go files. While this is understandable for historic reasons, it does make it hard to develop a port of Go to a new platform. The platform support should be better factored so this becomes easier.

This could be done in several ways, but my first idea, which is open for debate, would be to create a package platform with sub packages for architecture, OS, and arhcitecture-os combinations if needed, and then have sub-sub packages for the compiler and for the runtime. Another idea would be for such architecture support to be split of in Go modules.

This issue was first mentioned in #37503.

Proposal template

• Would you consider yourself a novice, intermediate, or experienced Go programmer?
  I am experienced, I have been using Go for 10 years since before v1.0.
• What other languages do you have experience with?
  Batch files, Basic, C, C++, D, Java, Javascript, Lua, Objective-C, Oberon, Pascal, PHP, Python, R, Ruby, Shell scripts, Visual Basic, ...
• Would this change make Go easier or harder to learn, and why?
  It would make it easier how to learn how to make a port of Go to a new OS or architecture and to maintain existing OS and architectures.
• Has this idea, or one like it, been proposed before?
  This idea is based on the proposal proposal: all: add bare metal ARM support #37503.
• If so, how does this proposal differ?
  In stead of porting to a specific platform, we see in that issue that too many files have to be changed to make a port. This issue is specifically about ameliorating that situation.
• Who does this proposal help, and why?
  People who want to port Go to a new platform, and people who have to support an existing Go platform.
• Is this change backward compatible?
  Yes, if refactored correctly, this should change nothing at all in how Go works.
• Show example code before and after the change.
  This is still to be debated how we will change the code.
• What is the cost of this proposal? (Every language change has a cost).
  • We have to perform a thorough refactoring, taking care not to cause any regressions.
  • We have to beware not causing any performance regressionsas well.
  • How many tools (such as vet, gopls, gofmt, goimports, etc.) would be affected?
    All tools that use platform-specific code will be affected. Those who use the run time API will not be affected.
  • What is the compile time cost?
    None, if done correctly.
• What is the run time cost?
  None, if done correctly.
• Can you describe a possible implementation?
  Groso modo, either use packages, or use modules for each platform and concentrate the platform -dependent code in them.
• How would the language spec change?
  Not applicable.
• Orthogonality: how does this change interact or overlap with existing features?
  No effect.
• Is the goal of this change a performance improvement?
  No.
• Does this affect error handling?
  No .
• Is this about generics?
  No.

[josharian]

The last time I worked on a port (risc-v), finding all the arch-specific code was far from the most difficult part. And this reorganization would take a lot of effort, cause a lot of code churn, and maybe cause import graph problems. (Just splitting up ssa rewrite rules per architecture, which I still want to do for other reasons, is pretty non-trivial, even using shortcuts like dot imports.)

One clear win would be reducing the amount of arch-specific code, or (in some cases) writing semi-shared code generators. We’ve done some of that, but there might be other opportunities outstanding. I don’t think that requires a proposal, though; certainly not a generic one (as opposed to one discussing a particular change to make).

[bcmills]

There are currently a lot of duplicate copies of, for example, the shims that go between the Go calling convention and the C calling convention, neither of which varies all that much across most platforms. (See #18352.)

[beoran]

@josharian perhaps it is true that it it isn't hard to find where the platform specific code is, but when we look at the diff in the original issue, it touches files all over the place. It seems to me that the platform support in Go could be better structured.

I can agree that in stead of a single large refactoring, this could be done gradually. This issue can then be used to plan and track these gradual changes.

[networkimprov]

How about a "configure" step which produces temporary files for a build? That should work for at least the trivial changes. If the build shouldn't rely on a shell script, you could go run configure, before building anything else.

A list of files for the tamago patch is in #37503 (comment)

@beoran I don't think the issue needs the proposal template since it isn't a language change.

[beoran]

We already have go generate for generating files, we could use that to generate more boilerplate and use it better.

I will remove the proposal template once @ianlancetaylor confirms it is not necessary.

Edit: When looking at the diffstat it is strange that files for architectures other than ARM have to be changed to add a new ARM port. That suggests code that is too tightly coupled.

[abarisani]

For starters if posix and unix OSes would have their own build tags a lot of:

# unix example
-// +build aix darwin dragonfly freebsd js,wasm linux netbsd openbsd solaris
+// +build aix darwin dragonfly freebsd js,wasm linux netbsd openbsd solaris tamago

or

# posix example
-// +build aix darwin dragonfly freebsd js,wasm linux netbsd openbsd solaris windows
+// +build aix darwin dragonfly freebsd js,wasm linux netbsd openbsd solaris tamago windows

Would have probably been spared.

Maybe an initial improvement could be having tags that represent familes to spare a large number of such changes?

[josharian]

Edit: When looking at the diffstat it is strange that files for architectures other than ARM have to be changed to add a new ARM port. That suggests code that is too tightly coupled.

One programmer's "too tightly coupled" is another programmer's "avoids unnecessary abstraction".

I'm not saying we have anywhere near the perfect balance now. But I think that striving for complete abstraction, like "adding a new port doesn't touch anything outside a particular directory" is both unrealistic and likely to add complexity rather than remove it.

But again, overall, I'm generally in favor of clean-up, organization, and de-duplication, evaluated on a case by case basis.

[magical]

Previous discussion about adding a unix build tag: #20322.

[ianlancetaylor]

I'm not sure it helps much to move all the platform-specific code out of cmd/compile/internal and cmd/link/internal and into a set of shared internal directories. We've already done the architecture split there; moving it elsewhere doesn't make things much easier.

I agree with @josharian that this is something to be judged on a case-by-case basis.

[ianlancetaylor]

I do think that we should revive #20322, currently on proposal-hold.

[networkimprov]

And how about using go generate for trivial changes, like +build lines?

[ianlancetaylor]

I don't quite see how we can use go generate for +build lines. Unless go generate will be used to modify files in place? I guess that could work.

But I'm not sure why using go generate is clearly better than something like #20322. Either way you need a mapping of build tags, and for the standard library we presumably want a standard mapping. It moves the build tag mapping into a separate tool, but in this case I don't know if the cost/benefit tradeoff pans out. Maybe I'm missing something.

[rsc]

This could be done in several ways, but my first idea, which is open for debate, would be to create a package platform with sub packages for architecture, OS, and arhcitecture-os combinations if needed, and then have sub-sub packages for the compiler and for the runtime. Another idea would be for such architecture support to be split of in Go modules.

I don't really understand what you are suggesting. If you mean to move all the current OS-specific code out of places like runtime, time, os, etc into a new single "platform" package, I'm not sure that's particularly helpful. The current packages have much narrower scopes and are easier to design than the "platform" package would be. Having to be able to encode every possible variation into "platform" will be unwieldy.

To take just one example, path/filepath has a Separator constant. It's not clear that platform should provide that instead. Then which will people import?

[rsc]

@beoran, suppose we adopt #20322's suggestion of adding a unix build tag. How much would that help tamago's diffs, assuming tamago decided to be "unix"?

[beoran]

Basically, what I am suggesting is that we think of different ways how we could make supporting different platforms easier together.

Having "platform" packages could be one way to do that, yes, but sine you know the Go source code better than I do, I am willing to accept that it might not be a viable approach.

However, as you said in the original issue:

But how it stays up-to-date is that developers stop checking in changes that break the port, because of builders and presubmit checks and the like. When there are bugs, it is inevitable that Go developers outside the one tamago maintainer have to look at the logs and see whether the fault is theirs or not.

To me, this suggests that the Go developers often have problems where one change for one platform can easily cause problems for other platforms. This then suggests to me that the problem might be that the code is too tightly coupled. This issue is about thinking of ways, together to improve this issue, so even for existing ports, it becomes easier to maintain them.

So perhaps having full platform packages is not the way to go, certainly not for directory separators and the like, but perhaps it could be possible to factor platform-specific code better, such as partial per platform packages, or using sub-packages per platform for the current standard library packages.

And yes, #20322 is likely to help this issue if tamago decided to be unix. Although, perhaps, an even better idea would be to rethink the way build tags are used currently for platform support. Build tags allow us to compile platform-specific code, but I always felt they are a bit of an unclean solution.

Another approach could be to allow GOOS to be set to "none", and then, allow this as a build tag and recognise this in file names, but not generate any different code, and cause all existing OS specific code that does not match "none" to be not compiled. The result will not link, perhaps not even compile, but it will be up to the people maintaining the port to make this work using patches. Likewise, unsupported but reasonable values of GOARCH, say, names of common architectures, could be allowed.

[ianlancetaylor]

To me, this suggests that the Go developers often have problems where one change for one platform can easily cause problems for other platforms.

I can see how that follows, but I don't think it's quite right. A more accurate way of saying this is that Go developers often have problems where a change to the general purpose code requires separate individual tweaks to each individual platform.

[beoran]

Thanks for clarifying that. The question is then, how can we alleviate this?

[rsc]

The way to alleviate this would be to focus on specific, concrete changes, such as the unix build tag. The specific suggestion in this issue, namely the centralized platform package, won't work well with the standard library and is basically a non-starter. But it would be good to identify other possible changes.

We definitely want to make it easier to build out-of-tree ports. It doesn't scale to put all the ports in the tree. But we want to find ways that don't complicate the rest of the build too much (why the platform suggestion fails).

It sounds like we should close this issue in favor of #20322 and any other as-yet-unfiled suggestions.

[beoran]

That is fair enough, since I admitted my specific suggestion may not work, so this issue had no actionable content left.

Let's focus on concrete issues. I will leave it up to people who are actually making ports to come up with more detailed suggestions. I do hope the Unix build tag will be added soonish. I hereby retract this issue.