proposal: Go 2: spec: remove init functions

[kevin-matthew]

I previously submitted the issue that the inclusion of a package can sometimes quintuple your binary's size because of the use of init functions. This issue was not original as others have reported the same:

• cmd/link: linker should be able to remove init functions with no side-effects #19533
• proposal: spec: lazy init imports to possibly import without side effects #38450
• layers.init cost too much time  google/gopacket#101

At the current state of Golang, these issues are only blamed on the init functions in questions, citing that these init functions just need to be better written. I argue that init functions as a whole need to be removed for the reasons below, note that I do not blame "bad code" for their removal. I'd like to prove that init functions are objectively a bad feature.

Note: when I say "init function(s)", I mean both func init()s and the initilizer variables (ie var myval = myValInit() in global scope)

init functions make programmers avoid imports, hindering comprehensiveness

If a package has an init function that is taking a long time to run, maybe don't use that package?

That's a problem for programmers wanting to use some types found in a package but don't actually want to execute any of the functions. Importing a package to use just its types makes the programmer's executable/plugin have additional (and useless) execution thanks to init. Sometimes forcing the programmer re-write the package's types, but doing so makes their executable/plugin incompatible/non-interchangeable with the original "upstream" package. Furthermore, that upstream package probably included important documentation and tests that are left behind by the programmer because all they require are the types, possibly hindering further work on the programmer's executable/plugin as well as have the programmer forced to manage even more code.

init functions have no way to allow the programmer to handle errors

Go is extremely focused on making sure the programmer focuses on handling errors rather than ignoring them. This was declared multiple times in both Go at Google as well as the FAQ.

However, init functions force everyone to ignore all possible errors. The author of an init function has no way to provide errors to the importer, so if an error would happen in an init function, more unexpected disasters will follow. A workaround for this would have the programmer check for the packages initError (for example) to make sure the init function didn't run into an error... but at that point, you might as well have the programmer execute the init function explicitly because it's no longer automatic.

init functions are not explicit to the programmer's actions

You must be explicit with everything in golang. init functions are never called explicitly. One could argue that the act of importing a package is an explicit action to call the init function, but I disagree: you have to go through the entire package to find the init function as well as all init declarations, a laborious task that makes importing a package more of an investigation rather than an explicit execution.

I'd also like to point out golang's policy on implicit int conversions...

The convenience of automatic conversion between numeric types in C is outweighed by the confusion it causes. When is an expression unsigned? How big is the value? Does it overflow? Is the result portable, independent of the machine on which it executes?

For reasons of portability, we decided to make things clear and straightforward at the cost of some explicit conversions in the code.

... notice the length golang goes through to make things straightforward and explicit by disallowing the smallest things such as implicit int conversion. Yet importing a package can implicity call an unspecified amount of init functions executing an unspecified amount of code that performs an unspecified amount of tasks for unspecified reasons... and it does this recursively (as a single import can have even more imports). That is a MASSIVE amount of execution that occurs just because of a simple import.

Going back to Go at Google, Rob Pike himself was complaining about circular imports by saying the following:

More important, when allowed, in our experience such imports end up entangling huge swaths of the source tree into large subpieces that are difficult to manage independently, bloating binaries and complicating initialization, testing, refactoring, releasing, and other tasks of software development.

Notice Rob Pike's despise of bloating binaries and complicated initialization when talking about things to avoid when making golang's import system. These are the two very issues we're running into thanks to init functions.

In conclusion.

There's a ton of packages that depend on init functions right now, but users of those packages are surprised with huge bloats to their binaries for no obvious reasons as init functions are overly-implicit. Plus, in most cases we've experienced, the outcome of init functions rarely affects the use of the package. This is wrong, Golang is very much about explicitness over convenience.

[ianlancetaylor]

While language changes do not have to be backward compatible, it's hard to see how we could adopt a language change that would break an enormous number of existing packages.

[kevin-matthew]

@ianlancetaylor

Perhaps in the future, legacy imports could be given an optional "flag" that will allow init functions to be executed. That way, the explicitness is preserved without leaving unmaintained packages broken.

[ianlancetaylor]

It seems that the desire here is to let packages be lazily initialized only when they are needed. But the language already supports this. We don't need to disallow initializers to permit lazy initialization.

Removing initializers removes a widely used convenient mechanism to safely ensure that package invariants are established.

Go has gone to a great deal of trouble to get initializers right, in the sense that they work reliably and predictably, which is not true in some other languages. It seems very hard to justify removing this existing feature.

If we did accept this idea, how do you suggest that existing code be rewritten systematically and safely to avoid initializers?

[beoran]

While I agree that using init functions is often not needed, they are essential to GUI or game libraries in Go which must lock the OS thread for technical reasons. Hence init remains needed for this use at least.

[kevin-matthew]

@ianlancetaylor

So as of now, importing a package has the following syntax:

ImportDecl       = "import" ( ImportSpec | "(" { ImportSpec ";" } ")" ) .
ImportSpec       = [ "." | PackageName ] ImportPath .

And as I've mentioned in my previous comment, I suggested adding a flag that would enable init functions would preserve go's explicitness. However, after putting some thought into it, I propose the inverse of that property. This means the programmer can specify a flag on a particular import that will inform the compiler NOT to include the init functions (as well as function initializers). For example:

// option 1:
ImportDecl       = "import" [ "idle" ] ( ImportSpec | "(" { ImportSpec ";" } ")" ) .
ImportSpec       = [ "." | PackageName ] ImportPath .
// import idle "fmt"
// import idle ("fmt"; ...)
// note: this isn't a good option because "idle" would colide with "PackageName"
 
// option 2:
ImportDecl       = "import" ( ImportSpec | "(" { ImportSpec ";" } ")" ) .
ImportSpec       = [ "." | PackageName ] ImportPath [ "idle" ] .
// import "fmt" idle
// import ("fmt" idle; ...)

Here I use the fictitious token "idle" for its specific meaning of "avoiding work". Other vocabularies to consider: "passive", "limp", "still", "rigid", "ossified", "inert", "static". So long that the vocabulary's connotation has a natural tendency to deter its use by newer programmers (as this feature can indeed be confusing unless the programmer knows what he/she is doing). And the exact syntax is also subject to debate, mainly in regards to option 1 which allows multiple idle imports, or option 2 making it more deliberate per-import.

Overall I consider this an elegant solution. I'd suggest we actually put this in Go 1 because it will solve the larger problems I originally mentioned without forsaking the function of packages that rely on init functions such as the ones @beoran pointed out. It simply wouldn't affect anything other than offering programmers better options for imports.

However, for Go 2, we are still limited by error handling and falling short of being an explicit language, for that reason I'd recommend my former solution of either getting rid of them completely or making init functions explicit.

Semantically there are still some questions to be answered... and admittedly I'm not prepared to have answered them because I cannot predict their impacts without some further feedback.

• Will a package that's imported "idly" have its dependencies imported "idly" as well (will idle imports be force-recursive)?
• What will happen if the programmer imported one package idly, and imported that same package again package normally (in the same package/file)? Should that even be possible?

[ianlancetaylor]

If a package can be imported without running its initializers, then there must be some mechanism to let that package work even though it was not initialized. If there is no such mechanism, then using import idle can't be used safely. But if there is such a mechanism, then why not write the package to always use it? It seems to me that this is a decision to be made by the package itself, not by other code that imports that package.

[kevin-matthew]

import idle can't be used safely but neither can a normal import. The former can do too little, the latter can do too much. However, right now everyone stuck with the latter when there's a demand for the former, I'd argue it's a matter of imbalance.

Indeed, import idle can cause issues by disallowing packages to be initialized. However, this liability is on the importer, and it is the importer that must explicitly express that they require an idle/inert package, and for that reason, there wouldn't ever be a problem that can be blamed on the limitations of the language. Comparatively, the current state is that init functions can hinder the importer's performance and limit the importer's visibility which is indeed blamed on the language's limitations.

There have been dozens of times I had to import a package into mine because I only required a handful of types and/or functions from that package only to be bloated by init functions that have nothing/no effect on what I was using the package for.

A normal import carries the risk of bloated/useless init functions... which is at the importer's risk but outside of the importer's control. import idle will also be at the importer's risk, however, within the importer's control. Thus import idle is a much more reasonable option of risk management for the importer.

[beoran]

I think init() functions are certainly overused, and should be replaced by explicit initialization wherever possible. However, we can do this without changing the language. The init function should be documented in such a way to dissuade it's use it except for special cases.

As for existing packages, if they are under an open source license, you may fork them to get rid of the init().

An import idle is a bad idea though, because you are trying to override in one package what another package will do when imported. This is a 'spooky action at distance'. Better modify the imported package to do what you want, if possible.

[ianlancetaylor]

import idle can't be used safely but neither can a normal import. The former can do too little, the latter can do too much.

I can't agree. Doing too much may not be ideal, but it is not unsafe.

[kevin-matthew]

@beoran

The init function should be documented in such a way to dissuade its use except for special cases.
As for existing packages, if they are under an open source license, you may fork them to get rid of the init().

This is certainly a great start. I'd consider this a proper solution for the time being.

[mdempsky]

That's a problem for programmers wanting to use some types found in a package but don't actually want to execute any of the functions.

Does this come up often? Typically if I want types from a package, I want functions and maybe variables from it too. And those functions/variables may depend on the package initializers for correct behavior.

If we had evidence that this use case really did come up often, I think we could add a "types-only" import where you only get access to the types and consts declared by a package, and it doesn't force initializers to run. However, this would have to be fairly restricted. E.g., we'd probably need to disallow storing types that are imported this way into an interface, unless the type's method set is empty.

I suspect it comes up rarely enough that it would be simpler for the package authors to just take responsibility for factoring an appropriate initializer-free subset.

Separately, I'll note there's work-in-progress for Go 1.17 to automatically remove side-effect-free initializers from programs when they're not needed. E.g., a global map that's initialized at program startup, but then none of the code that uses it is included into the final executable.

[ianlancetaylor]

Based on the discussion above, and the lack of strong support in the emoji voting, this is a likely decline. Leaving open for four weeks for final comments.

[griesemer]

No further comments. Closing.