proposal: spec: lazy init imports to possibly import without side effects #38450
Comments
bradfitz
changed the title
|
Would it be correct to say that you mean something like: if there are no references to any exported names of this package, then there is no need to run any initializers? I certainly understand that the current situation is frustrating, but it seems possible to implement this entirely in the compiler without changing the language. Basically we need to separate the initialization of each global variable into a separate init function. We put those init functions in the deadcode graph along with everything else, with a reference from the variable to the init function. The package init function then calls only the undiscarded init functions. |
Yes.
In many cases, yes, but not all. For one example that's only partially contrived: https://play.golang.org/p/2GLltT92h1t You can't do that automatically without changing language semantics. Or you can't get rid of imports to But, yes, if the various bugs like #19533 #14840 were fixed, it's unlikely I'd be filing this. I just think such toolchain magic continues to be years away and I wonder if a more explicit mechanism has a better chance of being implemented. |
|
How do you end up with those kinds of (Are these due to debugging constants, or paths rendered unreachable due to build tags?) |
|
It seems like it would make sense to start with first handling the cases that don't require changes to language semantics, and then evaluate what's left. |
They arise in a dozen different ways. That was just an example for brevity. Sometimes build tags, sometimes GOOS comparisons, sometimes just statically unreachable code. |
|
I have a very initial implementation for #19533 here: This splits generated init functions into separate symbols (using a relocation to bind them to the symbols they initialize), and the linker deadcode pass is then able to remove them in "some cases". In fact, it's not sufficient for the linker to see that the symbol is not used: it also needs to know that the init function itself is "side-effect free". For instance: might not be side-effect free, depending on the body of By the way, this also require to agree on the exact definition of "side-effect free"; for instance, many things are visible through a debugger. It would also be possible to have a compiler annotation for std functions which are commonly used in init functions, and we can't otherwise prove that are side effects free (even in the best possible world where we have a fully working side effect detection pass, they might still end up calling a syscall which needs to be somehow annotated anyway). So yes, this is actually a complicated issue that requires many development hours. It's possibly a task bigger than I can afford in my spare contribution time, so I'm not sure I'll get around completing it. |
|
@rsc, here's a more concrete example distilled from above.
If you don't use https://golang.org/src/crypto/md5/md5.go#L21 What I'd like to do from In fact, the only use of https://golang.org/pkg/crypto/#Hash.Available in x509 is: switch hashType {
case crypto.Hash(0):
if pubKeyAlgo != Ed25519 {
return ErrUnsupportedAlgorithm
}
case crypto.MD5:
return InsecureAlgorithmError(algo)
default:
if !hashType.Available() {
return ErrUnsupportedAlgorithm
}
h := hashType.New()
h.Write(signed)
signed = h.Sum(nil)
}... where it only uses the crypto.Hash registration mechanism after it's determined that it's not MD5. |
…cryption Saves a dependency on crypto/md5. See golang#38450 (comment) Updates golang#38450
Are side effects from indirectly-imported packages covered by the Go 1 compatibility promise? The following code technically works today... https://play.golang.org/p/lw5P2Vb5NXH package main
import (
"crypto"
"fmt"
_ "crypto/x509"
)
func main() {
h := crypto.MD5.New()
fmt.Println(h.Sum(nil))
} |
No clear rule either way. I'm sure we've even already broken it in the past without knowing since the fix is so easy for people. It's not something we automatically test for at least. We'd definitely avoid trying to break people if possible. |
…cryption Saves a dependency on crypto/md5. See golang#38450 (comment) Updates golang#38450
|
The other way to handle cases like this is to move all of the functionality to a new package that exports explicit, idempotent registration functions instead of implicitly executing them at Then, replace the declarations in the original package with forwarding shims to the new package, with the Users who want the “lazy init” behavior can import the new (light) package instead of the old one to eliminate the That approach does not require a language change. |
|
Instead of using a @bradfitz's original example would become:
|
|
The crypto package is bigger than the runtime, but ssl has to use it. I run it on the openwrt mips device, and its ROM is only 16M... |
rsc
changed the title
|
This proposal has been added to the active column of the proposals project |
Doesn't this description omit the (common?) pattern of, say, the Let's imagine: package main
import (
"database/sql"
"some/random/db/driver"
}
func main() {
if false {
fmt.Println("using ", driver.Version)
}
db, err := sql.Open("driver", ....)
}In this case there may be no references to exported variables in the Unless exported methods on unexported types are counted as exported symbols, of course. |
|
Go 1.21 will be able to remove dead code from map initialization and the like. From a semantic level, it seems clear we don't want to take on the complexity of a new kind of import right now. Better to fix the offending packages. |
|
Based on the discussion above, this proposal seems like a likely decline. |
|
No change in consensus, so declined. |
I've been doing a bunch of work on binary size reduction and a common problem that comes up is that the linker's dead code elimination can remove all the references to a package, but the one remaining reference is to the package's
inittasksymbol, which is then heavy & brings it a bunch more.That is:
Is equivalent today to:
... which slurps in a ton of packages and their init load and extra 1 MiB binary size increase, from 1.3 MiB to 2.3 MiB:
The only way to get around that is with build tags and more conditional compilation, which is gross.
What I'd like instead is a way to declare to the toolchain that for a given imported package that I'm fine with that package's init-time side effects (like normal) if I need them, but I'm also cool with omitting them if the linker decides that's fine.
That is, I want something like this this strawman syntax:
... so the x509 init (nor its deps) is never run if the toolchain's DCE doesn't want to.
The
go:lazyinitis saying that I'm not depending on any init-time work happening there for the import on that line. (Or perhaps it should be on the line before to be consistent with other//go:comments, or it shouldn't use comments)(Arguably this should be the default behavior and imports with side effects would need the declaration, but we can't for backwards compatibility anyway, so not worth considering.)
/cc @ianlancetaylor @griesemer
The text was updated successfully, but these errors were encountered: