Compiling for "internal" clients

.kotlin_metadata: what and why?

In the section "Compiling for external clients" we've traced how Kotlin Compilation Model works for exposing Kotlin API to non-multiplatform clients.

An observing reader might notice that there we've never compiler common sources alone: indeed, they were only accessed as part of a platform-specific compilation.

There are two most apparent consequences of that:

1. You don't know if your "common" code is "truly common" until you've compiled it for all targets.

As an exercise, try adding a reference to something like java.io.File to commonMain sources of TransitiveDependency and then run AssembleJvm. The compilation will pass absolutely fine! This is because those sources in JVM compilation are indistinguishable from the "true JVM" sources, so they have all rights to reference JVM-specific API. You will get an error only when you will try compiling for the non-JVM target (anything iOS in the current example)

2. It makes tooling life harder - specifically, IDEs.

Obviously, Kotlin Team doesn't want to say to you "hey, run the full build to figure out if you've accidentally used platform-specific API in the common code". The very explicit goal of Kotlin Multiplatform is to provide a sound and helpful analysis in IDEs.

So, imagine this: IDE should analyze DirectDependency/commonMain somehow. We know DirectDependency depends on TransitiveDependency. Against which artifacts we should analyze commonMain? So far we have only .jar and .klib. They kinda work, as they include Common API (we remember that platform artifacts are "fat" w.r.t. platform-common API), but they provide too much. If we use .jar, IDE will happily offer completion in DirectDependency/commonMain for declarations from TranstivideDependency/jvmMain. Not good.

Solution: compile commonMain alone into some Kotlin-specific form. Specifically, Kotlin Compiler will produce .kotlin_metadata, which is basically serialized Kotlin declarations. Currently, it is then packed to .jar-file and distributed in this way.

One might see a weird inconsistency that this is the second Kotlin-specific binary format for distributing Kotlin declarations, the first being .klibs we've encountered when studying K/N platform-specific compilation. Indeed, this is an inconsistency, that mostly happened due to the fact that common code appeared before .klibs. In the distant (or maybe not so distant, no promises 🙃 ) Kotlin of the future .klibs might be used there as well.

Hierarchies

So far the explanation was deliberately using the term "common" code with an example of commonMain.

In fact, Kotlin Multiplatform provides an ability to express different degrees of "common" code, instead of binary separation into "completely common code" and "fully platform-specific code" (read more here), in form of so-called "intermediate source sets".

This is specifically apparent for Kotlin/Native due to the sheer amount of Native targets, and actually the current project has a very typical example: there's an intermediate iosMain source set. Indeed, it shares code across only iOS targets (so it's not the same as commonMain), but it's still not the platform-specific code yet.

In fact, in our example there are unused source sets iosArm64(Main|Test), iosX64(Main|Test), iosSimulatorArm64(Main|Test) that correspond to completely platform-specific code for iPhone devices, iPhone simulator on Intel Macs or iPhone simulator on M1 Macs respectively. They are empty in our example, and often empty in the real world, as you rarely need some code/API which works/available specifically on the simulator, but not the device.

Intermediate source sets, in general, receive similar treatment to common source sets: they are compiled into .kotlin_metadata solely for purposes of additional code soundness check and tooling support.

However, the compilation of Native-shared source sets (like iosMain) is connected with additional challenges. Specifically, users expect most native libraries to be present there, like Foundation for example. However, there's no such thing as "Foundation, common across all iOS targets". To provide sound and precise code analysis, in the real world Kotlin toolchain actually builds this "common Foundation" on-the-fly, by intersecting APIs of all platform-specific versions of Foundation.

As this process requires a few additional tools and invocations, this is omitted from the current example.

Summary

Once again, it's important to reiterate why we're compiling .kotlin_metadata:

• In case you're publishing your Kotlin Multiplatform library to some repository, .kotlin_metadata will help consumers to get sound analysis in their common code against your library.

  • Note that if you depend on library sources, IDE will analyze commonMain against library' commonMain sources, and .kotlin_metadata is not needed

• If you want a quicker check of code soundness.

One can derive interesting corollaries from that:

• Corollary 1. If you're working in a monorepo and never publish your Kotlin Multiplatform Libraries, then you don't actually need to compile .kotlin_metadata. Indeed, because this is a monorepo, you will be fine with source-to-source analysis.

• Corollary 2. If you know that your library will be consumed only in platform-specific apps (Android, iOS apps), and never in other Kotlin Multiplatform Libraries, then you don't need .kotlin_metadata again. Indeed, as we've learned, platform-specific clients consume "fat" platform-specific artifacts (.jar or Framework) and are not interested in .kotlin_metadata