per-memory domain copies of libc globals

[andrewboie]

Is your enhancement proposal related to a problem? Please describe.
tl;dr: Library globals only have one instance even if the project has multiple logical applications sharing the same micro-controller. In addition we currently require 1 memory partition per library which scales horribly.

With CONFIG_USERSPACE, memory domains control user thread access to memory that isn't the thread's stack or areas marked at boot time for universal access, like program text.

If a Zephyr application uses a library, that library is statically linked into the combined application+kernel. If a Zephyr project contains multiple logical applications, all of these end up in a single statically linked binary.

It's possible to instantiate k_mem_partition to allow discrete access to library globals, but this still shares those globals with all users of the library, even if they would be otherwise completely isolated. Plus, having discrete memory partitions for every different library does not scale, because MPU regions are limited; having as few as 8 total is very common.

Describe the solution you'd like
At a fundamental level: library data/bss has unique instances per memory domain with a single MPU region to cover it.

Have a mechanism for allocating a properly sized and aligned block of contiguous memory for the data/bss of all shared libraries threads a memory domain will use. It should be possible to program an MPU region at its boundaries.

This memory could be reserved at build time, or we allocate it when memory domains are initialized. Initialization will also involve a data copy for the data section and bss zeroed. We'll call this the Memory domain library region (MDLR).

On context switch, the incoming thread will have access via a dedicated MPU region or the page tables to the MDLR for its memory domain.

When a thread references non-const library globals, some indirection is in place such that the data/bss for the library is in the current MDLR, analogous to how shared libraries work in Linux with the GOT for each process address space. The compiler/linker plays a role here and we should try to leverage something that exists. We don't want to write our own compiler or linker. Dirty post-processing hacks to the generated kernel ELF or intermediate .a files are on the table however as long as they are portable.

Unlike Linux, there is a requirement that the text and rodata for the library be contiguous with the kernel/application text/rodata, so we will not require extra MPU regions to grant access to it; it all must be covered by the boot regions for text and rodata (some platforms have a single read-only+executable region for both).

Only the data/bss must be position-independent.

Not every library needs this treatment; if only one memory domain uses a particular library, instead we would just want its data/bss merged into some memory partition that also has the rest of the doamin's globals (including any other library data/bss only used by that domain)

Additional context
This will really force us to distinguish between library code and subsystem code; only the latter needs to be a singleton entity in the kernel's space. I suspect a few items may switch places.

[andrewboie]

The lowest-touch solution I have so far does sort of resemble how thread-local storage works, where a register (or a segment descriptor on x86) is maintained representing a base address for the TLS, per thread. This could be abused to point to the MDLA instead for all threads in a domain.

In other words:

• Implement generic TLS Add Thread Local Storage (TLS) support #22185
• Compile/link all libraries in such a way that their non-const globals are treated as if they were declared with __thread, which will result in the bss/data ending up in .tbss and .tdata, with appropriate machine instructions to use the TLS register or data segment to get at it. I think this would unavoidably require source access for all libraries involved. I don't know what else would be required on the compiler/linker side to treat all non-const globals as if they were compiled with __thread.
• These libraries are still otherwise compile statically, we don't make them .so files
• For every memory domain, reserve a block of memory large enough for all .tdata/.tbss. This will be the domain-specific library memory.
• At runtime, for each thread, do not point the TLS register at a block of reserved memory in the stack buffer like we normally would. Instead, for all threads in a domain, point TLS instead at the domain-specific library memory.

Advantages:

• No need for PIC, code and rodata are statically linked, more efficient than true .so libraries
• No hacks needed at link time that I can see
• Single copy of text and data in the image
• One extra region for the MDLA

Disadvantages:

• MDLA must be the same size for all domains unless a domain uses no shared libraries at all
  • The TLS area is just a combined chunk of all the generated .tbss and .tdata
• We're essentially abusing the TLS spec, and what is supposed to be thread-local is actually hacked to be domain-local.
  • Breaks actual TLS; anything that actually has to be thread-local won't work.
  • We could special-case errno (we already have for a long time, Zephyr currently doesn't really support TLS as the compiler specifies it) and just say that TLS is not supported in this configuration.
• Source access would be required as the code must be compiled in a special way
  • For example we currently link in newlib as a third-party .a file.
  • Third-party firmware blobs I expect are a not-uncommon use case and sometimes you just get the binary

Unclear:

• Unclear how to get the library code to generate like this without going through the source and adding __thread to all non-const globals

[andrewboie]

One of the problems with using actual shared libraries is that the compiler assumes a fixed offset to the GOT from the library program text. All GOT lookups are IP-relative. This works fine in Linux because everything just gets mapped in the process address space.

In zephyr, we would need a GOT for every memory domain. In a physical memory map, the memory for each GOT would be at a different linear address, and the fixed offset guarantee doesn't hold.

One idea is to have a fixed GOT location which gets updated on context switch to the incoming memory domain, a copy is made. This would solve accessing the GOT.

Advantages:

• Plain old .so files should work without recompilation/source access, all the complexities are at link time
• Single copy of text and data in the image
• MDLAs don't need to be the same size for every domain, tailored to the actual libraries used since there is a separate GOT per library

Disadvantages:

• Extra overhead associated with PIC
• Extra memory partition for the fixed global GOTs. Maybe not so bad, we're saving many memory partitions by doing this for nontrivial applications anyway
• Extra context switching overhead to update the global GOT data, although at least it's predictable
• This approach falls over on SMP, where different memory domains can be active on different CPUs
  • with an MMU we can remap a per-cpu GOT page to a fixed virtual address
  • SMP+MPU wouldn't work.

Unclear:

• We still want all the library text/rodata to be contiguous with kernel/app text/rodata, so that we don't need extra MPU regions for each library to enable access. Very unsure if this is feasible, but this has been at least thought of before:
  • http://statifier.sourceforge.net/
  • http://magicermine.com/
• Unclear with multiple libraries how to control in the final linking the GOT location offsets for each library, they would have to all be in a contiguous region for the unified GOT area

[andrewboie]

There is a third option: link in multiple copies of libraries statically per memory domain.

Advantages:

• No funny tricks to get separate library data/bss per domain
• No extra indirection layers, PIC, etc
• Regular .a static libraries work
• Tighter control of what memory domains use what libraries, as they are more like separate applications, no equal-sized MDLA for every domain
• Separately built applications are something that has come up in discussions from time to time for other reasons.

Disadvantages:

• Multiple copies of almost the same program text for each library, using up either RAM or flash space (depending on XIP)
• For XIP systems, multiple copies of the data section, using up flash space
• Zephyr will need to be built and linked in a fundamentally different way, with each memory domain representing some different linking step
  • Discrete build for the core kernel and any IPC objects used by applications
  • Per memory domain build linked against the kernel
  • Final build step to create the image with everything arranged properly in memory
• Memory domains will be closely associated with the build and not the runtime constructions they are now, but maybe that's OK