can/should exports re-export?

[m4b]

Symbol re-exporting is when an export (publicly visible and linkable symbol) is tagged with meta-data that signals to the linker or dynamic loader/linker that the symbol either:

1. resides in another library with:
   a. same name
   b. different name
2. resides in the same library with:
   a. same name
   b. different name

Ostensibly, the point of the re-exporting mechanism is to:

• decouple the symbol from its implementation
• allow platform specific code for linking against the same symbol
• allow forwarding symbol names to other/different symbol names

There are various reasons why you'd want to do this, some applicable to wasm, some not.

In the three major binary container formats, each implement a version of re-exporting for their declared exports.

• PE32 uses the ForwarderChain
• ELF uses the GNU_IFUNC
• Mach-o uses special tagging in the export trie's entry

So can (or should) exports in wasm be able to re-export, and if so, how would this be implemented/represented?

[jfbastien]

Sounds like something for @dschuff's dynamic linking exploration!

[lukewagner]

This feature makes sense to me and I think it would be simple enough to implement and specify: effectively we'd just change the func field in the definition of export into a choice between a function-index (what func currently is) and an import-index.

[jfbastien]

Is there a point in having this pre-MVP?

[m4b]

On hindsight, the static linker probably isn't being clever in the libc case and probably dumps the whole lib into the static executable; I'd have to mess around with it more but I think you get what I mean

Also, so just to be clear it's not just a dynamic linking problem; the static linker has to resolve reexports as well.

Consider the following C program:

#include <string.h>
#include <stdio.h>

int main(){
  char deadbeef[] = "deadbeef\0";
  char efbeadde[] = "efbeadde\0";

  strncpy(deadbeef, efbeadde, 9);
  printf("%s\n", efbeadde);
  return 0;
}

If you compile with something like gcc -static deadbeef.c -o deadbeef on linux the resulting binary has no dynamic library imports, and all reexports are resolved (I'm fairly certain strncpy in later libc's is a GNU_IFUNC, but if not, above example mutatis mutandi with an appropriate ifunc).

NOTE: this will not work on OSX as they don't support completely static executables.

[jfbastien]

I expect that for MVP the program you suggest will only have imports from the embedder. Do you expect the same?

[m4b]

@jfbastien not sure if your question was intended for me, but the behavior I'd expect with statically linked, non-dynamic reexports is roughly as follows:

1. resolve the location of the import; i.e., follow the re-export chain (inter-module if this feature is allowed) until it terminates with a non-re-exporting symbol;
2. embed that export's code/function/expression in the importing binary in whatever way the spec defines how to statically import exports.

[m4b]

@lukewagner that's actually a very elegant way of thinking about it.

I should also note that this will implement inter-module re-exporting.

For example, on OSX _memset is:

1. initially exported in /usr/lib/libsystem_c.dylib as a re-export, inter-module, to /usr/lib/libsystem_malloc.dylib
2. /usr/lib/libsystem_malloc.dylib also inter-module re-exports _memset to /usr/lib/libsystem_platform.dylib with a new name __platform_memset.

I have to look up if GNU_IFUNC's implement inter-module re-exporting (I think they only do intra-module reexports).

So the export definition also would have to take care of the case of reexporting to symbols already in the binary (exported or local)

[dschuff]

@jfbastien The only thing we spec in wasm proper is a way to name imports and call them (e.g. import function "foo" from module "bar"). It's up to the embedder whether this results in calling a function defined by user JS, another wasm module, or some magic builtin module. The embedder will have to specify e.g. can you import from other wasm modules or does everything have to come from JS (presumably this wouldn't be ES6-module JS in the MVP because AFAIK ES6 module JS isn't implemented anywhere yet). That's probably outside the scope of the wasm spec, but it's something we'll probably want to define at least somewhere other than the source code of the various browsers because we'll want some compatibility. (e.g. we've said we don't want to spec posix, but a user who wants to write a wasm module that's usable in all the browsers will want to know what, if any, functionality can be expected to be available).

What we have in wasm-without-dynamic-linking isn't quite the same as traditional static linking, because we still have the notion of named modules and importing, whereas static linking does not (as you observed, the linker has to do the resolution at link time; that kind of link-time resolution would be outside the scope of wasm proper). For pre-dynamic-linking modules, I think the idea has been that we want to allow instances of different modules in the same page, and re-exporting would basically mean "inter-module" in this context.

I do think this might be a reasonable thing to have even in the pre-dynamic-linking import/export system. From the perspective of this spec it's pretty simple because there's a function table for local functions and one for imports, and exporting could conceptually work the same way for both of them.
It does impose an extra requirement on the embedding environment (because it would add the requirement to traverse chains rather than just one hop to resolve things) and there might be details that would have to be specified outside the wasm spec. I'm not sure where exactly the best place for that might be.

[dschuff]

I think @ncbray has been thinking about embedding environments recently...

[kg]

Re-exports get used for basic stuff in the Visual C++ runtime library too. Off the top of my head, strdup and free.

It does impose an extra requirement on the embedding environment (because it would add the requirement to traverse chains rather than just one hop to resolve things)

Couldn't we just specify it so that a re-export resolves the chain before exporting, so no hopping is ever necessary?

[dschuff]

Yeah I think you're right. Ideally we'd maybe we'd spec it so that both of the following would be allowed:

Suppose module B imports C.foo and re-exports it as B.bar. Module A imports B.bar

1. B.bar is just generated as a local thunk that calls C.foo. A call from module A would then be 2 calls.
2. A call in module A to B.bar would call C.foo directly (or by "directly" I just mean that a pointer to C.foo could go into A's import function table rather than a pointer to B.bar.

For 1, the loader would not need to know anything about modules more than one export-hop away.
In order to make 2 work, the loader would have to follow B's re-export. But the latter would be an optimization that we don't necessarily need to require.

[m4b]

above comment by @dschuff with two cases spells out the situation nicely. Note while in the case of 1. it puts no burden on the loader to resolve anything, it would require a check to see if the module C is present, otherwise you'd have a runtime error with a missing symbol; in the second case however, you can think of it as a compile time resolution and hence check for module C, whatever that might mean

I think it depends on how the re-export gets resolved, certain details of which @dschuff mentioned may be outside the scope of wasm proper.

Nevertheless, from how I'm understanding things (and please correct me), that in the non-dynamic link scenario with multiple modules on a single page, re-export resolution could be a compile-time resolution to the final resolved export. But this would be up to the implementation's embedder/linker whatever you want to call it.

Why we're resolving also depends on the use case; in the case of strncpy, et. al., the symbol is re-exported because it allows run-time switches to select the best implementation for a particular machine, but allowing the same symbol name to be imported. (i.e., it decouples the implementation from the symbol name).

In the binary I gave above, you can see this in the disassembly where the cpu features are tested and a different implementation is selected (sse2, ssse3, etc.):

Dump of assembler code for function strncpy:
   0x000000000041ced0 <+0>: cmpl   $0x0,0x2acf69(%rip)        # 0x6c9e40 <__cpu_features>
   0x000000000041ced7 <+7>: jne    0x41cede <strncpy+14>
   0x000000000041ced9 <+9>: callq  0x43ea30 <__init_cpu_features>
   0x000000000041cede <+14>:    lea    0x1891b(%rip),%rax        # 0x435800 <__strncpy_sse2_unaligned>
   0x000000000041cee5 <+21>:    testl  $0x10,0x2acf91(%rip)        # 0x6c9e80 <__cpu_features+64>
   0x000000000041ceef <+31>:    jne    0x41cf0b <strncpy+59>
   0x000000000041cef1 <+33>:    lea    0x7e48(%rip),%rax        # 0x424d40 <__strncpy_sse2>
   0x000000000041cef8 <+40>:    testl  $0x200,0x2acf4e(%rip)        # 0x6c9e50 <__cpu_features+16>
   0x000000000041cf02 <+50>:    je     0x41cf0b <strncpy+59>
   0x000000000041cf04 <+52>:    lea    0x13f35(%rip),%rax        # 0x430e40 <__strncpy_ssse3>
   0x000000000041cf0b <+59>:    retq   

[jfbastien]

Discussing just the MVP: IIUC re-exporting makes no sense if a wasm module always has to import from the embedder. It only matters if a wasm module is importing directly from another wasm module (which indirectly pulls in a third module's code). Correct?

I don't like this, because it effectively means we have some form of cross-heap sharing, instead of forcing the embedder to marshal everything. e.g. the following code would copy from two module's heaps if exported and directly callable:

static char my_buf[SZ];
void copy_in(char buf[static SZ]) {
  for (size_t i = 0; i != SZ; ++i)
    my_buf[i] = buf[i];
}

So back to my question: does this feature have any use in MVP? Do we want it before MVP?

[m4b]

(in the dynamic case) I should also note that it is very easy to compile-time resolve re-export chains. But this implicitly breaks the semantics and intention of re-exporting symbols (which is why it isn't done).

The reason for this is that you might have a library like libc which exports foo, which is a re-export from _libfoo, which might have any number of further re-exports to some final implementation.

But in the dynamically linked situation, if your static linker compile time resolved this symbol in your binary bar, you'd be importing the final resolved symbol's implementation - which defeats the whole purpose of re-exporting, since it doesn't allow libc to update the implementation of foo, say switching it out with a better implementation when _libfoo is updated, etc. --- and thereby auto-update bar's use of foo, irrespective of its implementation (because the final import in bar was the final resolved symbol's implementation, not foo) Again, the whole decoupling symbol from implementation argument.

OSX's system libraries do this extensively, and so does msvcrt.dll - all on purpose.