Add pthread.h in wasi sysroot to support pthread

[xujuntwt95329]

Hi,

The pthread library has been supported in wasm micro runtime. The pthread related APIs in wasm application should be imported from the runtime, such as pthread_create, pthread_join, etc. so the runtime can create an actual thread to execute the wasm function.

For example, suppose we have such a C code:

#include <pthread.h>

void *func(void *arg)
{

}

int main()
{
    pthread_t tid;
    pthread_create(&tid, NULL, func, NULL);
    return 0;
}

The pthread_create should be compiled as an import function:

(import "env" "pthread_create" (func $pthread_create (type $t1)))
......

In this situation, I think there should be a simple way to support pthread in wasi-sdk as we only need a few things:

1. type definition for pthread_t, pthread_mutex_t and so on
2. function prototype for pthread APIs such as pthread_create, pthread_join, pthread_self, etc.
3. undefined-symbol lists

To achieve 1 and 2, we just need a pthread.h under sysroot/include, just like WAMR's pthread.h, we also need to undef some macros like __NEED_pthread_t to avoid duplicate definition with sysroot/include/bits/alltypes.h
And to achieve 3, we need to add pthread API symbols into sysroot/share/wasm32-wasi/undefined-symbols.txt just like wasi APIs

In this way, we don't need to provide libpthread.a in wasi-libc, as the pthread related APIs should be implemented by the runtime.

Do you think this is a reasonable way to support pthread ?

[xujuntwt95329]

Furthermore, I'm trying to build some C++ workload which use std::thread, but I get this error:

error: <atomic> is not supported on this single threaded system

I noticed that the std::thread depends on pthread. In this case, if we have supported pthread, is there any other thing to do to support std::thread ?

Thanks a lot !

[sunfishcode]

The current situation is that there is a consensus among the people working on threads support in the WebAssembly core spec is that the current semantics are not yet suitable for use outside of Web contexts. See the following issues for background:

WebAssembly/threads#8
WebAssembly/threads#95
WebAssembly/threads#138

Following this guidance, WASI libc does not yet attempt to support pthreads. This guidance is of course open for discussion.

If you're looking to do something WAMR-specific and don't need anything in libpthread.a, then you don't need WASI libc to help; you can just define those APIs and provide the header file yourself.

[MaxDesiatov]

The lack of pthread headers becomes problematic when attempting to use higher-level APIs in WASI SDK, such as C++ <atomic>. While users can provide the header, this still requires recompiling WASI SDK from scratch, so I wonder if it would be possible to ship pthread headers in WASI libc after all with a no-op implementation to allow compiling C++ code that relies on <atomic>.

[sbc100]

BTW, with llvm 11 there will be no need for undefined-symbols.txt at all anymore since symbols can be marked as imported in the source code.. so using a recent build of llvm should make that issue at least go away.

[sbc100]

It is a shame that <atomic> doesn't just work.. since at the llvm we lower them all away in single threaded mode.

Are you sure it would require a rebuild of libc rather than just the addition headers? I think this issue (allowing atomics to work in single-threads builds) is somewhat of a separate issue to actually add pthreads and/or compiling with threading/shared memory.

[MaxDesiatov]

Could you elaborate please what do you mean by "the addition headers" here? This is the error that I get currently without the pthread header, building wasi-libc with THREAD_MODEL=POSIX and without building the WASI SDK with LIBCXXABI_ENABLE_THREADS=ON, LIBCXXABI_HAS_PTHREAD_API=ON and COMPILER_RT_EXCLUDE_ATOMIC_BUILTIN=OFF:

wasi-sdk/share/wasi-sysroot/include/c++/v1/atomic:560:3: 
error: <atomic> is not supported on this single threaded system

[sunfishcode]

When we initially set up all these builds, we configured them for "no threads" mode, because we care about code size and have since the beginning, and some libraries have smaller code size than if they are built in threads mode and we just lower atomics. I don't know if that's specifically true of libcxx though, so if someone could investigate that, that'd be a good start.

If it does turn out that libcxx is bigger when compiled in threads mode, a possible option would be to produce two builds, one where everything is built with threads disabled, and one where threads are ostensibly enabled but atomics are lowered.

[SamuraiCrow]

Circling back to this issue, It appears that thread-local storage is needed to activate multithreading at this time. As a POSIX newbie, what does it take to generate a single-threaded pthreads implementation that doesn't need TLS?

[MaxDesiatov]

AFAIK clang supports -femulated-tls, but I don't know if it's working for the Wasm target, and I haven't tried it with WASI. But that's where I'd start looking first.

[SamuraiCrow]

Thanks for the prompt reply!

[SamuraiCrow]

fatal error: error in backend: only -ftls-model=local-exec is supported for now on non-Emscripten OSes: variable errno
is what I get. Will it be possible to mock up a single threaded pthreads lib?

[MaxDesiatov]

Did you try adding the suggested -ftls-model=local-exec flag then?

[SamuraiCrow]

I didn't know the two could work together. I'll try again. Thanks a bunch!

[SamuraiCrow]

I got the same error as before.

[MaxDesiatov]

I think LLVM/clang would need some modifications to support -femulated-tls. I can't say anything on behalf of maintainers of wasi-libc, but I tried to mock TLS and submit patches to other repositories, and review feedback was that this should be resolved with -femulated-tls in clang/LLVM, instead of adding mock TLS implementations in client code.

[SamuraiCrow]

Ugh. Ok, thanks.

[MaxDesiatov]

To clarify, the motivation is that TLS support for WebAssembly should be implemented once on clang/LLVM level, instead of adding it separately every time in all client code that depends on TLS, which would ultimately save us a lot of effort in the long run.

[SamuraiCrow]

Oh ok.

[SamuraiCrow]

I found out I added the flags to the wrong spot. Now I got farther.

[SamuraiCrow]

I'm really close. It compiles and links but fails the test with:

build/wasi-libc.BUILT: build/llvm.BUILT
	$(MAKE) -C $(ROOT_DIR)/src/wasi-libc \
		WASM_CC=$(BUILD_PREFIX)/bin/clang \
		SYSROOT=$(BUILD_PREFIX)/share/wasi-sysroot \
		WASM_CFLAGS="-U_REENTRANT -femulated-tls -ftls-model=local-exec -O2 -DNDEBUG" \
		THREAD_MODEL=posix
	touch build/wasi-libc.BUILT

[MaxDesiatov]

what kind of error messages do you see?

[SamuraiCrow]

2 errors:

/home/samuraicrow/Documents/wasi-sdk/build/install/opt/wasi-sdk/share/wasi-sysroot/include/signal.h:2:2: error: "wasm lacks signal support; to enable minimal signal emulation, compile with -D_WASI_EMULATED_SIGNAL and link with -lwasi-emulated-signal"
#error "wasm lacks signal support; to enable minimal signal emulation, \
 ^
In file included from /home/samuraicrow/Documents/wasi-sdk/build/install/opt/wasi-sdk/share/wasi-sysroot/share/wasm32-wasi/include-all.c:63:
/home/samuraicrow/Documents/wasi-sdk/build/install/opt/wasi-sdk/share/wasi-sysroot/include/aio.h:21:18: error: field has incomplete type 'struct sigevent'
        struct sigevent aio_sigevent;
                        ^
/home/samuraicrow/Documents/wasi-sdk/build/install/opt/wasi-sdk/share/wasi-sysroot/include/aio.h:21:9: note: forward declaration of 'struct sigevent'
        struct sigevent aio_sigevent;
               ^
2 errors generated.

[SamuraiCrow]

I'm really close! Adding -D_WASI_EMULATED_SIGNAL \ to the test and #define _WASI_EMULATED_SIGNAL 1 to the expected results clears those errors. Now it's found a symbol called emutls_v.errno that I'm sure is also supposed to be there but needs to be stripped from the test results so it doesn't look compiler-specific.

[SamuraiCrow]

Adding that symbol to the expected results clears the test! Now I just need to test it and update the submodules in my fork of wasi-sdk to use my fork of wasi-libc!

[SamuraiCrow]

I may have identified a rare bug in asyncify: If there are two yields in the same function does resuming the second yield call return to the correct place?

[kripken]

@SamuraiCrow That should work in Asyncify. The mechanism is that we serialize an ID for the call location in the function, so we know how to return to the right place. We have tests for this that pass, but perhaps you've found a corner case somehow where we are buggy - if you can get a testcase that would be good.

[SamuraiCrow]

Perhaps I was looking at an oversimplified example of how Asyncify was supposed to work. If its design already accounts for the corner cases, it should meet my needs.

[SamuraiCrow]

Emscripten has a PThread stub that may accomplish what I was trying to do here instead of Asyncify. Asyncify looks rather involved.

[sbc100]

Does you code actually need to do threading? Or can it run in a single thread if needs be (i.e. can run on a system where pthread_create() always fails)? The pthread stubs in emscripten are a version of the pthread API that doesn't do anything and doesn't allow for the creation of any threads.. just FYI. Its uses in the case where you compile a thread-aware program but you are targeting a single-threaded runtime.

[SamuraiCrow]

There's a conditional compilation flag that allows single-threaded execution but I didn't know about it until recently.

[mmathys]

Any news to this issue? I am trying to figure out as well how to compile code with threads with WASI.

[sbc100]

There is ongoing work to support building with threading support. See #311

[turbolent]

@sbc100
I tried using wasi-sdk with -pthread and wasm-ld reported the error:

--shared-memory is disallowed by errno.o because it was not compiled with 'atomics' or 'bulk-memory' features.

I found https://stackoverflow.com/questions/70180309/how-to-link-wasi-libc-with-shared-memory-flag, where you mentioned this error stems from the fact that the wasi-libc distribution isn't build with -pthread.

With #311 merged, does that mean it is now possible to build a wasi-libc with -pthread? If that is the case, is make THREAD_MODEL=posix all that is needed?

[turbolent]

Looks like make THREAD_MODEL=posix works, exciting :-)
I also just saw #326, which mentions pointing wasi-sdk to the thread-enabled sysroot.

However, when trying to compile a simple pthread test program, I get errors for many undefined symbols:

wasm-ld: error: .../sysroot/lib/wasm32-wasi/libc.a(pthread_create.o): undefined symbol: __lock
wasm-ld: error: .../sysroot/lib/wasm32-wasi/libc.a(pthread_create.o): undefined symbol: __unlock
wasm-ld: error: .../sysroot/lib/wasm32-wasi/libc.a(pthread_create.o): undefined symbol: __thread_list_lock
wasm-ld: error: .../sysroot/lib/wasm32-wasi/libc.a(pthread_create.o): undefined symbol: __thread_list_lock
wasm-ld: error: .../sysroot/lib/wasm32-wasi/libc.a(pthread_create.o): undefined symbol: __wasilibc_pthread_self
...

Do the thread-enabled sysroot's libraries assume that the user defines these?

cc @abrown

[abrown]

With #311 merged, does that mean it is now possible to build a wasi-libc with -pthread?

I don't think the "threads in wasi-libc" story is quite ready for use yet; there are still big pieces that need additional work. However, progress is being made: #311 is a start and the recent merging of #325 is an important step forward. Some of the pieces that still need work: finish compiling in the remainder of the pthread_*.c implementation in wasi-libc, fix some duplicate symbol definitions of __get_tp that were introduced at some point, #326, rust-lang/rust#102372, figure out some way to test all of this, etc.

Do the thread-enabled sysroot's libraries assume that the user defines these?

Probably not; I would guess that this is part of finishing out the compilation of more pthread_*.c files in the wasi-libc THREAD_MODEL=posix build. If you're interested in helping out with that, feel free to contact me on Zulip; otherwise, I think the best approach is to wait a bit longer for more parts to be finished.

[turbolent]

@abrown Thank you for the update, that helps understanding what to expect. That's really great work!

Over the weekend, I had done some research into how pthreads support could be added to https://github.com/turbolent/w2c2 and tried to piece together the puzzle of what the libc would need to do, how the runtime could implement it, etc. just to realize there actually is a WASI proposal now, and that an implementation is in progress. Exciting! I'm looking forward to implementing the wasi-threads proposal once the libc has support for it. Unfortunately I'm not really familiar with the implementation details on that side.