Cranelift JITModule and multithreading

[mokhaled2992]

Hi everyone,

This is asking for help/advice. I have JIT struct very similar to the one in the toy demo. I only generate code that builds a wrapper function that only calls a rust function. Mainly following those examples; jit-minimal and cranelift-call rust function. Now I instantiate one single instance for each thread from that JIT struct and use each independently only in their own thread to call the function through the pointer I get from that respective JITModule instance. The problem is that I'm getting segmentation faults randomly (as in sometimes they happen sometimes not) when I try to call the functions through the returned pointers and this happens only in multi threaded testing, I never had this issue a single time when I tested my code in single threaded environment. Are there any limitations to using the JITModule in multithreaded environments or any common pitfalls that I should watch for? If needed, I can try to write a minimal example (although wouldn't be very different from the samples above).

I even tried without calling free_memory at all and still the same problem (only when running in multiple threads as mentioned)

Random guessing, Is there anything shared across JITModules? Perhaps something related to those relocation tables?

[bjorn3]

Nothing is shared between JITModule's. Instantiating a single JITModule per thread should be just fine. Have you tried getting a backtrace and referenced address for the crashes? Two things I could think of is that either somehow two memory mappings got more than 2GB apart from each without causing an immediate panic, or the memory allocator can't handle the fact that we change memory protection for allocated memory in multi threaded scenarios. Could you try enabling the selinux-fix feature to see if that fixes it? This switches from malloc to mmap.

[mokhaled2992]

Thanks for getting back so quickly! I will try this feature. I thought about panics but shouldn't I see this before invoking the function pointer? I mean at the steps of compiling/loading? Regarding the addresses, here is what I did already, I print the address I get from get_finalized_function, attach the debugger, get bt from the thread that received the signal. The last address in the bt is the same as the printed one.

[bjorn3]

The panic I was talking about should happen during finalize_definitions. If it didn't happen at that point, it won't happen at all.

Can you check if the memory at the the function address is mapped as r-x and not as rw-? You can check the /proc/<pid of crashing process>/maps file while the debugger is attached.

[mokhaled2992]

Yea verifying the permissions sounds like something worth checking. So are you suggesting that somehow setting the proper permissions fails silently? That would be really strange given that the segfault happens randomly, sometimes everything just works out in all threads. But I will check that, I already started a build with selinux-fix.

[bjorn3]

I think incorrect premissions is the most likely issue. If selinux-fix fixes it, incorrect permissions is almost definitively the issue. Not quite sure why that would happen without selinux-fix though.

[mokhaled2992]

@bjorn3 Thanks man ❤️. The selinux-fix did the job. Perhaps a note somewhere in the demo or in the README about this feature flag would be great and saves alot of frustration :)

[bjorn3]

Opened #4986.