Generated IR is not portable

[ptillet]

libdevice uses absolute path for the bytecode location. This is not portable. Not sure how to fix this, maybe libdevice location can be passed as an argument for compilation, rather than baked into the IR?

[Jokeren]

Yeah, that sounds like a good solution

[yuguo68]

We hit a libdevice.10.bc not found compile/runtime error from https://github.com/openai/triton/blob/972b761390c45bcb0d0091741067603b2ab2ed42/lib/Target/LLVMIR/LLVMIRTranslation.cpp#L126-L133.

Can we mitigate via a env variable for the libdevice.10.bc path?

[Jokeren]

Just curious, if this path does not exist, what's the actual path you get?

[ptillet]

I'm definitely surprised. Unless you share the IR (but not the cubin) between different machines then you shouldn't run into the issue

[yuguo68]

I believe it is related to our build system and maybe remote execution. The path is correct for local machine but failed to load at runtime.

[ptillet]

This is strange indeed. Does that mean that the TTGIR is generated on one machine, but then converted to PTX on another machine? Can I see the full stack trace of the error?

[yuguo68]

Unfortunately the stack trace only contains one line complaining missing libdevice.10.bc. _build(name, src, srcdir) ie executed remotely. https://github.com/openai/triton/blob/a13ddf08e2984d3d734f242e4bea589c3fbbbca4/python/triton/compiler.py#L1366

And using an env variable for the path does work.

[yuguo68]

The error should be from kernel compilation. Tried it again the full error stack is

Failed to load triton/python/triton/language/libdevice.10.bcTranslate to LLVM IR failedLLVM ERROR: Failed to translate TritonGPU to LLVM IR.

Looks like no space between the sentences.

[ptillet]

I don't think an environment variable would be a good workaround. The fundamental issue here seems to be that path is a relative path, so this is bound to create problems indeed.

Maybe something better would be for the frontend to copy libdevice.10.bc into the cache directory and have the IR contain the absolute path to libdevice.10.bc in the cache. Still not ideal, but that should be more fleixble than the current approach without requiring the use of an environment variable.

[yuguo68]

Maybe our use case is a bit special since we may compile kernels on remote machines which does not have triton source code. We already use https://github.com/openai/triton/blob/a13ddf08e2984d3d734f242e4bea589c3fbbbca4/python/triton/compiler.py#L1066
for ptxas path.

[ptillet]

I see. Ok then, feel free to submit a PR that adds an TRITON_LIBDEVICE_PATH env var for now.

[malfet]

How about using dladdr to get path to the shared object and then look for library relative to it

[malfet]

I hope fixed by #1176