[FEA]: Support a MetaTensor that does not hold data for debug purposes.

[Da1sypetals]

Is this a new feature, an improvement, or a change to existing functionality?

New Feature

How would you describe the priority of this feature request?

High

Please provide a clear description of problem this feature solves

Is it possible to support a MetaTensor that does not hold data for debug purposes?

Feature Description

We can utilize this to run the frontmost part of cuTile (python->hir->ir->TileIR) and either dump IR from each level for inspection or build tools like transpiler without having to create real data on GPU and even calling tileiras.

This comes from my personal experiments; (1) I have to develop on machines without Blackwell GPUs; (2) I have tried to implement a LSP server and cuTile -> numpy transpiler for debug purposes, but found it hard to catch up to the rapid development of cuTile mainstream.

Describe your ideal solution

If possible, I would like to discuss what the interface looks like and/or implement this.

Describe any alternatives you have considered

No response

Additional context

No response

Contributing Guidelines

• I agree to follow cuTile Python's contributing guidelines
• I have searched the open feature requests and have found no duplicates for this feature request

[gbonik]

Hi @Da1sypetals, thanks for you question. I believe it should be possible to implement a fake __cuda_array_interface__ to make this work (see https://numba.readthedocs.io/en/stable/cuda/cuda_array_interface.html ).

Here is an example that Works On My Machine (TM):

import cuda.tile as ct
from cuda.tile._cext import default_tile_context
from cuda.tile._compile import _get_final_ir

class FakeTensor:
    @property
    def __cuda_array_interface__(self):
        return dict(
            typestr="f4",  # float32
            shape=(100, 200),
            strides=None,  # C-contiguous format
            data=(0, False),
        )

@ct.kernel
def kernel(x):
    t = ct.gather(x, (0, 0))
    ct.scatter(x, (1, 1), t)

x = FakeTensor()
print(_get_final_ir(kernel._pyfunc, (x,), default_tile_context))

This still requires building the C++ extension, unfortunately. Please let us know if there are any gaps that we could address.
As for the rapid development, we hope the big disruptive changes will slow down soon, once we land certain features like lambdas etc.

[Da1sypetals]

Hi @Da1sypetals, thanks for you question. I believe it should be possible to implement a fake __cuda_array_interface__ to make this work (see https://numba.readthedocs.io/en/stable/cuda/cuda_array_interface.html ).

Here is an example that Works On My Machine (TM):

import cuda.tile as ct
from cuda.tile._cext import default_tile_context
from cuda.tile._compile import _get_final_ir

class FakeTensor:
    @property
    def __cuda_array_interface__(self):
        return dict(
            typestr="f4",  # float32
            shape=(100, 200),
            strides=None,  # C-contiguous format
            data=(0, False),
        )

@ct.kernel
def kernel(x):
    t = ct.gather(x, (0, 0))
    ct.scatter(x, (1, 1), t)

x = FakeTensor()
print(_get_final_ir(kernel._pyfunc, (x,), default_tile_context))

This still requires building the C++ extension, unfortunately. Please let us know if there are any gaps that we could address. As for the rapid development, we hope the big disruptive changes will slow down soon, once we land certain features like lambdas etc.

1. I have already tried __cuda_array_interface__, but it comes with limitations (as of my knowledge), the primary one is it does not seem to support bfloat16 or lower precision
2. Personally I would desire a solution without any C++ code... I have tried to workaround this in my personal repos, and would like to know if it is possible for the cuTile mainstream to include something like this. (line number may be wrong because this diff is based on a legacy commit)

diff --git a/src/cuda/tile/_ir/typing_support.py b/src/cuda/tile/_ir/typing_support.py
index 305b2c5..0de71b6 100644
--- a/src/cuda/tile/_ir/typing_support.py
+++ b/src/cuda/tile/_ir/typing_support.py
@@ -194,6 +194,12 @@ def typeof_pyval(val, kernel_arg: bool = False) -> Type:
         return TypeTy(val)
     if isinstance(val, Enum):
         return EnumTy(type(val))
+    
+    if hasattr(val, '__mock_tensor_metadata__'):
+        try:
+            return _type_handler['__mock_tensor_metadata__'](val)
+        except KeyError:
+            pass
 
     if hasattr(val, '__cuda_array_interface__'):
         try:
@@ -375,3 +381,56 @@ def from_cuda_array_interface(x: Any) -> ArrayTy:
                    base_ptr_div_by=arr_special.base_ptr_div_by,
                    stride_div_by=arr_special.stride_div_by,
                    shape_div_by=arr_special.shape_div_by)
+
+
+
+from cuda.tile._datatype import _enum_to_dtype, DTypeEnum
+
+
+dtype_map = {
+    "bool": DTypeEnum.B1,
+    "uint8": DTypeEnum.U8,
+    "uint16": DTypeEnum.U16,
+    "uint32": DTypeEnum.U32,
+    "uint64": DTypeEnum.U64,
+    "int8": DTypeEnum.I8,
+    "int16": DTypeEnum.I16,
+    "int32": DTypeEnum.I32,
+    "int64": DTypeEnum.I64,
+    "float32": DTypeEnum.F32,
+    "float64": DTypeEnum.F64,
+    "float16": DTypeEnum.F16,
+    "bfloat16": DTypeEnum.BF,
+    "tfloat32": DTypeEnum.TF32,
+    "float8_e4m3fn": DTypeEnum.F8E4M3FN,
+    "float8_e5m2": DTypeEnum.F8E5M2,
+}
+
+@register_type_handler('__mock_tensor_metadata__')
+def from_mock_tensor_metadata(x: Any) -> ArrayTy:
+    desc = x.__mock_tensor_metadata__
+    # Assume dynamic shape
+    shape = TupleTy(SizeTy(None) for i in desc['shape'])
+    dtype = _enum_to_dtype[dtype_map[desc['dtype_str']]]
+    if dtype.bitwidth % BYTE_BITWIDTH != 0:
+        raise ValueError("Only byte-aligned types should be supported by __mock_tensor_metadata__")
+
+    dtype_bytewidth = dtype.bitwidth // BYTE_BITWIDTH
+    byte_strides = desc.get('strides', None)
+    elem_strides = _compute_elem_strides(desc['shape'], dtype_bytewidth, byte_strides)
+
+    assert len(elem_strides) == len(shape)
+
+    arr_special = ArraySpecialization(
+            desc['data'][0], dtype.bitwidth, tuple(desc['shape']), elem_strides)
+
+    # Assume dynamic strides except for the ones marked static
+    strides = TupleTy(SizeTy(i) if static else SizeTy(None)
+                      for i, static in
+                      zip(elem_strides, arr_special.stride_is_static))
+
+    return ArrayTy(dtype, shape=shape, strides=strides,
+                   elements_disjoint=arr_special.elements_disjoint,
+                   base_ptr_div_by=arr_special.base_ptr_div_by,
+                   stride_div_by=arr_special.stride_div_by,
+                   shape_div_by=arr_special.shape_div_by)

(Added: _cext.py to mock the missing lib, this part is vibe coded)

from cuda.tile._context import TileContextConfig


def launch(stream, grid, kernel, kernel_args, /):
    """Mock launch function - does not actually launch kernel"""
    raise RuntimeError(
        "Cannot launch kernel: CUDA driver not available. This mock module only supports IR generation."
    )


class TileDispatcher:
    """Mock TileDispatcher class"""

    def __new__(cls, *args, **kwargs):
        # 正确处理 __new__ 方法
        instance = object.__new__(cls)
        return instance

    def __init__(self, arg_constant_flags, compile_func):
        self._arg_constant_flags = arg_constant_flags
        self._compile_func = compile_func


class TileContext:
    """Mock TileContext class"""

    def __init__(self, config=None):
        if config is None:
            from cuda.tile._context import init_context_config_from_env

            config = init_context_config_from_env()
        self._config = config

    @property
    def config(self):
        return self._config


# Create default tile context
default_tile_context = TileContext()


def get_compute_capability():
    """Mock get_compute_capability - returns a default value"""
    # Return Hopper (9.0) as default
    # Users can modify this if needed
    return (9, 0)


class ArraySpecialization:
    """Mock ArraySpecialization class for array type analysis"""

    def __init__(self, base_ptr, dtype_bitwidth, shape, strides):
        self.base_ptr = base_ptr
        self.dtype_bitwidth = dtype_bitwidth
        self.shape = shape
        self.strides = strides

        # 简单的启发式规则：假设所有 stride 都是静态的
        self.stride_is_static = tuple(True for _ in strides)

        # 假设元素不重叠
        self.elements_disjoint = True

        # 对齐信息：假设基本对齐
        self.base_ptr_div_by = dtype_bitwidth // 8  # 按字节对齐
        self.stride_div_by = tuple(1 for _ in strides)
        self.shape_div_by = tuple(1 for _ in shape)

3. Currently get_final_ir depend on TileContext (defined in C++ code?) which seems not referenced based on a simple code search. Is it possible to get rid of it, or does it serve some undocumented purposes? Could you please explain? Thanks a lot. If the _get_final_ir function can be independent on C++ code, can we split the _get_final_ir function into 2 halves (first pure python half, then python + C++ half) and expose a public interface for the first half?
4. Sorry for my bad English, if there is anything not clear please ask for my clarification.

[Da1sypetals]

Since we only need tensor metadata to compile the cuTile kernel (without tensor data), is it possible to first construct such metadata into a dedicated dataclass, compile the kernel, and launch the kernel with real data? This could make the FakeTensor workflow not an exception, but a "part of the main compiler workflow".

[gbonik]

Since we only need tensor metadata to compile the cuTile kernel (without tensor data), is it possible to first construct such metadata into a dedicated dataclass, compile the kernel, and launch the kernel with real data? This could make the FakeTensor workflow not an exception, but a "part of the main compiler workflow".

Yes, something along those lines is on our TODO list. Right now there is duplicated logic between the C++ dispatcher code (classify_arg() and friends) and the Python side (typeof_pyval etc.), which is very fragile. It would be great if the C++ dispatcher decided what the types are and passed them to the compilation pipeline. This would also be very useful for supporting ahead-of-time compilation. However, I don't know when we can get to it. I'll try to prioritize this when I have some time free from scheduled feature work.

I have already tried cuda_array_interface, but it comes with limitations (as of my knowledge), the primary one is it does not seem to support bfloat16

I believe bfloat16 is supported (typestring V2, see parse_typestr() in tile_kernel.cpp), but yes, I agree it is a limited interface, e.g. it is not possible to express float8 types.

Currently get_final_ir depend on TileContext (defined in C++ code?) which seems not referenced based on a simple code search. Is it possible to get rid of it, or does it serve some undocumented purposes? Could you please explain? Thanks a lot. If the _get_final_ir function can be independent on C++ code, can we split the _get_final_ir function into 2 halves (first pure python half, then python + C++ half) and expose a public interface for the first half?

The goal of TileContext is to store all global state, like the flags set by environment variables etc. This is still work in progress, but it is already used in hir2ir in order to check the debug printing settings (via ir_ctx.tile_ctx). The idea is that if another library needs to depend on cutile, it can create an isolated context that is unaffected by environment variables etc. We need to access the TileContext object in the C++ dispatcher, so we can't move it to Python for performance reasons. Is there a specific reason you want to avoid building the C++ extension? It has very minimal dependencies and should be relatively simple to build. If you need to build it on a Mac, or don't want to install the CUDA toolkit, we could in theory add some conditional compilation to build a "dummy" extension with stubbed out functionality (this is also on my personal TODO list).

As an alternative , I think maybe we could change _get_final_ir to accept a TileContextConfig instead of the full TileContext object, which is a pure python class.

I have tried to workaround this in my personal repos, and would like to know if it is possible for the cuTile mainstream to include something like this.

I'd be in favor of adding a more explicit compilation API, e.g. going the "part of the main compiler workflow" route. Also, once your LSP server reaches a usable state, would you be interested in upstreaming it? This could help us deal with the internal API breakages, as we could test it in the CI. It doesn't have to be part of the main package, as we can put it into the experimental package or similar.

[Da1sypetals]

Also, once your LSP server reaches a usable state

It is already usable in my sense, and it is here, but it is based on commit 7a8625 (before the introduction of hir), and it is probably buggy. Its type information comes from the ir after typecheck but before any optimization. There is also a to-numpy transpiler in transpile branch, which is also based on a legacy commit.

going the "part of the main compiler workflow" route

My naive idea can be illustrated in this flowchart

Is there a specific reason you want to avoid building the C++ extension? 

The main reason is to avoid building a C++ extension just for a debug tool where we actually don't need to use anything inside it, to Make it possible to do this level of debugging on computers without NVIDIA hardware or even Mac.

As an alternative , I think maybe we could change _get_final_ir to accept a TileContextConfig instead of the full TileContext object, which is a pure python class.

This would be great! As far as I know, syntax checking and type checking is done inside _get_final_ir, which means this approach makes it possible to raise any possible errors when converting a python program into a valid cuTile-python program without C++ libraries. I think this is pretty much everything needed to develop debug tools.

[gbonik]

The main reason is to avoid building a C++ extension just for a debug tool where we actually don't need to use anything inside it, to Make it possible to do this level of debugging on computers without NVIDIA hardware or even Mac.

You shouldn't need NVIDIA hardware to compile the C++ extension, a normal C++ compiler should be sufficient (we support GCC and MSVC, hopefully clang should work as well). I agree that it would be nice to drive the compilation pipeline with pure Python, but I'm afraid this may not be viable in the long term. The C++ extension needs to have the logic similar to typeof_pyval() to be able to quickly classify the arguments, and thus we'd need to duplicate the logic in Python (which we currently do, but ideally we need to get rid of this duplication).

In any case, I have staged a change to _get_final_ir that should hopefully address your use case:

def _get_final_ir(pyfunc,
                  args: Sequence[ir.KernelArgument],
                  config: TileContextConfig) -> ir.Function:

1. Now we take a TileContextConfig instead of TileContext. It can be obtained by calling init_context_config_from_env().
2. Instead of args, we take a sequence of KernelArgument structs, which is a renamed ir.Argument:

@dataclass
class KernelArgument:
    type: Type
    is_const: bool = False
    const_value: Any = None

I hope this should allow you to drive the compilation pipeline without creating concrete array objects. The KernelArgument class should work similarly to a "meta tensor", with the downside of having to also call get_constant_annotations() etc. yourself.

The change is stacked on top of a couple other in-progress PRs, so it should hopefully get into public main sometime next week.

[Da1sypetals]

I have synced the cutile to numpy transpiler to newest main branch (it's here, and it's definitely buggy), as a proof of concept that everything it need is the output of _get_final_ir