Support for other ops on MXTensor #3483
Replies: 4 comments 6 replies
-
|
@danielvegamyhre @vkuzo |
Beta Was this translation helpful? Give feedback.
-
|
hi there, sorry for missing this, we monitor issues but didn't know about the github discussions feature.
I think it's ok to add ops on |
Beta Was this translation helpful? Give feedback.
-
|
@vkuzo , you are right that we can infer the contraction dim using scale.shape and block size, but I think that would lead to unnecessary if-else, I think it might be better to add an attribute for contraction dim. I think for the quantization function, the contraction dim in the API makes sense. Want to know your thoughts. |
Beta Was this translation helpful? Give feedback.
-
|
One thing to consider is square blocked formats such as 32x32 blocks, which we plan to support in the near future. There isn't a single contraction dim in that case. |
Beta Was this translation helpful? Give feedback.
Uh oh!
There was an error while loading. Please reload this page.
-
Hi,
Currently there is support for some ops when using a MXTensor, however its still in the early phases I think, for e.g. if I try to use the operations like concat, chunk on a MXTensor, then it would fail.
I thought about their implementations and I think that we might need an additional attribute to MXTensor i.e. a
dimattribute. The motivation for this attribute is to convey which dimension is the contraction dimension (the dimension to be quantized), currently its assumed that -1 (the last dim) is the quantization dimension. However, in some cases e.g. transpose, that might not hold true and if we want to perform an operation like chunk, then the contraction dimension must be known and there are additional restrictions for some ops, an example is shown below.Want to know your thoughts.
Beta Was this translation helpful? Give feedback.
All reactions