Element-wise operation of tensors

[VarLad]

I wanted to ask if there's a way to

1. Get element-wise multiplication of two tensors
2. Something like a map where I can apply an operation on all elements in a tensor.

[lkdvos]

Currently, there is not really a way to do this for generic tensors. The issue is that in the case of generic symmetries, not all elements are present/stored, as the tensors are kept in a very particular basis to ensure efficiency. Could you maybe elaborate on the kind of application you have in mind?

As an aside, if you are not dealing with symmetries, you can access the tensor data of a non-symmetric tensor as t[], which returns a view of the data. Thus, something like this would work:

using TensorKit
t = Tensor(zeros, ComplexSpace(3)^3)
t[] .+= 1 # note the dot-assignment
t

and similarly, you can map over this as you would map over a regular array, keeping in mind that it is a view, so you need in-place assignments to change the original tensor.

[VarLad]

Could you maybe elaborate on the kind of application you have in mind?

Nothing too specific, was trying to implement a Linear Attention based architecture in TensorKit.jl

Wouldn't adding a function like hadamard make sense which represents the hadamard product of tesors? 🤔
A hadamard product should be a valid operation right?

I am aware of the other way to do this, just wanted to see if there was a defined operation for stuff like this.
Is there any future possibility of a map operation for tensors, or is working with tensor[] the way to go? 🤔

[Jutho]

Do you expect any benefit of using TensorKit.jl for your application over regular multidimensional Array objects? If you only want the tensor contraction functionality, that is covered by TensorOperations.jl.

TensorKit.jl is specifically about defining a set of methods and functionality that makes sense and is consistent for a very general a broad definition of tensors, including tensors with various kinds of symmetries.

Elemementswise operations (including the hadamard product) or general higher order functions such as map are not of that type.

[lkdvos]

I guess the very basic thing we require is that any of the higher order functions that make sense for a tensor map need to at the very least be basis independent, as the specific basis in which our tensors are stored can be considered as an implementation detail. Also, it cannot change the zero values to nonzero ones, so even simple addition is quite restricted.
In that sense, the only time we are not picking an efficient basis for the tensor, and there are no structural zero values, is when you have no additional symmetry, in which case it is probably as convenient to just unpack the entries using tensor[].

From a library point of view, I think it's fair to say we want to promote that our users write symmetry-agnostic code as much as possible, such that once you write something that works for non-symmetric tensors, it automatically also works for including symmetries. In that sense, overloading things like map or defining hadamard conflicts a bit with that goal.

This being said, I think at the very least for Abelian symmetries a Hadamard product should still make sense, and you are of course free to define your own element-wise functions. Feel free to shoot a message if you are having any issues figuring this out, but you could efficiently access all stored elements in a tensor as values(blocks(tensor)), which will return an iterator over the nonzero blocks in the block-diagonal basis used to store these tensors.