Idea for extending the api

[contactomorph]

I'm the creator of an experimental rust library called tensorism. It is unrelated to ndarray but it is about the same type of data: multi-indexes arrays (aka tensors, even if this term is technically incorrect). In creating such a library my purpose was to experiment on the API side of tensors: trying to define a convenient and expressive DSL. I think on this aspect the result is a success. Now I'm aware that in the crate ecosystem, there are already many mutually incompatible, half finished, mostly-abandonned libraries on many topics, this one included. So instead of proposing yet another incompatible set of types and abstractions for multi-indexed arrays, I'm considering to contribute to an existing library and propose similar ideas. As far as I can tell narray is one of the most popular. I could of course just propose a separate crate on top of ndarray (and maybe I'll do that). However I would also like to get the feedback from ndarray devs about my ideas and also ask them if they could be interested in them.

To go into details, the main idea of tensorism is to propose a DSL based on indexes which is both expressive and yet still retains good performance. This is mainly due to a macro (makes!) like for example in:

let z = make!(i j ~ sin(b[i, j]) + b[j, i] - sum(k ~ a[j, k, i]));

which is equivalent to

$$\forall i, j, \quad z[i, j] = \sin(b[i, j]) + b[j, i] - \sum_k a[j, k, i]$$

Inside macro make! character ~ is a special character. Any expression of the form ⟨index_list⟩ ~ ⟨expr⟩ can be understood as evaluate ⟨expr⟩ for every possible integer values of indexes inside ⟨index_list⟩. If such an expression is directly used as the macro parameter, the result is a tensor with as many indexes as provided inside ⟨index_list⟩. If it is a nested expression, it is evaluated as an iterable for all possible values of indexes inside ⟨index_list⟩.

The advantage of such syntax is according to me that

• it naturally handles index permutations and "axis" logic
• all aggregation operations (sum, product, min, max, quantiles, ...) can be externalised in the sense that the crate does not have to know them or even to know on which types they are applied. On the other hand, higher order methods require specific and exhausive implementations provided by the crate.
• the macro adds boundary checks once at the beginning and can then iterate inside using fast unsafe accesses while still offering a safe external API from the point of view of the user.

My point is that a similar API could totally be provided by ndarray. Any feedback appreciated.

[akern40]

Hey @contactomorph! Always happy to see suggestions and conversation about new features. This DSL seems very similar to Einstein Notation, although it appears to incorporate additional functions such as sin. Einstein notation was worked on in oracleofnj/einsum, although that last had a commit 3 years ago (although may still work just as well! You never know). I'm personally of the opinion that this sort of DSL is perfect for its own crate that sits on top of ndarray - one could argue that ndarray already has too many functions, and could even itself be split into multiple crates.

[contactomorph]

Hey @akern40

Thank you for your feedback.

This DSL seems very similar to Einstein Notation, although it appears to incorporate additional functions such as sin. Einstein notation was worked on in oracleofnj/einsum

The term Einstein notation normally only refers to a very limited convention (the fact that an index repeated twice implies an implicit sum). The use of indexes to express tensor calculus is older and due to Ricci and Levi-Civita. But anyway, I think that trying to conform to tensor calculus / Einstein notation is misled, except obiously if you are only dealing with real tensors in the mathematical sense. Tensorial operations are very specific (and for example component-wise multiplication absolutely does not make sense in the context of tensors). I believe that any DSL that using indexes to manipulate multi-index arrays should cover far more operations.

I'm personally of the opinion that this sort of DSL is perfect for its own crate that sits on top of ndarray - one could argue that ndarray already has too many functions, and could even itself be split into multiple crates.

I have sympathy for the viewpoint that more powerful features may belong to separate crates. However I think that one of the reasons why ndarray has too many functions is precisely because it provides higher order methods. This is an endless task and according to me a questionable goal. Note that in tensorism, all these kind of data transformations are external. Contrary to what you suppose in your comment, it does not incorporate additional functions such as sin. It just allows the users to apply external operations (such as sin) to every items. Technically in tensorism even operators like addition are external operations!

[akern40]

Ah ok ya I see what you mean - I guess Rust's macro system would let you embed arbitrary functions from the outer scope into the macro, allowing you to have compile-time support for this DSL when other languages struggle with this. I think that's an incredibly powerful feature, if it can be engineered efficiently and designed ergonomically!

I do stand by my assertion that this is something to go in a different crate. I agree that some of the higher-order methods in ndarray may be extraneous, and certainly we should figure out where the line is on adding new methods. I have some thoughts there that I'm not quite ready to talk about yet, but suffice to say I'm probably in favor of some deprecation or splitting down the line.

That being said, I am on a mission to make the ndarray ecosystem stronger. Part of this is ensuring that ndarray is communicating with, supporting, and showcasing awesome crates that build on top of its capabilities. I would like this to be part of our README and, eventually, part of a website. When tensorism (or wherever this DSL might live) gets to a point of maturity, I would love to broadcast that to our users in public ways. Would you be open to that idea?