Convert mathjs to Typescript

[husayt]

Hi Jos,
Just wanted to ask your opinion regarding switching matjhs to Typescript? Obviously there will be a cost to this, but when it's done it will be a win for JS users as well as for TS users. Also considering the size of the library, that will help to grow it further with minimal errors, as TS helps weeding out lots of nasty bugs.

If this is something you feel positive about, I will be happy to help.

[EricCrosson]

I've been looking through the issues to determine the current state of TypeScript support. I see pointers to the @types/mathjs definitions on DefinitelyTyped, but the latest version of mathjs with type definitions is major version 6, 2 major versions behind and nearly a year out of date.

I agree with @husayt, an implementation in TypeScript would provide numerous benefits:

• less chance of introducing bugs or regressions
• would clean up the third-party type definitions (pretty wonky to work around currently, numerous type unsafe type annotations are required for the overloaded functions)
• net increase in development speed
• easier for new contributors to get up to speed
• types would never be this out-of-sync with the underlying implementation again
• boost in quality of tooling to assist with refactors

Relevant issue re out-of-date DefinitelyTyped definitions: #1539

[josdejong]

Thanks for your suggestion @husayt. I'm positive about looking into converting the code base to TypeScript.

I expect that this will not be easy. There is quite some special stuff going on with dynamically creating functions. mathjs has these typed-functions which define (runtime) types for example, and generates index files based on factory functions etc. I can imagine getting all types right can be hard, and may become verbose, forcing you to write types twice and ending up with unwieldy generics to make stuff work. Maybe we want/need to make changes in the architecture to make mathjs play nice with TypeScript.

I think it would be good to first investigate what is required to get TypeScript into the code base, and then decide on if and to what extend we would like to integrate TypeScript. TypeScript can add value, but it also comes at a cost.

A pragmatic short term solution of course is to get the type definitions of @types/mathjs on par.

Anyone interested in picking this up?

[husayt]

@josdejong I am happy to start with this. Yes, it will take time to switch fully to Typescript. But the good thing with TS is that this can be done gradually and without breaking everything. As JS is TS we can start with very loose tsconfig and then try to introduce types, and as we are we are doing that we can switch to more strict settings. This will happen without breaking things. If there are any breaking changes required we can leave them to the end. Then we can decide if they should come in v9.x or not. This way most of the changes can be done as part of v8 i.e. nonbreaking changes. If you are happy with this approach I can do the first step.

[harrysarson]

If I may chime in, there is value in not needing a build script. It looks like we do still use build scripts to convert the files in src to files in lib but if I understand correctly this is a temporary measure whilst we work out how to get ESM working properly in node.

A nice feature of some javascript libraries that we should (I believe) aim to support is that a library can beimported "as is" and transpilation is a extra step if you need the library to run in an old browser (babel inserts shims for IE11 for example) or want to optimise (download size etc).

[josdejong]

Thanks @husayt! Yes it's very nice that we can implementing TypeScript gradually. But let's first do an experiment to investigate all the bears we will encounter down the road and have a clear view on how the code base will look like in the end. Then make a clear plan, before we start for real.

A first small, pragmatic step I think would be to improve the existing type definitions. That will yield a lot of value with very little effort.

@harrysarson I think we cannot get rid of the build tools any time soon. Part of it is to build ESM, CommonJs, and bundle output, but an other part is to generate index files (see the src/generated/* files). I'm looking forward to just having ESM though :).

[jeremylorino]

@josdejong looks like help is required here. if yall are moving forward with the convert lmk

[josdejong]

@jeremylorino yes help would be very welcome.

[jeremylorino]

@jeremylorino yes help would be very welcome.

@josdejong is there a project/branch/plan for this or has work not started? Don't want to redo work.

[josdejong]

Thanks @jeremylorino for your offer. There is nothing yet in that direction. I think it will be a large undertaking. It could very well be that we need to make some changes in the architecture in case it doesn't play nice with TypeScript, so it would be good to first create a minimalistic proof of concept to figure out those issues.

FYI: it's good to be aware of an other (early) experiment/idea regarding the architecture of the library discussed in #1975 which potentially would mean big changes in the architecture.

[josdejong]

Continuation of the discussion here: #2582 (comment)

In order to move mathjs to TypeScript the first step would be to add TypeScript support to typed-function.

We would like to see if we can tackle two challenges in one go though: the current architecture is relatively complex, and we miss TypeScript support. @gwhitney had some great ideas regarding the architecture and created a small POC named picomath, see josdejong/typed-function#138 (comment). I think a good starting point would be to try create a POC with picomath + typed-function + TypeScript. It may be that the boundaries between picomath and typed-function will change or event disappear. I'm not sure about that yet but want to keep an open mind and think out of the box in this regard.

[gwhitney]

Three major issues I see that would need to be explored right away in a proof-of-concept for Typescript along the lines Jos just suggested:

1. Naively, we would want the type of fn in

const fn = typed(n:number => 'Hello') // Can avoid signature keys, since TypeScript knows the arg (and value) types

to be (number) => string, but we would want the type of fn in

typed.addConversion(s:string => 0) // no need for a struct, TypeScript knows the from and to types just from conversion function
const fn = typed(n:number => 'Hello')

to be (string|number) => string. Is that feasible in TypeScript? It seems to mean that the type of typed changes when typed.addConversion is called... is that a possible thing? I'd suggest that either a proof of concept of this sort of thing happening or a proposal for how else automatic conversions might be handled, would have to be an early subgoal for TypeScript conversion. (I don't think mathjs wants to give up automatic conversions altogether for the sake of converting to TypeScript; if anything, more automatic conversions have been cropping up in mathjs lately.)

2. The entire picomath proof-of-concept was predicated on constant mutable function objects that could acquire new additional behaviors. As mentioned in Proposal: provide mutating operations on typed functions typed-function#138, the naive first implementation of such a thing produces something like a factor of two slowdown in executing typed functions -- clearly unacceptable. I really like the basic outlines on the picomath approach (unsurprisingly ;-) but as far as I can see that approach is on ice unless/until someone comes up with a mutable function implementation in typed-function that can support it with acceptable speed. I have one idea -- basically put as much of the current specialized implementation into the immutable function body as possible -- but I am dubious as to how much it will improve on the slowdown and have been waiting to try it until I can manage the time to help get mathjs v11 out.

3. Even if we can make (2) fast enough in JavaScript, adding new implementations to an existing typed function seems to modify its type, creating another issue very much like (1). Is that feasible? Possibly a solution to (1) would suggest a solution to this as well, but I am not enough of a TypeScript jockey to really have any idea how to do (1) or this.

[jeremylorino]

Thanks @jeremylorino for your offer....

@josdejong @gwhitney let me get caught up. It's been a while since I've looked at the code. I'll follow up soon.

[josdejong]

No problem at all. Please keep us posted if you try out some experiments with this. It's a big undertaking.

[jeremylorino]

@gwhitney

Naively, we would want the type of fn in

const fn = typed((n: number) => 'Hello') // (n: number) => string

// no need for a struct
// TypeScript knows the from and to types just from conversion function
typed.addConversion((s: string) => 0)

// but we would want the type of `fn`
// to be (string|number) => string
const fn = typed(n:number => 'Hello')

Is that feasible in TypeScript?

Short answer, no. Something like this would have to take place TS Playground

const fn1 = typed((n: number) => 'Hello');
fn1(1);

const typedConversion = typed.addConversion((s: string) => 0);
const fn2 = typedConversion(n => 'Hello');
fn2(1);
fn2("2");
fn2(false); // Argument of type 'boolean' is not assignable to parameter of type 'string | number'.

I know this is only the first issue in the set you mentioned; yet it seemed like the most important.

So now I have to ask, why do we need typed-function? (Is this a package that could be deprecated with Typescript?)

I know it provides runtime "type" checking

That should help us move the needle.

[jeremylorino]

@josdejong FYI typed-function#89 is pretty close.

I pulled it down and got it to work on construction.test.ts with some type modifications.

"Simple" lib; complex typings lol

[gwhitney]

So now I have to ask, why do we need typed-function? (Is this a package that could be deprecated with Typescript?)

Well, I am pretty sure we want runtime type dispatch, i.e. different behaviors for a function depending on its input type. That's not something typescript provides out of the box. It doesn't have to be a rewrite of typed-function that provides type dispatch, it could be another facility, but I feel like it's needed at the core of math.js/ts.

I like your idea that adding a conversion returns a new function factory that types properly with the conversion, but don't quite see how the library can then offer up the latest greatest factory with all of the conversions added by any module... it seems like the charm of mathjs/typed-function right now is that conversions are added one place and then used elsewhere without the client having to worry about what conversions are in effect; and for customization, more can be added later. It feels like in TypeScript something in that model will have to be relinquished. But maybe there's a clever enough architecture to keep all those characteristics.

[jeremylorino]

Ok that helps a lot. Let's start by defining what features
of typed-function are required by mathjs.

typed-function	mathjs	comments
Runtime type dispatch - different behaviors for a function depending on its input type.	✅	Can you show me a good example of how this is being used in mathjs.

---

...but don't quite see how the
library can then offer up the latest greatest factory

...

more can be added later.
It feels like in TypeScript something in that model...

I need some examples in mathjs where this matters.

Typescript has some tricks up its sleeves but there are
going to be instances where we have to change how
we approach a problem if we actually want
the benefits of Typescript.

[gwhitney]

how [runtime type dispatch] is being used in mathjs

Every operator of mathjs is defined "piecewise" by giving its behavior on different input types. The code for a complex input typically has nothing at all to do with the code for a real matrix input, say. This is a very valuable modularity in the design of mathjs. If I am defining the "absolute value" for a quaternion, for example, I don't even have to think about how it's computed for a bigint. But on the other hand, when I am using mathjs, I can simply call the abs() operator on any entity I might have from any type that mathjs supports, and it will do what I want. That's what I mean by runtime type dispatch, and as far I can see it's absolutely baked in as a presuppostion of mathjs. Did this make things any clearer? I was a little confused by the question actually...

[gwhitney]

more can be added later.
It feels like in TypeScript something in that model...

I need some examples in mathjs where this matters.

Well, again, t think this is more about how mathjs is used, not how it's implemented. For example, one of the things that attracted me to mathjs is that we will need a data type that represents (finite or infinite "to the right") sequences of whole numbers. We don't expect such a type to be built in to a JavaScript CAS, because its a bit specialized, but it's very attractive to have a CAS in which we can load up the base system, add our type and some operations on it, and then use entities of the new type alternating with pre-existing types, etc -- they all just work together.

Is this helping any with clarifying things?

Sure, it would be great if for my new sequence type the compiler can barf if I add a number to a sequence (which should add it elementwise) but then try to compute its multiplicative inverse (which would be defined for a number but doesn't make sense for a sequence). But it's more important that it be able to select and execute the right behavior for a scalar plus a sequence in the first place.

[jeremylorino]

@gwhitney awesome!

more about how mathjs is used, not how it's implemented

This helps me determine my approach.
Let me digest and I'll get back.

[gwhitney]

The discussion in josdejong/typed-function#89 is probably also relevant

[josdejong]

Referencing josdejong/typed-function#123 here. There are some interesting POC's there, though we're not there yet.

Much of the discussion of converting mathjs to Typescript boils down to first having full TypeScript support in typed-function. That is complicated because like Glen explains, mathjs/typed-function currently is working via dynamic, runtime merging of functions (and enriching them with automatic type conversion), and that bites static typing. We'll need to keep thinking out of the box, maybe we can accept certain limitations if that makes it possible to become statically typed. Just thinking aloud here.

[gwhitney]

Note that the Pocomath proof of concept on offer for #1975 now allows return-type annotation, which should at least in some sense be a step closer to TypeScript.

[josdejong]

Ah, I see: https://code.studioinfinity.org/glen/pocomath/commit/31add66f4cc1f5768c8e0697214dbcd624754bf0

That is nice, indeed one step closer towards TypeScript 😎

[gwhitney]

Folks interested in this discussion should probably be aware of this other apparently completely independent effort to produce a TypeScript CAS: https://github.com/cortex-js/compute-engine. It seems quite new, and has a much less extensive list of implemented operations at the moment, and doesn't seem to state type extensibility as one of its explicit design goals, but still it's more of TypeScript CAS than mathjs is at the moment ;-)

[gwhitney]

Having nosed about CortexJS just a bit more, I will say that CortexJS seems much more heavily symbolic than mathjs, and primarily deals with functions on symbolic entities, which are basically ASTs or s-exprs from good ol' lisp. There are some numerical evaluation functions. As such, I think it presents a much more tractable case of TypeScript typings: most operations take symbolic expressions to symbolic expressions, and mostly just the numerical evaluators deal with other types, and I think they mostly just take symbolic expressions to those other types.

[josdejong]

That looks interesting 😎