glMatrix v4.0 - Request for feedback

[toji]

glMatrix was a project I started 12 years ago(!), because at the time there weren't too many good options for doing the type of matrix math realtime 3D apps required in JavaScript. I never thought that anyone outside of myself and a few early WebGL developers would use it, and certainly didn't anticipate it becoming as popular as it did.

Fortunately for everyone, the landscape for 3D on the web looks a lot different today than it did over a decade ago! There's a lot of great libraries that offer comprehensive tools for creating realtime 3D web apps, usually with their own very competent vector and matrix capabilities built in. Many of these offer features or syntax niceties that glMatrix hasn't been able to match due to it's history and design ethos.

The web itself has also evolved in that time. When I published the first version of glMatrix Chrome was on version 5, Firefox was at version 3, and Internet Explorer was still a thing developers cared about. Node.js and TypeScript hadn't even been released! We've made astronomical strides in terms of the capabilities of the Web platform since then. For example, none of the following existed (or at the very least were widespread) at the time glMatrix was first developed:

• let and const
• Arrow functions ((x) => { return x; })
• JavaScript classes
  • which includes Getters and Setters
• JavaScript Modules
• Template literals
• Spread syntax (someFunction(...args);)
• Even Float32Array wasn't around until shortly after!
• Oh, and WebGL itself was still in development

Over the years glMatrix has been updated in small ways to take advantage of some of these things, it hasn't strayed too far from it's original design. But these days we can do so much better, and despite the excellent competition that I strongly believe there's still a need for a solid, standalone vector and matrix math library.

I've had a bunch of ideas floating around in my head for how glMatrix could be updated to take advantage of the modern web platform, but haven't had an opportunity to work on it till recently. (Let's be honest, the library has needed some maintenence for a while now.) Now that I've had a chance to try some of it out, though, I feel pretty confident that it's a viable direction for the future of the API.

So let me walk you through my plans for a glMatrix 4.0! Feedback highly appreciated!

Backwards compatibility first and foremost

glMatrix has a lot of users, and they have a lot of carefully written algorithms using the library. It would be unrealistic to expect them to do complete re-writes of their code base just to take advantage of a nicer code pattern. So the first principle of any update is that backwards compatibility is always priority number one.

This doesn't mean that EVERYTHING is carried forward, mind you. I think it's appropriate for some functionality to be labeled as deprecated and for some lesser used, fairly awkward bit of the library to be dropped. (I'm looking at you, weird forEach experiment that never quite worked the way I wanted.)

But the majority of the library should be able to be used with minimal or no changes to existing code, and new features should cleanly layer on top of that existing code rather than requiring developers to make a wholesale switch from the "old way" to the "new way".

Lots of ease-of-use improvements

glMatrix was designed for efficiency, but that left a lot to be desired in terms of ease-of-use. There's only so much that JavaScript allows in terms of cleaning up the syntax, but with some slightly unorthodox tricks and taking advantage of modern language features we can improve things quite a bit.

Constructors

The current syntax for creating vector and matrix objects isn't ideal (I'll be using vec3 for examples, but everything here applied to each type in the library):

let v1 = vec3.create();
let v2 = vec3.fromValues(1, 2, 3);

We'd much rather use the familiar JavaScript new operator. Turns out we can without losing any backwards compatibility simply by declaring our class to extend Float32Array!

export class Vec3 extends Float32Array {
  constructor(...values) {
    switch(values.length) {
      case 3: super(values); break;
      case 2:
      case 1: super(values[0], value[1] ?? 0, 3); break;
      default: super(3); break;
    }
  }
}

This allows us to use a few variants of a typical constructor.

let v1 = new Vec3(); // Creates a vector with value (0, 0, 0)
let v2 = new Vec3(1, 2, 3); // Creates a vector with value (1, 2, 3)
let arrayBuffer = new ArrayBuffer(32);
let v3 = new Vec3(arrayBuffer); // Creates a vector mapped to offset 0 of arrayBuffer
let v4 = new Vec3(arrayBuffer, 16); // Creates a vector mapped to offset 16 of arrayBuffer
let v5 = new Vec3(v2); // Creates a copy of v2

It's pretty flexible, and not that complex to implement! And of course because Vec3 is still a Float32Array under the hood, you can pass it into WebGL/WebGPU (or any other API that expects Typed arrays or array-like objects) with no conversion.

gl.uniform3fv(lightPosition, v5);

Static methods

For backwards compatibility we'll keep around vec3.create() and friends, but have them return instances of the new Vec3 class instead of raw typed arrays. In order to keep everything together, they'll become static methods on the Vec3 class. Same goes for every other existing method for a given type.

export class Vec3 extends Float32Array {
  static create() { return new Vec3(); }
  static fromValues(x, y, z) { return new Vec3(x, y, z); }

  static add(out, a, b) {
    out[0] = a[0] + b[0];
    out[1] = a[1] + b[1];
    out[2] = a[2] + b[2];
    return out;
  }

  static sub(out, a, b) {
    out[0] = a[0] - b[0];
    out[1] = a[1] - b[1];
    out[2] = a[2] - b[2];
    return out;
  }
}

Used as:

let v1 = new Vec3(1, 2, 3);
let v2 = new Vec3(4, 5, 6);

Vec3.add(v1, v1, v2); // v1 is now Vec3(5, 7, 9);
Vec3.sub(v1, v1, [1, 1, 1]); // v1 is now Vec3(4, 6, 8);

As a minor design aside, I felt pretty strongly that as a class the type names should begin with an uppercase, but that does break backwards compat since in the original library all the "namespaces" were lowercase. This can be resolved by having the library defined a simple alias:

export const vec3 = Vec3;

Which then allows you to import whichever casing you need for your app, and even mix and match.

import { Vec3, vec3 } from './gl-matrix/vec3.js';

// This is all fine.
let v1 = new Vec3(1, 2, 3);
let v2 = new vec3(4, 5, 6);
Vec3.add(v1, v1, v2);
vec3.sub(v1, v1, [1, 1, 1]);

I would probably encourage migration to the uppercase variant over time, though.

Instance methods

Once we have a proper class backing out vectors and matrices, we can make many of the methods for those types instance methods, which operate explicitly on the this object.

export class Vec3 extends Float32Array {
  add(b) {
    this[0] += b[0];
    this[1] += b[1];
    this[2] += b[2];
    return this;
  }

  sub(b) {
    this[0] -= b[0];
    this[1] -= b[1];
    this[2] -= b[2];
    return this;
  }

  // etc...
}

Turns out that this doesn't conflict with the static methods of the same name on the same class! And it makes the syntax for common operations much easier to type and read:

let v1 = new Vec3(1, 2, 3);
let v2 = new Vec3(4, 5, 6);

v1.add(v2).sub([1, 1, 1]); // v1 now equals Vec3(4, 6, 8);

Actually there's two ways of going about this. One is that you implicitly make every operation on a vector apply to the vector itself, as shown above. The other is that you have each operation return a new instance of the vector with the result, leaving the operands unchanged. I feel pretty strongly that the former fits the ethos of glMatrix better by not creating constantly creating new objects unless it's necessary.

If you don't want to alter the values of the original object, there's still reasonably easy options that make it more explicit what you're doing and where new memory is being allocated.

let v3 = new Vec3(v1).add(v2); // v3 is now v1 + v2, v1 and v2 are unchanged.

And, of course you can mix and match with the older function style too, which is still handy for applying the result of two different operands to a third value or simply for migrating code piecemeal over time.

v1.add(v2);
Vec3.sub(v3, v1, [1, 1, 1]);

Attributes

Vectors being a real class means we can also offer a better way to access the components, because lets face it: typing v[0] instead of v.x is really annoying. Getters and setters to the rescue!

export class Vec3 extends Float32Array {
  get x() { return this[0]; }
  set x(value) { this[0] = value; }

  get y() { return this[1]; }
  set y(value) { this[1] = value; }

  get z() { return this[2]; }
  set z(value) { this[2] = value; }
}

Now we can choose to reference components by either index or name:

let v = new Vec3(1, 2, 3);
// Now this...
let len = Math.sqrt((v.x * v.x) + (v.y * v.y) + (v.z * v.z));
// Is equivalent to this...
let len2 = Math.sqrt((v[0] * v[0]) + (v[1] * v[1]) + (v[2] * v[2]));

All of method implementations internally will continue to lookup components by index both because it's a bit faster and because it allows for raw arrays to be passed in as temporary vectors and matrices, which is convenient.

Swizzles!

And hey, while we're adding accessors, why not borrow one of my favorite bits of shader syntax and add swizzle operators too!

export class Vec3 extends Float32Array {
  get xxx() { return new Vec3(this[0], this[0], this[0]); }
  get xxy() { return new Vec3(this[0], this[0], this[1]); }
  get xxz() { return new Vec3(this[0], this[0], this[2]); }
  get xyx() { return new Vec3(this[0], this[1], this[0]); }
  get xyy() { return new Vec3(this[0], this[1], this[1]); }
  // etc...
  get zzx() { return new Vec3(this[2], this[2], this[0]); }
  get zzy() { return new Vec3(this[2], this[2], this[1]); }
  get zzz() { return new Vec3(this[2], this[2], this[2]); }
}

Swizzles can operate between vector sizes as well. In practice it looks like this:

let v1 = Vec3(1, 2, 3);
let v2 = v1.zyx; // Vec3(3, 2, 1);
let v3 = v2.zz; // Vec2(1, 1);
let v4 = v1.xyzz; // Vec4(1, 2, 3, 3);

(These do break the "don't allocate lots of new objects rule a bit, but as a convenience I think it's worth it.)

Operator overloading

v1 = v2 + v3;

... is what I WISH I could implement here. Got your hopes up for a second, didn't I?

But no, JavaScript still stubbornly refuses to give us access to this particular tool because some people shot themselves in the foot with C++ once I guess? Check back in another decade, maybe?

TypeScript

I'm sure this will be mildly controversial, but I'm also leaning very strongly towards implementing the next version of glMatrix in TypeScript. There's a few reasons for this, not the least of which is that I've been using it much more myself lately as part of my job, and my understanding is that it's becoming far more common across the industry. It also helps spot implementation bugs, offers better autocomplete functionality in various IDEs, and I feel like the tooling that I've seen around things like documentation generation is a bit better.

As a result, having the library implemented natively in TypeScript feels like a natural step, especially considering that it doesn't prevent use in vanilla JavaScript. We'll be building a few different variants of the distributable files regardless.

Older Browser/Runtime compatibility

While I do feel very strongly about backwards compatibility of the library, that doesn't extend to supporting outdated browsers or runtimes. As a result while I'm not planning on doing anything to explicitly break it, I'm also not going to put any effort into supporting older browsers like IE 11 or fairly old versions of Node. Exactly where the cutoff line will land I'm not sure, it'll just depend on which versions support the features that we're utilizing.

ES Modules-first approach

glMatrix has used ES Modules as it's method of tying together multiple source files for a while, and I don't intend to change that. What I am curious about is how much demand there is out there for continuing to distribute other module types, such as CommonJS or AMD modules.

One thing I am fairly reluctant to continue supporting is defining all the library symbols on the global object (like window), since the library itself is already reasonably large in it's current form and the above changes will only make it larger.

Which brings me to a less exciting topic:

Caveats

All of the above features are great, and I'm sure that they'll be a net win for pretty much everybody, but they come with a few caveats that are worth being aware of.

File sizes will be larger

The addition of all the instance methods on top of the existing static methods, not to mention the large number of swizzle operations, would result is the library growing in size by a fair amount. I don't have numbers just yet, but I'd guess that the total size of glMatrix's distributable files growing by about 1/3 to 1/2 is in the right ballpark.

Obviously one aspect of building a well performing web app is keeping the download size down, and I don't want to adversely affect that just for the sake of adding some conveniences to the library.

I also, however, expect that most any developers that are truly size-conscious are already using a tree shaking, minifying build tool that will strip away any code that you're not actively accessing.

To that end, glMatrix's priority would be to avoid doing anything that would interfere with effective tree shaking rather than to try and reduce the base library size by only implementing a bare bones feature set.

length is complicated

One natural attribute that you'd want on a vector alongside the x, y, z, and w attributes is a length that gives the length of the vector itself, something that's already computed by vec{2|3|4}.length(v);

Unfortunately, length is already an attribute of Float32Array, and gives (as one would expect) the number of elements in the array.

We don't want to override length in our vector classes, since that would give nonsensical results in contexts where the object is being used as a Float32Array, which means that while we can retain the static length() function for backwards compat we'll need an alternative for the vector instances. I landed on magnitude (with an shorter alias of mag) as the replacement term, though I personally know I'm going to goof that one up at least once, and spend more time than I should wondering why the "length" of my vector is always 3. 😮‍💨

Vector/Matrix creation time will be slightly worse

This is a big one, as one of the most frequent criticisms leveled at glMatrix is that the creation of vectors and matrices is expensive compared to other similar libraries. This is because (for reasons that honestly escape me) creating TypedArray buffers or views is a slower operation in most JavaScript environments than creating a JavaScript object with the same number of elements/members.

My overall response to this has been that for many apps you'll eventually want to convert the results to a Float32Array anyway for use with one of the graphics APIs, and that avoiding creating a lot of temporary objects is a good practice for performance anyway, regardless of the relative cost of creating those objects. Both of those principles are still true, but it doesn't change the fact that this aspect of glMatrix is simply slower than some competing libraries.

The above proposals will not improve that situation, and in fact are likely to make it a bit worse. Having some extra logic in the extended classes constructor before passing through to the super() constructor will unavoidably add some overhead for the sake of providing a much nicer, more flexible syntax for developers.

If I were starting fresh I may very well take a different approach, but as I said at the top of this post backwards compat is very important to me for a library like this, so this is an area where I'm willing to accept this as a relative weakness of the library to be weighed against what I consider to be it's many strengths. Your milage may vary.

Accessors/instance methods will have overhead

As nice as it is to be able to access the object components by name, using the getter v.x is likely to always be a bit slower than accessing v[0] directly. Similarly some of the instance methods are likely to just pass through to the equivalent static method, especially in cases that would otherwise involve a significant amount of code duplication. For example:

export class Mat4 extends Float32Array {
  multiply(b) {
    return Mat4.multiply(this, this, b);
  }

  static multiply(out, a, b) {
    // Full matrix multiplication implementation.
  }
}

While I'm not necessarily a fan of adding functionality that's known to be less efficient than it could be, in this case I think that the aesthetic/usability benefits are worthwhile. And it's worth considering that there are plenty of times where the clarity of the piece of code will be more valuable than ensuring it uses every clock cycle to it's maximum potential. (I mean, lets be realistic: We're talking about JavaScript here. "Perfectly optimal" was never in the cards to begin with.)

I am happy knowing that in cases where the difference in overhead has a material impact on an application's performance, the conversion from accessors to indices, or to calling the static version of functions directly can be made painlessly and in isolation.

Preview

The code snippets in this post are all pretty simplistic, but I've put a fair amount of effort into validating this approach already, and currently have a WIP version of a potential glMatrix 4.0 available to look through in the glmatrix-next branch of this repo. It is definitely not in a generally useable state at this point, but looking at the vec2.ts, vec3.ts, and mat4.ts files should give you a good idea of how things are likely to look when everything is done.

Most of my efforts so far have gone into ensuring that things can work the way that I wanted, toying with file and directly structure, ensuring that the generated docs are clear and useful, and learning far more than I anticipated about TypeScript's quirks. But now that I'm satisfied that it's possible I wanted to gather some feedback from users of the library before pushing forward with the remainder of the implementation, which will probably be largely mechanical, boring, and time consuming. I'll likely take a week off work at some point to finish it up.

Thank you!

Thank you to everyone who has made use of glMatrix over the last 12 years! It's been incredible and humbling to see all the amazing work that it's been part of. And an even bigger thank you to everyone who has contributed to or helped maintain the library, even when I personally haven't had the time to do so!

[hmans]

Yes to all of these! Some comments:

• Having both a functional and object-based API is great 😍
• I think .x, .y etc. accessors are great, even at whatever performance cost they incur. The documentation could guide users towards using the property accessors for convenience, and the index accessors for performance.
• Don't care much about swizzling. I assume these will be for vectors only? I'm hoping you auto-generated the code in vec4.ts. :-)
• I've seen some libraries implement custom operator faux-overloading through Babel plugins. Maybe this could be an optional thing to provide to people who're already transpiling.

[toji]

Yes, the swizzle code is auto generated! 😆 I am definitely not patient enough to do it manually. And yes, it would just be for the vectors. I don't think it makes sense for quaternions or matrices. I am considering attributes that return vectors representing matrix columns or rows, though.

[Maksims]

Mostly a great improvements while keeping backwards compatibility!

In realtime context, allocations are expensive, especially Float32Array objects are a bit more expensive than simple objects. So it is best to avoid it as much as possible, or provide clear API design to communicate it to the user. One are of such I found that if a getter used from object, then it is expected to be very simple operation, and not an expensive allocation.
In this case while swizzles are great API design - they eventually can lead to unexpected allocations while looking harmless. Using a method notation instead of getter, looks worse from esthetically, but feels more serious.

Regarding of calling static method from class method, it is worth thinking which of them will be more popular. As I would assume class method is more often used so within class static method it could call class methods but not the way around.

[DavidPeicho]

I would be careful with static methods, as it broke bundlers tree-shaking up to 2020. I remember that it happened in major bundling tools: webpack, esbuild, etc... I guess they all use the same tool for optimization / minification etc... It might not be the case anymore, but it would be great to give it another try.
I had back then a sample to debug it, and a single class export could break everything with a static method.

As mentioned above, I would love to see swizzle with an out reference as well to prevent the allocation

[hmans]

One minor argument against moving to classes that inherit from Float32Array. One thing I really appreciate about working with gl-matrix is that the user of my libraries can just provide arrays of numbers as arguments to functions that then feed them to gl-matrix.

In a little game library I'm currently working on, I can do

autorotate: [1, 2, 3]

With the new version as described above, I think I then have to always do:

autorotate: new Vec3(1, 2, 3)

Which is just ever so slightly "worse" (purely in the sense that it increases boilerplate in userland.)

It would be great if the functional-style static functions like Vec3.add(a, b) are still able to operate on normal arrays like the one above. (Which, of course, leads to the question if we will then need to always stick with the functional-style API to make sure arrays can still be used in userland, which then eventually poses the question if we truly need a class-based API.)

Aaaaaah API design, I love/hate it :-)

[ibesora]

Love all this. Swizzling support is awesome.

Minor question about your performance concerns, particularly around accessors/instance methods overhead: Do you have any metrics on that? I wouldn't expect an extra function call to be noticeable

[sketchpunk]

"Vector/Matrix creation time will be slightly worse". How about creating two Vec3 objects that share the same functionality. Like a FVec3 extends Float32Array and JVec3 extends Array, then each one can somehow pull in the same methods, getters & setters.

I've been using a vec3 extended Float32Array for a few years now but in the last year or so I've been moving away by using regular javascript arrays for math heavy applications that doesn't need that gl compatibility that float32arrays provide. Having both an Array and Float32Array type can give users a choice of what sort of data structure to use.

Also, for your constructors, maybe you'd like to try out overloading. Here's how I initialize a vector 3 in typescript. Intellisense can then tell you the various ways to initialize it by having each constructor defined.
https://github.com/sketchpunk/oito/blob/main/packages/core/src/Vec3.ts#L18

So the following is possible

• new Vec3( [0,1,0] )
• new Vec3( 0, 1, 0 )
• new Vec3( 1 ) // great for scale vectors

[shannon]

I am excited for this!

One thing I might offer as a suggestion is the constructor may lead to an unexpected behavior when compared to shader code.

const v1 = new Vec3(5);

As with shader code I would expect it to create a value of (5, 5, 5) here, instead it makes a value of (0, 0, 0, 0, 0), a Float32Array of length 5. Whether this is crucial or not I can't say but it would be nice to have parity there.

Your code would just require an additional check:

export class Vec3 extends Float32Array {
  constructor(...values) {
    switch(values.length) {
      case 3: super(values); break;
      case 2:
      case 1: typeof values[0] === 'number' ? super(3).fill(values[0]) : super(values[0], values[1] ?? 0, 3) ; break;
      default: super(3); break;
    }
  }
}

let v1 = new Vec3(); // Creates a vector with value (0, 0, 0)
let v2 = new Vec3(1, 2, 3); // Creates a vector with value (1, 2, 3)
let v3 = new Vec3(v2); // Creates a copy of v2
let v4 = new Vec3(5) //Creats a vector with value (5, 5, 5)

let arrayBuffer = new Float32Array([0, 1, 2, 3, 4, 5, 6, 7, 8]).buffer;
let v5 = new Vec3(arrayBuffer); // Creates a vector mapped to offset 0 of arrayBuffer (0, 1, 2)
let v6 = new Vec3(arrayBuffer, 16); // Creates a vector mapped to offset 16 of arrayBuffer (4, 5, 6)

This also makes sense because the length as the first argument is no longer useful as it is expected the length will always be 3.

[toji]

Lots of great comments, thanks! Replying to a few real quickly:

Regarding of calling static method from class method, it is worth thinking which of them will be more popular. As I would assume class method is more often used so within class static method it could call class methods but not the way around.

As I have it right now that's not really possible because the static methods are formulated such that the first operand and the output are always the same object, which isn't guaranteed in the static versions. Also, I want the static versions to continue to work on things like raw arrays, so we can't assume that we can always call through to the instance method.

I would be careful with static methods, as it broke bundlers tree-shaking up to 2020.

That's not something I was aware of, thanks for bringing it to my attention! I'll do some more research on it, and if you have any links to related issues/docs/fixes/etc I'd appreciate it.

One thing I really appreciate about working with gl-matrix is that the user of my libraries can just provide arrays of numbers as arguments to functions that then feed them to gl-matrix.

Yeah, I appreciate that too! And the changes described here won't break it, it just requires a bit of discipline on your part as a library author. You'd just need to follow the pattern that the library itself is going to follow: Any time glMatrix returns a new vector/matrix it'll be an instance of the class, but any time it accepts a vector or matrix it only has to be an array-like object with enough elements.

In the TypeScript code I've done so far I declare types such as Vec3Like or Mat4Like which I use for almost all function params.

export type Vec3Like = [number, number, number] | Float32Array;

export class Vec3 extends Float32Array {
  static add(out: Vec3Like, a: Readonly<Vec3Like>, b: Readonly<Vec3Like>): Vec3Like {
    out[0] = a[0] + b[0];
    out[1] = a[1] + b[1];
    out[2] = a[2] + b[2];
    return out;
  }
}

This prevents the use of any of the instance methods in the function implementations, but retains maximum flexibility so it's a quirk I'm willing to deal with. It means that all of the following are still valid:

let v1 = new Vec3(1, 2, 3);
let v2 = new Float32Array([4, 5, 6]);
let v3 = [7, 8, 9];

v1.add(v3);
Vec3.subtract(v2, [9, 9, 9], v1);
v1.multiply(Vec3.scale([0, 0, 0], v3, 2.5));

Minor question about your performance concerns, particularly around accessors/instance methods overhead: Do you have any metrics on that? I wouldn't expect an extra function call to be noticeable

Not good ones yet. I don't expect it to be a lot, but it's probably something you could measure if you were, for instance, doing a tight loop of cascading matrix updates over a large scene graph.

I primarily bring it up because when it was first released I pitched glMatrix as "stupidly fast" and proudly showed benchmarks of it handily trouncing competing libraries at the time. That's not the case today, and I'm not really interested in pursuing the title of "fastest possible thing" at the expense of usability any longer, though performance is still one of my largest considerations. As such I feel it's noteworthy when design decisions are made that involve a performance compromise, big or small, but won't let it be a blocking factor unless it's egregious.

How about creating two Vec3 objects that share the same functionality. Like a FVec3 extends Float32Array and JVec3 extends Array, then each one can somehow pull in the same methods, getters & setters.

I've thought about it. Still trying to work out how to do so without making the API significantly more annoying to work with, especially across multiple libraries. In the meantime, as I pointed out above, the library will still function perfectly well with raw arrays in most cases.

const v1 = new Vec3(5);
As with shader code I would expect it to create a value of (5, 5, 5) here, instead it makes a value of (0, 0, 0, 0, 0), a Float32Array of length 5

Ooh! I'd considered doing expansion of the scalar value here previously but dropped it because I was trying to make the constructor simpler. Having scalar inputs accidentally change the length of the underlying array is bad, though, and I'm embarrassed I overlooked it. It's worth implementing the (admittedly nice) feature just to avoid the problem. Besides, it sounds like there's multiple people who would like that feature anyway! Thanks for pointing this out.

[arodic]

Hi Brandon,

I'm happy to see you are getting back to this project!

Typescript is the way to go, no doubt (I used to be sceptic until I was forced to get into it for work). Also esmodule-first approach is a trend I'd love to see more!

As for backwards compatibility, it is really neat that you've put so much effort to please the current users but having multiple ways of using the library can add confusion. With that in mind, I'd suggest to put deprecation on fast track. Perhaps put deprecation warnings in second release and deprecate old API right after that. That way, users can rip the band-aid off in two simple upgrades.

I landed on magnitude (with an shorter alias of mag) as the replacement term

Did you consider len()?

Swizzling is nice but I almost never use it JS. By the time I need to swizzle my vectors I'm already in GPU land but that's just me.

Would you consider adding unit vectors? I find them handy all the time!

As for library size, I don't think you should feel constrained by it. As long as it is tree-shakable, just have fun writing useful code :) Have you thought about adding geometric primitives, camera matrix utils and so on?

[peterreeves]

I landed on magnitude (with an shorter alias of mag) as the replacement term

What about dimensions?

Everything else looks good, especially TypeScript support. WebGL projects tend to be very large, and large projects are generally written in TypeScript for sanity reasons.

[toji]

Did you consider len()?
What about dimensions?

len() is already a static alias for VecN.length(), and my concern there is developers looking at the static methods and seeing length/len and then looking at the instance attributes and seeing length/len and thinking "Ah! It's the same!" when it very much is not.

As for dimensions, that suggests to me a width/height/depth rather than a scalar value. (IE: The dimensions of a photo are 3"x4", but the length of a Vec2(3, 4) is 5.)

Honestly there's just not any great options here that I've come across.

Would you consider adding unit vectors? I find them handy all the time!

I'm not sure what a "unit vector" means in that context? Like a vector that's constrained to always be 1 unit in length? Could you explain a bit more?

[donmccurdy]

@toji the proposed API changes sound really helpful from usage perspective!

However, I'd be really careful of assuming tree-shaking will just work, without testing against a couple of the more modern bundlers. The issue about tree-shaking class methods in Rollup appears to have closed without any resolution.

Things have generally improved since the comment in rollup/rollup#349 (comment), but I (still) feel that most bundlers are far worse at tree-shaking than Google Closure Compiler today, and very finicky about it. For example — I could easily imagine a line of code like this breaking all dead code elimination on the Vec3 class:

const swizzle = flip ? 'zyx' : 'xyz';

const [a, b, c] = vec[swizzle];

If that turns out to be an issue, perhaps a backup option would be to continue exporting operators as individual functions accepting arrays, but to also provide the classes (with instance methods, no static methods) for users who prefer them.

[toji]

So it turns out that doing all of the swizzle variants I was hoping for on Vec2, Vec3, and Vec4 would likely double the size of the library when minified. 😨 I know I said I wasn't too concerned about the library size but... that's significantly more than I was anticipating, and I'm not really willing to take THAT much of a size hit for one feature, especially one that will probably see reasonably light use.

Gonna experiment with injecting them dynamically (which totally goes against the TypeScript ethos, but whatever.) and if that doesn't work then I may just have to drop swizzles all together.

[nshen]

I prefer drop swizzles

[toji]

Fiddled around with the swizzle code last night and got to a place that I'm happier with.

I changed up the autogen code so that it writes out all the necessary symbols for the swizzle operations to a single, separate file (swizzle.ts), which can then dynamically generate the code necessary for them. But because the dynamic generation takes a moment (a couple of ms on my test machine) I don't have them automatically inject as a side effect of including the file, but instead it exports a function (EnableSwizzle() for now, will probably change that later) which can be called anywhere in your code to enable swizzle operations on all the vector types.

The primary upside to this is that if you don't care about the swizzle operators then you simple don't call that function and there's no overhead involved. If you're importing the types separately, rather than as a bundle, then you won't even need to download the swizzle table (the largest part). If you are using the bundled version the size impact has gone WAY down (from ~50Kb to ~5Kb), and tree shaking should more reliably cull those symbols out if you don't explicitly enable it.

The downsides are that there's an extra step involved if you want this feature, the implementation itself may be a bit slower, depending on how your JS engine optimizes it, and TypeScript doesn't recognize the swizzle operators because they're dynamically declared. That's unfortunate, but I'm wondering if I can get around it by having a generated .d.ts file that explicitly declares each swizzle variant. I'll have to explore that some more later.

Latest WIP code has been pushed to the glmatrix-next branch for anyone who is interested in taking a look!

[sketchpunk]

I'd be interested in adding a Transform object to gl-matrix. Is it something I can do on your glmatrix-next branch so it can launch with v4.0 or is it something I should wait for you to be completely done before contributing. Well, probably the first question should be does it make sense to add a transform object to gl-matrix.

https://github.com/sketchpunk/oito/blob/main/packages/core/src/Transform.ts

[nshen]

@sketchpunk It would be really useful if the transform object could like toCssTransform()
https://developer.mozilla.org/en-US/docs/Web/CSS/transform

[mreinstein]

If I were starting fresh I may very well take a different approach

That is what I'd rather see you do. v4 is going to be a big undertaking, and if you're going through all that bother I would rather see what you come up with, being able to take risks and explore new ideas without being chained via compatibility limitations.

What you're describing is different enough from traditional gl-matrix paradigm that I think it warrants this. There is absolutely no shame in making a new thing; gl-matrix will still live on and be amazing; it will just be different from the next great thing you design. You could even pick a name that isn't tied to gl. :)

operator overloading is what I WISH I could implement here.

Same. Maybe this could be a killer feature in a v4 fresh module not encumbered by legacy concerns. Years back I was looking through gl-matrix's issue tracker and someone raised the idea of a preprocessor/compile step. Maybe this is insane. But then again we live in a world where typescript has taken over, and what is that but an even more comprehensive pre-processor. :) The cumbersome syntax is probably my biggest gripe with gl-matrix, but I gladly put up with it because the awkwardness mostly comes from having to work around javascript spewing objects everywhere in most APIs, and gl-matrix doesn't generate any memory garbage.

because lets face it: typing v[0] instead of v.x is really annoying

I know that's a popular sentiment but is it really that bad typing v[0] ? I mean there's nothing special about .x or .y, they are just symbols we've all gotten used to. After a while v[0] and v[1] become fairly natural to look at.

using the getter v.x is likely to always be a bit slower than accessing v[0]

I don't want to go through every point in the list of ideas for v4 but getter/setter proxy performance is a concern. It's been a few years since I checked but in v8 these are costly. I dropped pixi.js for game dev because after doing some CPU profiling I discovered internally it uses getter/setters in various places like the transform components, accessing these tend to snowball, and it chews up a surprising amount of CPU when you're building non-toy sims/games that operate on thousands of objects in a tight render loop.

Using gl-matrix@3 and somewhat naive webgpu, I was was able to cut the cpu usage in half compared to pixi, and this was only a few weeks ago. Certainly not a scientific measurement but it seems like there are still some perf issues there.

Thank you to everyone who has made use of glMatrix over the last 12 years

Thank you for producing one of the best matrix libraries. People don't appreciate how important these low-level primitive handling libraries are. They are so foundational to make jank free games/sims/animations. I'm excited you have time to spend on this!

[mreinstein]

If you are feeling inclined to spend time on a successor to gl-matrix, I now have author privs for the matrix npm package. I'd be happy to hand this over to you if you'd find it useful @toji

[miko3k]

Strength of glMatrix is its absolute interoperatbility thanks to its universal representation. I personally love this feature. Classes will make it harder.

I would prefer more evolutionary changes, my personal list:

• proper typings: function add<T extends vec3>(out: T, a: ReadonlyVec3, b: ReadonlyVec3): T. I wanted to make PR for this but I would probably require conversion to typescript. I gave up in the end.

• deprecating weird functions, like vec2.transformMat4 (vec3.transformMat4 performs w-division; vec2.transformMat4 does not #415)

• split one function per module, to enable better tree shaking. This is harder than it seems though because of the current module structure. Swizzles could be normal functions. vec3.swizzleXXY and automatically dropped by the bundler.

• there's a only a small number of functions that allocate memory. I would provide two versions of each (vec3.createArray/vec3.createFloat32) and while keeping ARRAY_TYPE as "default". This would enable client code to choose better fitting version.

I realize that classes provide better ergnomy, however I strongly feel they will turn this into different kind of library.

[ova2]

A long time ago I've tried this approach. We have real-time UIs with many points. The problem with this approach was the increased app size. Native arrays are lightweight, but Float32Array has a big size. I prefer to do transformation from native arrays to Float32Array just before rendering step. I could not detect any performance optimizations when working directly with Float32Array. You really have to do a prototype first to be able to messure optimizations before implementing a new approach.

[shannon]

@toji I know you covered this in the description but I wanted to leave some feedback regarding the move to Typescript.

Over the years I have moved away from having any build step in my development process (bundling comes only at the publishing stage now). There are many reasons for this so I won't try to state them all here, but I have rather enjoyed the fact that gl-matrix was one of the libraries that I could just do import { mat4 } from 'https://cdn.jsdelivr.net/gh/toji/gl-matrix@v3.4.1/src/index.js', or get as fine grained as I would like using the individual source files. There are more and more libraries implementing standard ESM so this is becoming easier and I am almost at a point where I can do away with NPM entirely.

I wanted to try experimenting with the v4.0 branch but I soon discovered this approach is no longer possible and that there really is no easy way for me use it right now since it has not been published anywhere in a transpiled form. gl-matrix is something I use a lot and it feels a bit like a step backwards.

There has also been a lot of open discussion about Svelte moving away from using Typescript code (while still using JSDoc to implement Typescript based type checking).
Twitter Discussion: https://twitter.com/jutanium/status/1639341148157140993
This article gives a nice overview of that discussion: https://dev.to/thepassle/using-typescript-without-compilation-3ko4.

The points being made by Rich Harris and others are very pertinent when it comes to a library like gl-matrix. Anyways, I don't really expect you to change the entire code base, but I had hoped this feedback can be of some value for you.

[toji]

Hey @shannon!

This has been an interesting journey for me, because while I've gotten quite comfortable working in TypeScript overall I'd never tried to build middleware with it before now. I've been playing with different ways of building, importing, and using the library in other projects (I'm intent on inflicting any pain on myself before asking the community to endure it), and trying to find the right configuration for maximum ecosystem compatibility.

Being perfectly honest: It sucks. Not really the language or the tooling, but the packaging and distribution story is a mess. There's not a clean way to distribute "A TypeScript Library" that everyone can just use, because you have to match about 50 different assumptions about how their own project is set up to be compatible. Seems like there's some patterns that can be used to make it a bit better or worse, but overall it's just not practical.

Which is a shame, because there's aspects of TypeScript that are really quite nice, even for a library like this. Things like being able to specify certain arguments as Readonly for example, is a great way both to express the intent of the function and keep your implementation honest. I've found that generating docs is cleaner with TypeScript too.

So I'm going to some advice that I got when I first started lamenting the packaging situation online: glMatrix 4.0 will still be written in TypeScript, but when it's packaged for distribution it'll just be JavaScript with some .d.ts files alongside. This seems to be the single most compatible approach across the entire ecosystem (and even it still suffers from incompatibility potholes, but that's more JavaScript's fault than TypeScript at that point.) So if you want to just import directly from a CDN into your vanilla JS project it'll just work, and if you install and import the distributed files in a TypeScript app you'll still get the definitions. Feels a bit silly to write in one language and then compile to another in order to let projects written in the first language consume it, but that just seems to be where TypeScript is at right now.

I'll have the build files available in at least three forms: Separate ES modules files for individual imports, a bundled ES module file with the full library in it, and a Common JS bundle as well, since I think that's still a reasonably common use case. If anyone wants to copy the TypeScript source into their own project and configure it manually they're welcome to do so, but it won't be something I try to support directly.

Anyway, hope that puts some concerns at ease! There will be a build step but it won't be you that has to do it. :)

And sorry that it's taking so long to get this published in an easily consumable place! I actually want to work on that really soon because I need it for my own project. Need to look into the best way to publish "beta" packages with NPM and the like.

[shannon]

Thanks @toji I completely understand your point of view. I don't want to rush you either so I will experiment when it's published :-)

[mreinstein]

Over the years I have moved away from having any build step in my development process (bundling comes only at the publishing stage now)

@shannon 100% agree with you, but unfortunately we're in the minority these days.

[ibgreen]

@toji good to see you returning to this great project and thanks for taking the time to share a roadmap.

FWIW, the vis.gl / deck.gl ecosystem already contains an implementation of gl-matrix-based classes that seems quite similar to what you propose. One of our core component is the @math.gl/core module which contains Vector, Matrix etc classes subclassed from arrays with methods that call gl-matrix functions on this...

We built these wrappers back in 2017 and have used them successfully in many frameworks and applications over the years. Our implementation seems fairly similar to the direction you outline, so if nothing else could be useful as a reference.

Possibly the biggest difference from your proposal is that we chose to subclass from Array rather than Float32Array.

• I don't think subclassing Float33Array was possible back then
• but more importantly, 32 bit precision would simply be too limiting for our use cases
• we need to maintain higher precision math on the JS side (mainly geospatial 3D use cases) before converting final results to Float32 for shader usage.
• Also we wanted our class instances to be able to be used wherever standard JS arrays were expected. So in our case, all the functional gl-matrix functions still work with instances of our vector and matrix classes.

It would certainly be nice if this developed to a point where vis.gl could replace our own @math.gl/core wrapper module with a new official class based gl-matrix, however that depends on the direction your design takes. Regardless all our code is MIT licensed so if you see anything you like, feel free to use it without asking.

PS - Unfortunately, while we have been using gl-matrix as the base for math.gl since 2017, we are just about to fork and drop the gl-matrix dependency altogether since we have not found a way to work around gl-matrix lack of ES module support: #455. We'll basically be adding copies of the gl-matrix functions we use to @math.gl/core until the problem is resolved.

@Pessimistress

[blfunex]

For swizzling why not return vector compatible tuples instead of creating the vectors (JS Array vs Vector f32 array buffer view), I reckon js engines have mastered js array optimisations better than array buffer views.

[blfunex]

Since swizzling is opt in, typescript for it should also be opt in, like npm i -D @types/gl-matrix/swizzle, to add them through global interfaces into gl-matrix own definition.

[Swoorup]

Is there a way we could also support higher precision? As @ibgreen mentioned earlier, 32-bit precision is bit limiting for my use case as well.

Hence I've always been setting glMatrix.setMatrixArrayType(Array) at initialisation to retain the precision

[shannon]

Would it make sense to make it configurable to extend Float64Array instead of Float32Array?

[shi-yan]

Is there a way to specify the handedness of glMatrix?

i.e. change it from the right handed system to left handed system.

because I want to use it for WebGPU, and according to the WebGPU spec, it uses the DirectX coordinate systems (left handed system).

[elalish]

Oh no, if WebGPU uses left-handed, I blame you, @toji 😝

Left-handed matrix math should have been expunged from the world years ago... In fact it had been until some DirectX software engineer in the 90s who never took physics thought "I'll just do this my own way".

[psnet]

So long thread, want to say that TS is good choice to put sources in and it will be good if backward compatible will be as much as possible, projects using this lib cannot rewrite themselves for new API...

[bnwa]

Is this still alive or is there a notion of an ETA for v4 stable?

[anxpara]

i, too, am wondering what the roadmap looks like right now

[toji]

This is still alive, I've just been swamped with a lot of other priorities and so haven't been able to spend as much time on it as I was hoping. I'm probably going to take several days over the next month or so and dedicate time to just moving this forward, especially given the excellent feedback on this thread and the obvious interest in it.

Thanks for your patience, all!

[toji]

@Swoorup: Is there a way we could also support higher precision? As @ibgreen mentioned earlier, 32-bit precision is bit limiting for my use case as well.

@shannon: Would it make sense to make it configurable to extend Float64Array instead of Float32Array?

I agree that this is an important use case and thanks for reminding me to investigate it! I've been looking into this today and it's unfortunately not as trivial as it was in the prior version of the library if I want to stick to the Typescript inheritance patterns I've been using so far. Fiddled around with Mixins for a bit to try and make it work but it starts to really mess with the type checking and docs generation. I'm starting to think I might just do a really dumb, basic thing and have a build script that just copies the file and does a search and replace Float32Array -> Float64Array. 🙄

In any case, though, I wanted to get opinions about the mechanism for exposing the different precisions. setMatrixArrayType() won't work any more with the patterns I'm using, so I was instead considering exposing the types as Vec3_F32, Vec3_F64, etc internally, and then aliasing Vec3 to Vec3_F32 by default. But you could still change the "default" at the import step like so:

import { Vec3_F64 as Vec3 } from './gl-matrix/vec3.js';

Is that workable for your use cases? (It would also allow for easier mixing of precisions).

@shannon: It looks like the Quat.fromEuler method is missing the order option added in the v3.4.1.

Huh. I have no idea why I missed that. Wasn't intentional. I'll restore it in the next build, thanks for pointing that out!

[shannon]

I agree that this is an important use case and thanks for reminding me to investigate it! I've been looking into this today and it's unfortunately not as trivial as it was in the prior version of the library if I want to stick to the Typescript inheritance patterns I've been using so far. Fiddled around with Mixins for a bit to try and make it work but it starts to really mess with the type checking and docs generation. I'm starting to think I might just do a really dumb, basic thing and have a build script that just copies the file and does a search and replace Float32Array -> Float64Array. 🙄

Yea I really had expected something like this to work:

const Vec2Factory = <T extends Float32ArrayConstructor | Float64ArrayConstructor>(TypedArray: T) => { 
  class Vec2 extends TypedArray {
    // ... snip ...
  }

  // Instance method alias assignments
  // ... snip ...

  // Static method alias assignments
  // ... snip ...
  
  return Vec2;
}

export const Vec2_F32 = Vec2Factory(Float32Array);
export const Vec2_F64 = Vec2Factory(Float64Array);
export const Vec2 = Vec2_F32;
export const vec2 = Vec2_F32;

The interesting bit is that Typescript doesn't present an error with just the above snippet. It just doesn't extend it and add any of the methods (i.e. set) to the class.

The problem I see is that the Float32Array and Float64Array interfaces are incompatible mainly because the type itself is used in a parameter or returned in various methods (i.e. filter). So I created a new FloatArrayInterface generic that accepted Float32Array | Float64Array.

Then I was able to get the following error:

A mixin class must have a constructor with a single rest parameter of type 'any[]'

So with even more fiddling around the conctructor I got this to work but it may seem a bit odd.

import { EPSILON, FloatArray } from './common.js';
import { Mat2Like } from './mat2.js';
import { Mat2dLike } from './mat2d.js';
import { Mat3Like } from './mat3.js';
import { Mat4Like } from './mat4.js';

interface FloatArrayInterface<T extends Float32Array | Float64Array = Float32Array | Float64Array> {
  readonly BYTES_PER_ELEMENT: number;
  readonly buffer: ArrayBufferLike;
  readonly byteLength: number;
  readonly byteOffset: number;

  copyWithin(target: number, start: number, end?: number): this;
  every(predicate: (value: number, index: number, array: T) => unknown, thisArg?: any): boolean;
  fill(value: number, start?: number, end?: number): this;
  filter(predicate: (value: number, index: number, array: T) => any, thisArg?: any): T;
  find(predicate: (value: number, index: number, obj: T) => boolean, thisArg?: any): number | undefined;
  findIndex(predicate: (value: number, index: number, obj: T) => boolean, thisArg?: any): number;
  forEach(callbackfn: (value: number, index: number, array: T) => void, thisArg?: any): void;
  indexOf(searchElement: number, fromIndex?: number): number;
  join(separator?: string): string;
  lastIndexOf(searchElement: number, fromIndex?: number): number;
  readonly length: number;
  map(callbackfn: (value: number, index: number, array: T) => number, thisArg?: any): T;
  reduce(callbackfn: (previousValue: number, currentValue: number, currentIndex: number, array: T) => number): number;
  reduce(callbackfn: (previousValue: number, currentValue: number, currentIndex: number, array: T) => number, initialValue: number): number;
  reduce<U>(callbackfn: (previousValue: U, currentValue: number, currentIndex: number, array: T) => U, initialValue: U): U;
  reduceRight(callbackfn: (previousValue: number, currentValue: number, currentIndex: number, array: T) => number): number;
  reduceRight(callbackfn: (previousValue: number, currentValue: number, currentIndex: number, array: T) => number, initialValue: number): number;
  reduceRight<U>(callbackfn: (previousValue: U, currentValue: number, currentIndex: number, array: T) => U, initialValue: U): U;
  reverse(): T;
  set(array: ArrayLike<number>, offset?: number): void;
  slice(start?: number, end?: number): T;
  some(predicate: (value: number, index: number, array: T) => unknown, thisArg?: any): boolean;
  sort(compareFn?: (a: number, b: number) => number): this;
  subarray(begin?: number, end?: number): T;
  toLocaleString(): string;
  toString(): string;
  valueOf(): T;
  [index: number]: number;
}

type FloatArrayConstructor = { new (...value: any[]): FloatArrayInterface, BYTES_PER_ELEMENT: number };

/**
 * A 2 dimensional vector given as a {@link Vec2}, a 2-element floating point
 * TypedArray, or an array of 2 numbers.
 */
export type Vec2Like = [number, number] | FloatArrayInterface;


function Vec2Factory<T extends FloatArrayConstructor>(TypedArray: T) {
  /**
   * 2 Dimensional Vector
   */
  class Vec2 extends TypedArray {
    /**
     * The number of bytes in a {@link Vec2}.
     */
    static readonly BYTE_LENGTH = 2 * TypedArray.BYTES_PER_ELEMENT;

    /**
     * Create a {@link Vec2}.
     */
    constructor(...values: any[]) {
      switch(values.length) {
        case 2:{
          const v = values[0];
          if (typeof v === 'number') {
            super([v, values[1]]);
          } else {
            super(v as ArrayBufferLike, values[1], 2);
          }
          break;
        }
        case 1: {
          const v = values[0];
          if (typeof v === 'number') {
            super([v, v]);
          } else {
            super(v as ArrayBufferLike, 0, 2);
          }
          break;
        }
        default:
          super(2); break;
      }
    }

    // ... snip ...
  }

  // Instance method alias assignments
  // ... snip ...

  // Static method alias assignments
  // ... snip ...

  return Vec2
}

export const Vec2_F32 = Vec2Factory(Float32Array);
export const Vec2_F64 = Vec2Factory(Float64Array);
export const Vec2 = Vec2_F32;
export const vec2 = Vec2_F32;

The args on the constructor are listed as any type but when you actually try to create a new Vec2 it shows the appropriate types from the super class (ArrayBufferLike), and will error if you try something else.

I think you would probably just replace your FloatArray with the above interface. And then wrap each class in factory as I have done here. I can submit a PR if this makes sense. I honestly don't know as I am not as familiar with Typescript as I am with Javascript.

*Edit: I see now this does mess up the constructor parameters because we had wanted to allow Vec2(1, 2) and this no longer works (it throws Argument of type 'number' is not assignable to parameter of type 'ArrayBufferLike'). I can try to poke around some more but I don't think Typescript is going to allow this mixin. microsoft/TypeScript#37142

[shannon]

Alternatively, if you are ok getting rid of the new keyword on the api side you can do this:

// ... interface snippet from above ...

function Vec2Factory<T extends FloatArrayConstructor>(TypedArray: T) {
  /**
   * 2 Dimensional Vector
   */
  class Vec2 extends TypedArray {
    /**
     * The number of bytes in a {@link Vec2}.
     */
    static readonly BYTE_LENGTH = 2 * TypedArray.BYTES_PER_ELEMENT;

    // ... constructor removed ....

    // ... snip ...
  }

  // Instance method alias assignments
  // ... snip ...

  const factory = (...values: [Readonly<Vec2Like> | ArrayBufferLike, number?] | number[]) => {
    switch(values.length) {
      case 2:{
        const v = values[0];
        if (typeof v === 'number') {
          return new Vec2([v, values[1]]);
        } else {
          return new Vec2(v as ArrayBufferLike, values[1], 2);
        }
      }
      case 1: {
        const v = values[0];
        if (typeof v === 'number') {
          return new Vec2([v, v]);
        } else {
          return new Vec2(v as ArrayBufferLike, 0, 2);
        }
      }
      default:
        return new Vec2(2);
    }
  }

  factory.BYTE_LENGTH = Vec2.BYTE_LENGTH;

  // Static method alias assignments
  factory.sub = Vec2.subtract;
  // ... snip ...

  return factory;
}


export const Vec2_F32 = Vec2Factory(Float32Array);
export const Vec2_F64 = Vec2Factory(Float64Array);

export const Vec2 = Vec2_F32;
export const vec2 = Vec2_F32;

import { Vec2_F64 as Vec2, vec2 } from './gl-matrix/vec2.js';
const a = Vec2(1, 2);
const b = Vec2([1, 2]);
a.add(b);

vec2.sub(a, a, b)

[toji]

I like using new for a couple of reasons. For one, it's using things the "right way", and I find that it's generally better to try to stick with the design patterns as their meant to be used rather than get too cute with JavaScript tricks, in that as browsers evolve they are more likely to optimize the common patterns and more likely to break or deoptimize One Weird Trick-style code.

It also serves as a pretty clear indicator to devs about the type of work being done. new "feels" more expensive than a function call (despite the fact that that's often laughably untrue), and as such provides a mental nudge to use it accordingly.

That said, I'm still open to changing up the patterns here if there's a clear benefit to doing so, but in this case I'm not convinced that that jumping through template hoops is actually going to yield a better, more usable result than just doing the dumb copy thing. (I tried out that route here, we'll see how I feel about it after a few more revisions.)

The problem I see is that the Float32Array and Float64Array interfaces are incompatible mainly because the type itself is used in a parameter or returned in various methods (i.e. filter). So I created a new FloatArrayInterface generic that accepted Float32Array | Float64Array.

I came to the same realization, so I added a type FloatArray = Float32Array | Float64Array that I'm using for all the Vec/Mat*Like types now. It's simpler than what you proposed, though, and there's aspects of the interface direction that you showed that I kind of like, so it's worth experimenting more. I'm also seeing some aspects of the code you posted that are variants on what I was trying that might make a difference, so I'm definitely going to keep toying with it!

Thanks for the feedback!

[toji]

Just published a new beta version to npm for all you fine people to try out!

There are still things I'm investigating, but there was also enough new here that I felt it was worthwhile to push a new version and get feedback. I'm trying to both pull in feedback from this thread and address a variety of longstanding issues files against the library.

Beta.2 Changelog

• Support for Float64Array backing of all types (See README for usage details)
• Added Mat4.normalFromMat4() method (calculates the transpose inverse and drops the translation component)
• Added missing order arg back to Quat.fromEuler()
• Added Mat4.frustumZO, allowed frustum methods to accept an infinite far plane
• Explicitly add | null to return types of methods that may return null. (Such as matrix inversion)
• Cleaned up some temp variable use to enable better tree shaking
• Added toRadian() and toDegree() to common.ts
• Added Vec*.abs() static and instance method to all vectors
• Cleaned up some comment typos
• Added main, module, exports, and types to package.json

[christjt]

Oh no, if WebGPU uses left-handed, I blame you, @toji 😝

Left-handed matrix math should have been expunged from the world years ago... In fact it had been until some DirectX software engineer in the 90s who never took physics thought "I'll just do this my own way".

WebGPU doesn't make any assumptions of handedness that your app uses. Typically you just define whatever coordinate system you want, and then finally apply the projection matrix which also bakes in the transformation from the coordinate system used by your app to NDC, which in WebGPUs case is indeed left handed. But absolutely nothing is stopping you from writing your entire app in a right-handed coordinate system of your choice.

@toji As WebGPU has z defined from 0..1 in NDC, it also opens up the possibility to reverse z order, which I assume a lot of us will be doing.

I am a little torn here on how much I think gl-matrix should assist in enabling this because it is simply a matter of baking in another transform into the projection matrix produced by gl-matrix. I think the current solution is fine, and it is ok to be opinionated and not expand the API to include handedness when creating the perspective matrix, but maybe it should be documented? Also, if a lot of users struggle with this, then maybe we could add some convinience methods to Mat4 to convert to and from other coordinate systems if we can find an API that makes sense. I am sure you have considered this though, what do you think?

[typhonrt]

Hi @toji.

I'm finally able to get around to investigating the state of beta.2. I can verify that the tree-shaking changes are valid over beta.1, however I'd like to once again discuss a better package.json configuration as per several of the points brought up prior discussion. This involves an efficient separation of the classic and modern API (and f64 variants), type declarations, and package.json configuration.

I'll create a fork of beta.2 and put together what I think is the best set of solutions including a really slick TypeDoc / documentation example that accurately represents the contents of the package. I've put in a bit of time working on TypeDoc tooling in the past year for automatic API docs from a well configured package.json w/ types. I expect this work to be done sometime next week, so do hope you can review and consider this effort as a proposed way to distribute gl-matrix.

I've been shipping my fork of beta.1 for the last year with modern API only for my larger framework that re-exports gl-matrix.

[toji]

I'm definitely interested in taking a look, @typhonrt! I'll admit that futzing around with package.json and friends is one of my least favorite aspects of library maintenance, simply because it's often opaque to me how some of the values in there affect other people's workflows. I'm happy to consider ways to make it more flexible and less fragile.

[typhonrt]

I'm definitely interested in taking a look...

Almost done... I've taken a very thorough approach to this effort to deliver a rock solid option to review that is 100% modern continuing off what you have done so far vs outright replacement in the build process. Hopefully tomorrow / maybe Monday. Just mentioning this small delay as if what I present is approved of course merging w/ the glmatrix-next branch will be much easier if no commits are made there at this time. I'm not expecting that per se, but just a consideration.

Edit (6/9/24): I've been putting together a video overview of the proposed v4 distribution I've created. I finished all of the coding / maintenance last week. Just dropping the code will not explain the problems of "beta2" and my proposed changes. The video & code drop will be available this week.

[typhonrt]

Alright @toji et al.

I have got all of my proposed distribution changes complete along with a video overview due to the comprehensive nature of the work involved. This was a lot of work and I treated the process with full due diligence in the aim of speeding along the transition of gl-matrix to a fully released version 4. There is a lot here. I came up with a way to cleanly handle the new 64-bit variation of the API introduced in beta.2, fixed all Typescript errors in the repo, general code cleanup (formatting of tests), ensured full support of CommonJS / ESM Node distribution w/ all types available, added new Typescript ambient module declarations for the swizzling API, generated API docs that completely represent the library and more. It's a lot to take in really, but this effort fully modernizes gl-matrix and IMHO makes it production ready.

A video overview is available here that discusses the changes in reasonable detail:
https://www.youtube.com/watch?v=dZbk5E9nTbw

I don't often make large potential PRs like this to projects without prior negotiation, so hopefully this is well received; also, the first time I've made a video on the process to describe a potential PR. I found that this is likely an easier way to kickstart a discussion. I do hope you can take the time to review things and continue the discussion. My motivation as a downstream consumer of gl-matrix is to make all of the libraries I depend upon as rock solid as possible in an effort to also make my framework rock solid.

The approaches taken show a very modern way to release a Node package using Typescript and NodeNext module resolution. Hence my PR and single commit is titled "NodeNext Overhaul".

---

Resources

Main fork repo:
https://github.com/typhonjs-svelte-scratch/gl-matrix-beta2/tree/glmatrix-next

New and complete package API Docs:
https://typhonjs-svelte-scratch.github.io/gl-matrix-beta3-dist/

There is a Github distribution for this release that can be used right now by assigning gl-matrix in package.json in dependencies as follows:

{
  "dependencies": {
    "gl-matrix": "github:typhonjs-svelte-scratch/gl-matrix-beta3-dist#glmatrix-next"
  } 
}

I am more than willing to take the time to fully discuss this potential PR and work with you on getting gl-matrix v4 to a full production release.

[toji]

This is really fantastic, @typhonrt! I'm blown away by the amount of effort you've put into this, thank you!

I watched through your video, took a look at the generated docs, and tried to use your build in my own code and have a couple of initial bits of feedback:

First, I wasn't able to get your build to work in one of my more recent project, and it seems like it's mostly due to the dist/_lib/f32/* files using non-relative imports. Specifically it got caught on import { GLM_EPSILON } from 'gl-matrix/common'; in multiple files. I imagine this should be relatively easy to sort out, but it does highlight a priority for me: As much as possible I want the library to be usable in node and the browser without requiring a build step on the part of the app. (It's fine if the library itself needs a build step, that's pretty much a given.) To be honest this has been one of the most difficult part of not just developing this library but web dev in general lately. Everyone uses different environments with different rules about things like module path resolution and it's really easy to pull in a library that simply doesn't work because they use a different environment than you.

It may not be possible to satisfy every possible build environment, but I think this particular issue will be easy to fix and mostly want to ensure that the library stays as environment-neutral as possible.

The other issue that came up was I noticed that in the (much improved!) vector docs there's now lots of entries for every possible swizzle, which makes it difficult to browse for the more commonly used methods and attributes. I'm curious if you have any thoughts on ways that we could document the swizzles but have them separated out from the rest of the vector docs. This is tricky because I think the work that you've done to make the swizzles work better with the type system overall is amazing, but I suspect it also makes it harder to handle them in a unique way. In any case, I just don't want them harming readability of the docs.

But putting some of those concerns aside, I really can't compliment you enough on the effort you've put in here! I nodded along with most of your video explanation, having run into some of the problems you pointed out before and being surprised by others simply by virtue of not having much experience with building packages for environment XYZ. I'm very happy to work with you to get over some of the remaining hurdles and roll your updates into the repo, as it feels like it will offer a more robust developer experience all around. Thank you again, and I look forward to working with you to make v4 production ready!

[typhonrt]

I'd be glad to address the concerns you brought up @toji

Docs:

The other issue that came up was I noticed that in the (much improved!) vector docs there's now lots of entries for every possible swizzle, which makes it difficult to browse for the more commonly used methods and attributes.

I have made a 2nd commit that groups the swizzle API additions in the vector classes by the Swizzle API category using @category. I also corrected BYTE_LENGTH with it now being placed in the Static category and created a Constructor category. I added typedoc.json for clearer configuration of TypeDoc and added defaultCategory and categoryOrder for a consistent category order in the classes that use @category. Check the refreshed example of the API docs to confirm if all looks good on the docs front.

Edit (post docs / 2nd commit): One thing I did notice though is that using a default category as Member is not necessarily ideal with the inherited symbols from the backing array class when inherited is selected in the docs settings. With TypeDoc it seems once you start using @category it does group all symbols from the inherited class in the default category. It might be best to add @category to all methods with @category Methods then anything inherited from Float32Array, etc. gets put into a default Other category.

CDN Build:

First, I wasn't able to get your build to work in one of my more recent project, and it seems like it's mostly due to the dist/_lib/f32/* files using non-relative imports. Specifically it got caught on import { GLM_EPSILON } from 'gl-matrix/common'; in multiple files. I imagine this should be relatively easy to sort out, but it does highlight a priority for me: As much as possible I want the library to be usable in node and the browser without requiring a build step on the part of the app. (It's fine if the library itself needs a build step, that's pretty much a given.) To be honest this has been one of the most difficult part of not just developing this library but web dev in general lately.

I agree that there are challenges here and they should be addressed. The Node package in the first commit that I created does require a bundler for web distribution. I should note though that using NodeNext module resolution and sub-path exports does solve the clean implementation of the 64-bit API, so it does have value. I'm mainly in the Node / Browser / Bundler camp. The demo I did show of my framework running in the browser does go through the bundling process without Babel as ES2022 (94% of installed browsers support) is required for my framework.

The solution is to create all inclusive ESM CDN bundles of both variations of the library. I was going to do this, but didn't want so many moving parts in this potential PR, but can definitely address this. It certainly is possible to involve Babel if you really want to cast a wide net in the CDN bundles. Now IMHO these CDN bundles should be ESM and loaded from a standard import route pointing to the bundle or using import maps. The world as of this moment is firmly tilting towards ESM.

Edit: A UMD bundle should be provided; previously I mentioned not providing one.

My general idea here is to create a dist-cdn folder that contains separate bundles that are all inclusive for the 32/64-bit variations of the library. This also brings up general concerns around consumption in alternate runtimes like Deno and Bun. Neither of which I have used yet personally. There are several ESM CDNs that automatically deploy Node packages like esm.sh, jsdelivr, and unpkg. Deno to my understanding uses esm.sh. Just like Typescript lagging behind on modern Node features like sub-path exports (for ~4 years!) my understanding is that not all ESM CDNs are created equal at this point and may not support correctly sub-path export based Node packages.

However, it's my general understanding that one can handle this by setting main and types fields in the top level of package.json to the ESM CDN bundle and most of those ESM CDNs will refer to that instead of the exports field. Any consumption via Node / bundler use still refers to exports over main / types, so that is fine. There are also specific fields one can add for the various ESM CDNs to specifically point them at an entry point. A minor complication there with only having one entry point for some of these ESM CDNs is that this will refer to the 32-bit API. There will be a specific 64-bit bundle, but that will have to be resolved with a direct path loading on the CDN in question.

One of the big pains to my understanding with Deno and such for instance is that there is no easy way to test a Node package before it is released and consumed by esm.sh. For complete testing this may require me to release a non-announced version of the lib under my own control so that it hits the ESM CDNs then I can test on Deno and such. I'd then remove that package from NPM. Taking this approach will at least reduce the churn of pumping out extra beta releases of gl-matrix itself.

Build Consideration:

Now that we are discussing CDN bundles and such I'd like to bring up the possibility of switching to Rollup for the library bundling. I specifically didn't want to change the build process much that you put in place with beta.1 / beta.2. With Rollup though it will be possible to have one configuration file to handle everything from building the library to including the CJS builds and extra CDN bundles without the need for a separate script. Likewise esm-d-ts my tool that creates the bundled type declarations can be used from Rollup, so it really would be a single build file. Run Rollup and it just goes.

Immediate course of action:

So potential courses of action that I'd like your input on before I start any work:

• Create ESM CDN bundles

  • The 32-bit bundle will include the classic remapped API / the 64-bit version will not.
  • Should Babel be used? (My vote is no, but I understand if this is desired; this is less desirable for Deno consumption for instance I gather).
  • Add extra fields to package.json for main, types, and various ESM CDN fields that are known.

• Create UMD bundles.

• Test in Deno & Bun. The large part of the JS world does not use either FYI, but for completeness sake.

• Possibly switch to Rollup for building library. See section above.

• Update README to give CDN instructions and a Deno example.

Mid to longer term action items:

• Add ESLint for code formatting errors.

• Add .editorconfig for automatic IDE configuration.

• Possibly add Playwright and headless browser tests for CDN bundles.

---

That is a fair amount of action items above. I'd of course like to find a happy medium where I can make a PR that will be accepted. A fair amount of the tasks above are additional hardening beyond the current NodeNext refactor. I'd be glad to be involved in everything that it takes to get a final production release, but finding that merge point too and not have it go on too long is ideal as well. IE I'd like to do more follow up through additional PRs rather than one mega one.

[typhonrt]

I suppose the above is a bit TL;DR... The initial work that I posted about was the Node package overhaul itself. The remaining work of coming up with some tidy all-inclusive pre-built distributions of the library is completed. So @toji pick your poison so to speak from dist-cdn for pre-built direct in browser use cases. There are ESM 2016 / 2022 options w/ the 2016 version transpiled. There is also a fully functional ES5 / UMD version. I've fully tested them manually including testing the UMD version in RequireJS (whoa haven't used that in about ~10 years). No real need to add Playwright for testing the CDN bundles. I did have to get Rollup involved in making the UMD CDN bundles as esbuild doesn't support UMD. You can see how that is accomplished here.

Tomorrow I'm finishing up some really neat additional documentation of the CDN bundles that is co-hosted with the main Node distribution API docs. There are full types for the CDN bundles and package.json should be configured correctly for ESM CDNs to pick up the right bundle which means Deno should work, but remains to be tested as that requires releasing the package. After the CDN docs are done the only remaining maintenance task that is a nice to have is to add ESLint / .editorconfig to the party. So probably tomorrow or Tuesday for 100% completion.

Just dropping this here for the UMD / RequireJS proof of life (the long tail is supported):

[toji]

Sorry if I'm having a bit of trouble keeping up. Been busy with other things. Also I'm not going to be available for reviews for most of the upcoming week, so I promise that lack of feedback isn't due to anything on your end. 😄

Updated docs are much easier to parse, thanks! Swizzles still take up a ton of page space, but at least their far easier to navigate around now. I do think adding @category to all the methods like you mentioned is probably the right thing to do long term, but that can easily be handled in a follow-up!

I'll say up front that if UDM can be supported with minimal effort then that's great, but I'm actually not too concerned about supporting it if it proves to be a maintenance burden or imposes design decisions or dependencies we'd rather not have. I don't think it's used much in modern JS development and would rather focus on making the more common, recommended paths work well.

I do view Deno and Bun support as higher priority since they're becoming more popular, but my understanding is that it should take minimal to no changes vs. node support? Fingers crossed that that's the case. 🤞

In respect to the bundling, I think it's a good idea to have the various bundles available, but I'm also not sure if I understand why the separate files can't be made compatible with both browsers and node? Specifically, I don't think I understand the benefit you mentioned with the absolute paths when it comes to using the 64 bit vs 32 bit version of the lib? From looking at the folder structure it seems like both of them could access the common files by using the same relative path, since they reside in the same relative depth in the directory tree.

Finally, a couple of quick notes:

• Yes, ESLint is a good idea.
• Wasn't aware of .editorconfig, but sounds nice!

Sorry I don't have time for more in-depth feedback, but again I'm really impressed with this effort and think it should be ready to merge in as beta.3 after a couple of relatively minor details are sorted out! Thank you!

[typhonrt]

Sorry if I'm having a bit of trouble keeping up. Been busy with other things. Also I'm not going to be available for reviews for most of the upcoming week, so I promise that lack of feedback isn't due to anything on your end. 😄

No worries. I spent the last week finishing the "2nd half" / CDN Bundle configuration, optimized the build process, improved the API doc support, and got ESLint v9 and latest TS ESLint support all hooked up and source in line with it.

Here is the latest video covering the work in the last week that provides clarity / answers questions:
https://www.youtube.com/watch?v=y0J_RkTfrag

API docs are updated. There is placeholder information in the CDN README that I'd be glad to update after beta.3 drops and it can be flushed out with full examples of using ESM CDNs / Deno setup, etc.

Main fork repo:
https://github.com/typhonjs-svelte-scratch/gl-matrix-beta2/tree/glmatrix-next

Updated docs are much easier to parse, thanks! Swizzles still take up a ton of page space, but at least their far easier to navigate around now. I do think adding @category to all the methods like you mentioned is probably the right thing to do long term, but that can easily be handled in a follow-up!

I already got the @category aspect sorted. So in the next week or so I'll be updating typedoc-pkg to use TypeDoc 0.26.x. There are two new features of 0.26 that will make things nicer. When using categories they are now going to be foldable / collapsible in the main body of a particular page. Also there is a new feature to be able to host additional markdown files that will appear in the left-hand side navigation allowing more in depth API usage / tutorials, etc. to be broken out from the main README. I should have this done around the time you get to the code review.

I'll say up front that if UDM can be supported with minimal effort then that's great, but I'm actually not too concerned about supporting it if it proves to be a maintenance burden or imposes design decisions or dependencies we'd rather not have.

It's all good and handled including type declaration support even for the IIFE / global use case.

I don't think it's used much in modern JS development and would rather focus on making the more common, recommended paths work well.

Re: UMD, indeed probably not used much at all these days. I did make a point of highlighting import maps and make sure everything is good to go for the most modern way of consuming ESM via ESM CDNs. The video highlights this use case.

I do view Deno and Bun support as higher priority since they're becoming more popular, but my understanding is that it should take minimal to no changes vs. node support? Fingers crossed that that's the case. 🤞

I think Bun will be more like Node in terms of consuming NPM packages. I did review the Deno documentation more and somewhat recently they did add the capability to load packages from NPM, but ESM CDNs remain the recommended way to work with Deno. Things should be covered for this use case and I'll look into testing everything after a beta.3 release of gl-matrix.

In respect to the bundling, I think it's a good idea to have the various bundles available, but I'm also not sure if I understand why the separate files can't be made compatible with both browsers and node? Specifically, I don't think I understand the benefit you mentioned with the absolute paths when it comes to using the 64 bit vs 32 bit version of the lib? From looking at the folder structure it seems like both of them could access the common files by using the same relative path, since they reside in the same relative depth in the directory tree.

In the build optimization work which is covered in the video the Node / NPM distribution is now fully bundled via ESBuild. This provides clarity that you can't copy individual files and run them raw on your local web server. If you use import maps it is possible, but if you are doing that using an ESM CDN is best. I also show a very easy way for the scant few folks who want to create a custom gl-matrix build to run on a web server that can be added to the API docs / consumption use cases. Check the video here: Custom Bundles.

Finally, a couple of quick notes:

• Yes, ESLint is a good idea.
• Wasn't aware of .editorconfig, but sounds nice!

I got the latest ESLint v9 + latest typescript-eslint support setup and the source & tests fully covered. typescript-eslint was nice to highlight further improvements to the TS code.

Sorry I don't have time for more in-depth feedback, but again I'm really impressed with this effort and think it should be ready to merge in as beta.3 after a couple of relatively minor details are sorted out! Thank you!

No worries again. In many ways the work I did rewinds accrued technical debt and prepares gl-matrix for modern distribution on Node and well beyond, so things should be good for the next 5+ years regarding distribution. I'd be glad to give the Deno / Bun verification aspects a solid run through once beta.3 drops.

The next maintenance task that could be tackled is getting the source code coverage in tests up to 100%. When I loop around to update TypeDoc support to 0.26 next week I'll take a couple of hours of knocking out low hanging fruit there. I think it'd take no more than a day to get this sorted especially with the Vitest UI involved.

But otherwise I do need wrap things up as from my perspective things are in a very sound production grade releasable state.

[rubick24]

Maybe we can use this approach to remove those empty alias funtions in class defination:

export class Vec2 extends Float32Array {
  // ...

  static sub = this.subtract
  static mul = this.multiply
  static div = this.divide
  static dist = this.distance
  static sqrDist = this.squaredDistance
  static mag = this.magnitude
  static length = this.magnitude
  static len = this.magnitude
  static sqrLen = this.squaredLength
}

export interface Vec2 {
  sub: typeof Vec2.prototype.subtract
  mul: typeof Vec2.prototype.multiply
  div: typeof Vec2.prototype.divide
  dist: typeof Vec2.prototype.distance
  sqrDist: typeof Vec2.prototype.squaredDistance
}

;(
  [
    ['sub', 'subtract'],
    ['mul', 'multiply'],
    ['div', 'divide'],
    ['dist', 'distance'],
    ['sqrDist', 'squaredDistance']
  ] as const
).forEach(v => (Vec2.prototype[v[0]] = Vec2.prototype[v[1]] as any))

[rubick24]

I also think it would be better if the return type of static method like copy can be extact Vec2 rather than Vec2Like when the out paramter is Vec2:

static copy<T extends Vec2 | Vec2Like>(out: T, a: Readonly<Vec2Like>) {
  out[0] = a[0]
  out[1] = a[1]
  return out
}

const x = Vec2.copy([0, 0], [1, 2]) // x: [number, number]
const y = Vec2.copy(Vec2.create(), [1, 2]) // y: Vec2
y.normalize()

[typhonrt]

I also think it would be better if the return type of static method like copy can be extact Vec2 rather than Vec2Like when the out paramter is Vec2:

This seems like a reasonable change for the methods that it applies to. Before I make such a change to beta 3 it would be great to get input from @toji.

[typhonrt]

@toji et al,

Alright.. I didn't disappear. After a short summer OSS break I set myself out to thoroughly update my TypeDoc theme. IE the Default Modern Theme (DMT). It's a theme augmentation that takes the generated output of the default TypeDoc theme and significantly improves features, UX, and accessibility. Just like my hope / proposed beta.3 for gl-matrix I am in discussion with the TypeDoc maintainer to mainline the DMT, so everyone can benefit that uses TypeDoc.

TypeDoc 0.26.x added some new features that are useful, but IMHO they aren't particularly complete or fully baked from a UX perspective. It took me the last 5 weeks full time to get the DMT updated, but I significantly improved these features and more. The two features that are most useful for gl-matrix are the ability to add additional Markdown files to the API docs and folding of doc content sections. Both of these features the DMT improves. The content folding is particularly relevant to gl-matrix and something that Toji was concerned about especially with the swizzle accessors being visible in the docs.

Please see this video to get a side by side comparison between the DMT (on the left) and the default TypeDoc experience (on the right):
https://www.youtube.com/watch?v=MFJW6FxfvCg

I strongly believe that a package like gl-matrix needs A+ API docs. With the work I did over the summer on package re-organization / maintenance and the API docs progress I'd really like to think things can be greenlighted soon and a beta.3 release made. I'd be more than happy to follow up on any of the CDN details and flush out that side of the docs too.

You can view a live version of the API docs here:
https://typhonjs-svelte-scratch.github.io/gl-matrix-beta3-dist/index.html

Do check out the previous post above as you can try out the proposed beta.3 release now.

For those interested in producing similar API docs like I have for gl-matrix all you really need is typedoc-pkg as it combines everything into one zero / low config CLI; all of my TypeDoc related packages are here:

• @typhonjs-typedoc/typedoc-pkg
• @typhonjs-typedoc/typedoc-theme-dmt
• @typhonjs-typedoc/ts-lib-docs

I just updated ts-lib-docs for complete coverage of all built-in TS 5.6.2 lib declarations as well.