Allow passing typed array of ms timestamps for date axis without conversions

[nirvana-msu]

I need to handle a large number of data points, and I also need lines mode (not markers) so I'm using a "fancy" scattergl plot. Annoyingly, there appears to be no way to just feed ms epoch timestamps to plotly directly as a date axis value (say, as a BigUint64Array)... as a result, there's a huge added overhead added for parsing Date inputs (the most effcient thing it seems to accept currently) back into ms timestamps.

There is seemingly a shortcut here:

function dt2ms(v, _, calendar, opts) {
        if((opts || {}).msUTC && isNumeric(v)) {
            // For now it is only used
            // to fix bar length in milliseconds & gl3d ticks
            // It could be applied in other places in v3
            return +v;
        }
        ...

but the opts are not passed through from makeCalcdata call in scattergl - no way to opt-in for this as a user.

TypedArray is currenty allowed for linear and log axis types here, but date axis just falls through to calling dt2ms, and ultimately to the terribly slow (even for Date inputs, when you need to proccess hundreds of thousands points) dateTime2ms. It is so frustrating there is no way to pass through exactly what is produced in the result anyway.

[judfs]

I too noticed that Date formatting was limiting my performance. Did a stress test and saw dt2ms as 44% of the profile.

The following screenshots are the same code with just the axis type changed:

type: "date",

type: "linear",

This is a stress test of a real time plot with a firehose data rate. Eventually my plot updates start breaking and some datapoints fail to get updated. Notice the gaps in the orange trace on the right. I probably have done thing suboptimally, but I think the point holds.

You can see with my demo the linear plot is able to keep up for about 130 (210-80) more seconds and 229,500 (365400 - 135900) more datapoints per trace or 688,500 (_*3) total more points than date.

I would have expected a workaround to be use linear and provide a custom axis format, but that very much seems to not be a thing (#1464)
Tested versions: "react-plotly.js": "2.6.0", "plotly.js": "2.16.1"`

[gvwilson]

Hi - we are trying to tidy up the stale issues and PRs in Plotly's public repositories so that we can focus on things that are still important to our community. Since this one has been sitting for several years, I'm going to close it; if it is still a concern, please add a comment letting us know what recent version of our software you've checked it with so that I can reopen it and add it to our backlog. Thanks for your help - @gvwilson

[dfrtz]

Not specifically asking for this to be reopened, but did want to provide an update (based on the previous comment about if it is still a concern) that this still impacts rendering as as of version 3.0.1. We have some users that can generate graphs with 100k+ datapoints in one of our applications, and even though the timestamps are already in ms format, about 50% of the rendering time is spent on the conversion logic being called from dt2ms since it does not allow controlling the options from "date" type xaxis values.

If anyone comes across this thread, and is looking for a somewhat "hacky" solution to the problem, you can patch this:

ax.d2c = ax.r2c = ax.d2l = ax.r2l = dt2ms;
ax.c2d = ax.c2r = ax.l2d = ax.l2r = ms2dt;
ax.d2p = ax.r2p = function(v, _, calendar) {
  return ax.l2p(dt2ms(v, 0, calendar));
};

with this:

ax.d2c = ax.r2c = ax.d2l = ax.r2l = (v, _, calendar, opts) => dt2ms(v, _, calendar, {...opts, msUTC: true});
ax.c2d = ax.c2r = ax.l2d = ax.l2r = ms2dt;
ax.d2p = ax.r2p = function(v, _, calendar) {
  return ax.l2p(dt2ms(v, 0, calendar, {msUTC: true}));
};

and it resolves the issue. There may be side effects, but I have not seen any yet, so YMMV.

[gvwilson]

@LiamConnors worth adding to docs?