Consider Helper-ifying Shapes and Selections

[spencer516]

Warning: this might get pretty long. Sorry!

One of the reasons why I'm excited about this project is that I think many of us have been looking for a good intermediate abstraction between D3 and highly configurable components (as you nicely put in your talk at EmberConf!).

And, some of the failings, as I see it, of some of the other attempts (some of which are my own!) end up hiding some of the critical d3 stuff inside functions that can't be easily overridden or composed.

In Mike Bostock's splitting of d3 into a number of separate modules, I think it would be in our best interest to respect those modules and bring them into the maximum-plaid ecosystem in a composable way. So to demonstrate what I mean, I'm going to go through some theoretical steps in how to progressively enhance a template-driven visualization from something that moves through the levels of abstraction as I see them:

First step: No d3 at all.

<svg width={{width}} height={{height}}>
  {{#each people as |person|}}
    <rect 
      x={{person.name}}
      y={{subtract height person.age}}
      width={{divide width people.length}}
      height={{person.age}}
      />
  {{/each}}
</svg>

The next logical step, in my mind, would be to add scaling functions as x={{person.name}} simply won't work anyway. (This is going to be lispy-as-shit, but we'll move away from that in later steps).

<svg width={{width}} height={{height}}>
  {{#with (hash 
      xScale=(band-scale people (append 0 width))
      yScale=(linear-scale 
        (append 0 (max people 'age')) 
        (append 0 height) 
        accessor='age'
      )
    ) as |scales|}}
    {{#each people as |person|}}
      <rect
        x={{compute xScale person}}
        y={{subtract height (compute yScale person)}}
        width={{compute xScale.bandwidth}}
        height={{compute yScale person}}
      />
    {{/each}}
  {{/with}}
</svg>

So far, we've used some of d3 somehwat directly through ember helpers. (Using, for now, ember-d3-scale)

But, this is the real world and that template is really hard to read. Now, this is where I think maximum-plaid (or any other addon) can come in: an abstraction on top of scales well-suited for bar-charts. The above could become something like this:

<svg width={{width}} height={{height}}>
  {{#plaid-bar-scales height width people y-accessor='age' as |barDataMaker|}}
    {{#each people as |person|}}
      {{#with (barDataMaker person) as |barData|}}
        <rect
          x={{barData.x}}
          y={{barData.y}}
          width={{barData.width}}
          height={{barData.height}}
        />
      {{/with}}
    {{/each}}
  {{/plaid-bar-scales}}
</svg>

SIDENOTE: This bar-scales component is yielding a helper. This really isn't possible in Ember today. Certainly could be some workarounds...but, to me, it looks like a good use case for a new primitive: the helper-macro. (See this tweet from Edward Faulkner)

So far, we've still only really used d3 scales, but we have much more expressive capabilities and flexibility. Adding labels on top of the bar, for example, would be trivial. And because the actual rendering parts are still just plain-old Handlebars templates, I think it's relatively easy to reason through.

The next step that we might be inclined to take would be to convert the rect into a d3 shape. If we were to make this conventional based on the shape of barDataMaker, then the template above could become:

<svg width={{width}} height={{height}}>
  {{#plaid-bar-scales height width people y-accessor='age' as |barDataMaker|}}
    {{#each people as |person|}}
      {{plaid-rect (barDataMaker person)}}
    {{/each}}
  {{/plaid-bar-scales}}
</svg>

This solution, so far, works. But, those of use who love d3 love the animation capabilities that are enabled by d3-selection. I would like to see this be something that can become a drop-in-ish replacement for the each helper.

So, a simple d3 selection that would support transitions might look like this:

<svg width={{width}} height={{height}}>
  {{#plaid-bar-scales height width people y-accessor='age' as |barDataMaker|}}
    {{d3-selection people 
      transition=(d3-transition 'elastic' duration=200)
      shape=(plaid-rect data-builder=barDataMaker)}}
  {{/plaid-bar-scales}}
</svg>

But, we may also want control over the enter and exit states of the bar. For that, we could add enter-shape and exit-shape properties that determine what the shape looks like when animating in and out:

<svg width={{width}} height={{height}}>
  {{#plaid-bar-scales height width people y-accessor='age' as |barDataMaker|}}
    {{d3-selection people 
      transition=(d3-transition 'elastic' duration=200)
      enter-shape=(plaid-rect data-builder=barDataMaker height=0)
      exit-shape=(plaid-rect data-builder=barDataMaker height=0)
      shape=(plaid-rect data-builder=barDataMaker)}}
  {{/plaid-bar-scales}}
</svg>

This d3-selection could alo be abstracted away in the plaid library so that we end up with a template that looks something like this:

<svg width={{width}} height={{height}}>
  {{#plaid-bar-scales height width people y-accessor='age' as |barDataMaker|}}
    {{plaid-bar-chart data-builder=barDataMaker data=people fill=(cat-color-scale 'rainbow')}}
  {{/plaid-bar-scales}}
</svg>

Obviously a bar chart is fairly trivial and probably doesn't address many of the uses cases for a visualization libarary. But, hopefully, some of the helpers described in here and the ideas for how to compose a visualization together might be a good conversation starter on how to move the work on maximum-plaid forward.

I'd love to hear some thoughts. :)

[ivanvanderbyl]

@spencer516 thanks for the awesome work laying out these ideas. I'll put together a reply in a few days once I've fully decompressed from EmberConf 😄

[spencer516]

Awesome. Yea, I think there's certainly a lot to consider. The thoughts I had here also led me to start another conversation here.

[ivanvanderbyl]

Okay sorry about the delay with this, it's been a hectic few weeks since EmberConf, but I've had some time to digest this and have some interesting conversations with other people working with charts. There's definitely a lot of interest in this like you said 😄

Yesterday I played around with rendering to canvas and SVG using the same component (Check the feature branch), it works quite well, but it does mean we'd need components which work with and without a tagName, which I think we can work around by creating tag-less wrapper components. The reason for this is when rendering to canvas, we create a <canvas> instead of <svg>, then get a 2d context on that canvas and pass it down to each component. If a context is provided, that component just doesn't render a template, and instead applies all transformations to a canvas context. All of the d3-shape primitives support .context(canvasContext) on their constructors. A side note: Canvas render performance is insane compared to SVG.

After considering your approach, I think we should provide a higher level abstraction over SVG and D3, and instead think of this in terms of a grammar for composing visualisations. This way we're not coupled to the primitives of SVG and rendering to canvas always remains possible.

To achieve this we need to identify the elements and guides we want, and how they can be composed. I've expressed some thoughts about this in ARCHITECTURE.md, which is still mostly incomplete but I thought I'd share so you can give feedback/throw it away/start again or whatever. It was just some ideas which came to mind at 3am last night.

I spoke to Taras, and he's working on something similar to your ember-d3-scales, except with helpers for literally everything in D3. It's very tightly coupled to D3, in a way which I'm not that keen on, but it does prove that it can be done. I also like the idea of doing as much as possible in the template as it means less generating components, which I'll admit is a bit of a hangover from earlier Ember and legacy code I evolved. I think the helper story is really just starting to take shape as people realise some useful conventions. Thanks by the way for putting together that thread on developing a proper helper story, that was interesting to read.

I've also been thinking about what it means to make this expressive, and what sort of charts we should aim to be able to produce. Mike has put together quite an extensive list of things which hold a pretty good benchmark in my mind https://bl.ocks.org/mbostock

Anyway, a few lessons I've picked up on so far:

• didRender hook is really slow, it fires quite later after the render buffer is cleared, which leads to graphics which flash between states. A better way is using scheduleOnce('render', ...) which will apply any additional stuff to our visualisation immediately after the element has rendered.
• Interaction will probably require its own primitives. It's easy to do a cursor component which applies markings for the currently hovered points on a line chart, but this doesn't apply well to bar charts, or pie charts. So instead I was thinking we'd have a wrapper component which specifies a type of interaction on everything below it, and then the elements we're rendering provide their own hints to that interaction block about how they respond to interaction. I know that's vague, but I'll put together an example soon.
• As a rule, we should aim to use CSS for all presentation styling, with exception to anything where styling is encoded by data, such as the categorical colour of a line or bar. But things like the text size of axis or line stroke can be done with CSS.

[spencer516]

Awesome.

First question: what was the conclusion with Taras? Is he planning to work with this project or is he planning on doing his own thing? (Funnily enough, I'll be chatting with him this evening, so I'll bring this up).

I think that Architecture document is a great idea and a great start. I think you're right that in terms of benchmarks, being able to reasonably reproduce each of those examples is a good way to think about it.

Do you think that it's worth maybe actually trying to hash out what the implementation might/could/should look like for a healthy cross-section of those examples? Maybe a separate repository of theoretical examples? (That could turn into actual examples later, ala bl.ocks.org)

[ivanvanderbyl]

So our main conclusions from my chat were: We should flesh this out and see where it gets and touch base in a month or so, and he'll do the same with his approach, essentially using it in anger :)

I've created a blank repo https://github.com/ivanvanderbyl/maximum-plaid-gallery to produce examples. I was thinking about maybe making it work similar to blocks or twiddle, essentially just loading from gist. But maybe something for later on, right now I'll just deploy that to maximum-plaid.com and throw up some examples.

[spencer516]

Interesting. I was certainly hoping that after your talk at Ember conf, more people would want to jump on board the Maximum Plaid train. (You have a logo! Which, you know, is pretty much all that matters 😉 ).

If I wanted to open up some issues with implementation thoughts/ideas through specific use cases, would you want me to open a PR against the gallery even if it wouldn't actually work yet? Just trying to figure out where the best place is to put sketches/ideas if I have any as more of this is fleshed out.

[ivanvanderbyl]

That'd be awesome if you want to create some issues, at least to have some discussions and track where we get with implementing them.

I'd like to start with some line charts and bar charts as they cover a lot of ground initially. And there's quite a lot of variation between different ideas. Area charts as pretty similar to line charts from an implementation perspective. When we start talking about stacked/stream layouts with areas is when things get interesting, as they require data transforms specific for that purpose. I guess pies are similar. So maybe we have a layout component which is responsible for translating x,y values for multiple series into something which a stack layout can handle.

[BrianSipple]

I already chatted briefly with @ivanvanderbyl, but I'm just checking in to note my interest in getting involved across ember-cli-d3-shape and maximum-plaid as well. (Apologies for being a few weeks late 😄). I'll try to dig through what we have so far over the next few days to get a feel for where I might be able to start helping, but @spencer516, this is definitely some great insight in its own right. Much appreciated!