[WIP] Observable update refactor

[SimonDanisch]

This PR will take a while to materialize, but I already wanted to start thinking this through.

Refactor how internal events are handled

Motivation

Right now we heavily use Observables to propagate events internally, right up to the GPU memory.
A single scatter plot (with axis to be fair) creates 4403 observables:

using GLMakie
start = Makie.Observables.OBSID_COUNTER[]
f, ax, pl = scatter(1:10)
id_end = Makie.Observables.OBSID_COUNTER[]
Int(id_end - start) # 4403

Further problems with our observable infrastructure:

• updating observables is not thread-safe and kind of slow because it handles all events by iterating a listener array and calling all functions via invoke_latest.
• it's not possible to update multiple attributes efficiently, without double updates, and without running into resizing problems (e.g. when adding new points to scatter with new colors). The double updates are especially bad for WGLMakie over a slow network
• CairoMakie doesn't need any of this, so it's pure overhead
• We need to track a lot of ObserverFuncs to unregister them when a plot gets freed, which is also expensive
• Observable creation alone is relatively expensive (up to 20% of plot creation time)
• GLMakie is especially bad, because it registers an additional observable for every attribute, to check if any attribute has changed for the on demand renderloop
• We could quite easily achieve thread safety for Makie, if we introduce a lock for setproperty! and make the attribute conversions pull based. This should also work quite easily for GLMakie, since it will pull the updates from it's own event thread/task, which doesn't care about where the update came from.

Proposal

Get rid of all observables in Plot

instead have two Dicts and a Set of changed fields:

user_attributes::Dict{Symbol, Any}
computed_attributes::Dict{Symbol, Any}
changed_fields::Set{Symbol}

Now, when accessing any of these in the backend, we will go through all changed_fields and run any convert_arguments/convert_attributes, that relies on that value and will write all values to computed_attributes.

isempty(changed_fields) can also be used as an unexpensive check for GLMakie's on demand renderloop, and in setproperty! we could also already tell the parent scene that there are changed plots.

We can also still make setproperty! directly notify the backend by having an observable/condition triggered in setproperty!.

I would also remove the args vs kw thing and make them all attributes, if that doesn't make to many problems.

Since we have the mapping from arg_n -> attribute_name, it should be possible to make this backward compatible.
We will also add Dict{Symbol, Observable} to have plot.attribute still return an observable for backwards compatibility, but I think we should not rely on this in Makie and the backends itself, so that we don't end up with all attributes always materializing back as observables.

Problems

• calculated_attributes! depends on connecting the calculations via observables. This is a mess, but I also am not 100% sure how to do this going forward. Same goes for convert_arguments, but I kind of hope without observables the code should actually become easier - but it's still a bit hard to judge.
• Axis/Dim converts rely on observables - this was already a difficult decision back when I wrote it, but Observables made it very easy to have dim convert local state:

function dim_convert(..., data::Observable)
    local_state = copy(data[])
    on(data) do new_data
        # the simplest local state required for dim converts
        # This is harder to implement,
        # if `dim_convert` gets called for any change instead.
        if new_data != local_state
            ...
        end
    end
end

• We still kind of need a compute graph if we don't want to run everything every time (e.g., which functions to run if argument 1 changed, or if only one color changed). I almost think this can be somewhat hardcoded, since we only have convert_attributes, convert_arguments and calculated_attributes! that need to run on each update.
• plot!(... ; other_plot.attributes...) wouldn't propagate the updates anymore. plot!(recipe_plot, ...; color=recipe_plot.color) neither. I feel like this needs a cleaner API anyways, but simply forwarding the observables here surely is simple from the users perspective. Instead of the above Dict{Symbol, Observable} for backwards compatibility, we could also return something like color=AttributeView(other_plot, :color), which the plot constructor connects internally to the other plot object (plot.color isa AttributeView). AttributeView would have the large problem, that on(f, plot.color) won't easily work, which must be widely used across the Makie ecosystem.

[ffreyer]

Some thoughts from prototyping:

user_attributes::Dict{Symbol, Any}
computed_attributes::Dict{Symbol, Any}
changed_fields::Set{Symbol}

It would be good to extract this into a separate type, e.g. UpdatableAttributes in my prototype, rather than adding this machinery to Plot directly. That way it can be reused for Scene, Axis, etc if it is compatible. For recipes/attribute pass-through/attribute connections between plots it may also be useful, since it enables creating and connecting attributes before creating the plot object. (Like now)

Dense computed_attributes are probably not needed. For recipes a parent attribute change can directly propagate to the child plots inputs. For primitives an update can directly write to a backend variable. We probably do need store some intermediates though, e.g. converted args, calculated colors, projected positions (e.g. hvlines). These could be marked as protected or private, i.e. not autocomplete, not be (easily) setable but still getable and be kept in the same dict as user attributes (or be split off).

Some attributes are irrelevant to the backend for the plot they are attached to and thus shouldn't trigger updates. E.g. a legend label with no legend present, or inspectable which is queried when needed. If it doesn't have an update function, we don't need to resolve its updates. (This can be resolved at update-function-insertion time)

For recipes/attribute connections to work the backend needs to know when anything has updated. We can either:

1. Check every object for updates every (potential) frame (to avoid redundancy this probably needs to collect all the objects, rather than check parents)
2. Make change-notifications trickle down so the primitive is seen as "changed". This needs to happen immediately (not on update resolution). I'm not sure if these would align themselves to the scene/plot tree, and if a plot can only have one parent. Pseudocode:

function setproperty!(obj, ...)
    ...
    foreach(parent_update_notification, obj.children)
end

function resolve_updates!(obj)
    foreach(resolve_update!, obj.outedated_parents)
    empty!(obj.outdated_parents)
end

Connecting Observables could be difficult because of the pull to push transition. If we make sure that every UpdatableAttributes object connects to the backend then I think it shouldn't be a problem. In that case we can just register update functions, which would then propagate an update request to the backend when they need to be called. If this connection does not exist we would probably need to add an immediate-update option.

[ffreyer]

Notes from Discussions with Simon

My prototype goes in the direction we want to go, but we are aiming to do things more incrementally.

The potential goals of this refactor are:

• allow simultaneous updating of attributes and/or arguments
  • to fix synchronous update problems (e.g. resizing x, y causing BoundsErrors)
  • to reduce update duplication (e.g. set x, y resulting in 2 updates)
• skip redundant updates, i.e. only pull updates once a frame
• reduce number of Observables in general (and avoid invoke_latest)

Initial Plan

For plots we are looking to implement an update function:

function update(plot::PlotType; attr...) # also args, e.g. x = ...,
    plot.updated = union(plot.updated, keys(attr))
    foreach((k, v) -> plot[k].val = v, pairs(attr))
    plot.dirty[] = true
end

This function sets multiple attributes and/or args to new values based on the given kwargs. The changed names are tracked in plot,updated. After setting them all, plot.dirty triggers an update resolution:

function resolve_update!(plot::PlotType)
    args = get_args(plot.changed)
    attr = get_attr(plot.changed)

    if !isempty(args)
        plot.calculated[:converted] = convert_arguments(plot)
    end

    calculated_attributes!(plot) # multi input

    for k in attr
        plot.calculated[k] = convert_attribute(plot, k) # single input
    end

    return
end

There are still a bunch of questions here, e.g.:

• Can updates chain? (E.g. maybe something like arg -> converted -> transformed -> boundingbox)
• What do we do with calculated_attributes?
  • remove the function and fully inline them?
  • Do they run before or after convert_attribute?
  • If before, should they run convert_attributes as a side effect and remove the relevant convert_attribute functions from the set?

The backend would react to dirty in the next step, updating its uniforms, textures, etc. Finally changed needs to be cleared and dirty should be reset:

listeners(plot.dirty)
  resolve_update!(plot)
  <backend>_update!(plot)
  ...
  cleanup!(plot)

For GLMakie (and WGLMakie?) it may also be useful for plot.dirty to propagate up so we can check root_scene.dirty for the on-demand renderloop.

For compatability with the current observables, Attributes may automatically trigger update!().

getindex/getproperty should probably return from the input attributes so that users get what they set

Initial Version and Future Extensions

We are planning to implement this for one or a few primitive plot types at first. That way we can probe how these ideas work out, what falls into place and what doesn't.

In the future we can:

• extend this interface to all primitive plot types
• extend the interface to recipe plots, i.e. all plots
• adjust GLMakie (WGLMakie?) to be pull based, i.e. check plot.dirty from the rendertask and trigger its listeners if needed
• move to a less hardcoded system like my prototype (i.e. with a vector of (inputs, callback) instead of hardcoded resolve_updates!() function)