WIP: adding themes

[sglyon]

This is request for comments regarding adding styles to plotly.py (another feature from the PlotlyJS.jl library I'm missing on the python side).

The docstrings In this PR have more details (as does the doc section linked to above), but the main idea is that a style defines default arguments for figure attributes. This can happen either for layout attributes, attributes for traces of a specific type, and attributes for traces of any type.

For example, suppose we have this plot:

import plotly.plotly as py
import plotly.graph_objs as go

def make_fig():
    # Create random data with numpy
    import numpy as np

    N = 100
    random_x = np.linspace(0, 1, N)
    random_y0 = np.random.randn(N)+5
    random_y1 = np.random.randn(N)
    random_y2 = np.random.randn(N)-5

    # Create traces
    trace0 = go.Scatter(x = random_x, y = random_y0,
        mode = 'markers', name = 'markers'
    )
    trace1 = go.Scatter(x = random_x, y = random_y1,
        mode = 'lines+markers', name = 'lines+markers'
    )
    trace2 = go.Scatter( x = random_x, y = random_y2,
        mode = 'lines', name = 'lines'
    )

    data = [trace0, trace1, trace2]
    layout = go.Layout(xaxis={"title": "My xs"}, yaxis={"title": "this is y"})
    return go.Figure(data=data, layout=layout)

The plot would look like this with no style applied:

If we do make_fig().apply_style(go.STYLES["ggplot"]) we get:

Or make_fig().apply_style(go.STYLES["seaborn"]) gives:

Or make_fig().apply_style(go.STYLES["fivethirtyeight"]):

Before merging

I haven't totally finished this code as I wanted to present the concept and get some feedback (even an 👍 or 👎 would be great).

I think I will need some guidance from the team on the best way to incorporate this feature. Here are some questions for you:

• Does this make sense to be a method on go.Figure?
• Should we allow users to set a default style for a python session. Something like matplotlib does with their plt.style.use command? The default for this setting would, of course, be an empty style.
• If the answer to the above is yes, then I think it makes sense to allow styles to be context managers that apply to all figures created in within the context
• If there is a notion of default style, when should it be applied? On the Julia side I chose to apply the style when the figure is encoded as JSON. That was the "last possible moment" to apply the style (before viewing or sending to plot.ly cloud), giving users the longest time possible to tweak their plots before the style is applied.

Things I need to do before finishing this PR:

• Tests
• Examples to plotly docs? Not sure if that needs to happen before a merge. It would be awesome if some of the plot.ly team could pitch in with that
• Resolve any issues/suggestions that come up in response to the above questions.

---

EDIT 2-6-18

I've changed the name style to theme in the code, so you would need to make the same change in order to run the examples above

[jackparmer]

👍 Definitely a long-overdue feature for this library IMO.

A few suggestions from me:

• We've historically used the phrase "themes" instead of "styles" in other parts of the platform.
• @tarzzz sketched out a Python themes implementation idea a few years back, you might be interested in reviewing this: Introducing Templates/Themes #425.
• Storing one JSON object per theme as a file makes more sense than having one function per theme (see Tarun's thread above). It will make themes easier to add, edit, extend, and share. You could post a gist of a new plotly theme for example and use themes across languages.
• Fortunately, Plotly's server already has a schema for storing themes, it has been battle-tested over a few years so I would recommend considering this:
  https://github.com/plotly/streambed/blob/master/shelly/themes/README.md †
• Here are a few example themes that are serialized this way:
  https://github.com/plotly/streambed/blob/master/shelly/themes/default_themes/theme_templates/duluth.json †
• Bonus: Write a function that takes a figure object and saves it as a themes file. See: themes and default styling #217
• A few other theme ideas: https://github.com/plotly/streambed/issues/4306 † - I can help add these after this is merged.
• Themes should feel consistent across trace types - not just line and scatter, but maps, 3d plots, polar charts, etc.
• https://github.com/jmmease/ipyplotly has the ability to update figures in situ. It would be great if themes were compatible with this. For example, you could flip through a few themes in a figure without generating multiple copies. @jmmease will have a better idea if the approach outlined so far will be compatible with ipyplotly

† Note: streambed links above will only work for GitHub users with streambed access

Does this make sense to be a method on go.Figure

I think so for now. @jmmease will have a better idea about future proofing for ipyplotly

Should we allow users to set a default style for a python session.

Yes

Examples to plotly docs?

Plotly can definitely take care of the docs side after this is merged.

Happy to take another look at this after themes are serialized in files.

/cc @tarzzz @jmmease @chriddyp

[jonmmease]

Some thoughts with respect to the ipyplotly functionality coming to plotly.py. There are two approaches that I can think of that would work consistently with this new paradigm.

1. Themes are transformations of the JSON structure that are explicitly applied after a Figure is constructed. Just like the fig. apply_style method in this PR (if I understand it correctly). Here the end result of applying the theme is identical to the result of the user manually setting all of the themed properties. Since the theme is a transformation, there is no notion of a figure having a theme. And so the application of the theme doesn't affect the style of traces or axes that are created after the theme is applied.

2. Themes are JSON specifications that are passed to the Plotly.js side as properties of the top level figure. Plotly.js uses the theme specification to decide on defaults for unspecified layout/trace properties. ipyplotly already supports transporting these default values back to the Python so that should work naturally. In this case, the figure has a theme and the theme does influence the style of new traces and axes as they are created.

Supporting global themes and theme switching would be straightforward in approach 2, but I think they could be problematic for approach 1. The issue is that with the ipyplotly updates it will be possible to create a Figure with some initial data and then display it. And then properties can be updated and traces can be added to the Figure and the updates will be immediately reflected in the display. So there's not really a "last possible moment" anymore during which to apply the theme. Feel free to let me know if this isn't clear and I can elaborate more.

Has there been any discussion on the Plotly.js of setting default properties based on a theme specification?

[jackparmer]

I like thinking of themes as transformations that take a Plotly figure and a themes object as input. Maybe fig.apply_theme() should return a new fig instead of mutating it in place? E.g.

themed_fig = fig.apply_theme(theme_obj)

This way, when we migrate to ipyplotly, a user has to be more intentional about applying a new theme.

Has there been any discussion on the Plotly.js of setting default properties based on a theme specification?

A new top level Plotly.js method such as themedFigure = Plotly.theme(figure, themeObj) would be the ideal way to do this, but probably isn't on the near-term Plotly.js checklist unfortunately, just because of other priorities. Themes will likely be a case where Plotly.py leads a new beta feature which Plotly.js later implements in a language agnostic way. Plotly's Chart Studio once supported themes saved in this serialized format, however, so I think Plotly.js could adopt the same or similar theme format in the future, after Plotly.py blazes the trail.

[sglyon]

Wow, thanks for the comments! Looks like I stumbled up on a topic with some history.

I will take a little time to review past attempts/how the themes are stored on streambed.

But, I do have some comments now

---

@jackparmer: definitely a good idea to have one theme per json file. I wrote as functions while I was testing and haven't take the time to move them to json files. It will be very easy

---

@jackparmer Do you think that the former JSON representation of themes is the way we will ultimately want to do it?

While fairly similar, there are a few differences between the two representations (I'll refer to the one here as "style" and the other one as "theme" -- even though the eventual name will be theme). Here's what I can think of so far:

• In themes, I like the idea of the reference section where we can define reusable components. Brilliant! This should definitely remain in the final theme implementation
• In styles, I have the notion of global_trace, which are trace attributes that are applied whenever possible. For example, setting marker.color there and applying the theme will set the marker.color on all trace types that support a marker color. I think this goes a long way towards your point that "Themes should feel consistent across trace types - not just line and scatter, but maps, 3d plots, polar charts, etc."
• In themes there seems to be special treatment of plotting axis. In styles I just had all axis updates be associated with the layout object. I like the simplicity of not having to learn what cart_axis_update is (when I already know what layout is), but the cost is repeating things for xaxis, yaxis, and zaxis. With the reference section this shouldn't be as much of a problem though (define one axis and apply it to those three properties of the layout).

I think a first step towards an implementation that is mergeable will be reaching consensus on JSON representation of themes.

---

@jackparmer @jmmease

This implementation of themes came from PlotlyJS.jl where we have long had the ability to update figure attributes in place (e.g. display a plot then restyle markers to be red or add a new trace).

I'll talk through how I handled it there and hopefully we can iterate on ways to improve it so we can find the optimal way to apply this in the ipyplotly system.

First, here's where information about styles is stored:

• There is a PlotlyJS.DEFAULT_STYLE that defaults to an empty style, but can be changed by an environment variable
• There is also PlotlyJS.CURRENT_STYLE that starts off as the DEFAULT_STYLE but can be set by the user at runtime to be an arbitrary style. Resetting CURRENT_STYLE will return it to DEFAULT_STYLE
• Plot objects also have a style, which defaults to CURRENT_STYLE when the plot is created, but can be set with passing an instance of PlotlyJS.Style to the style keyword argument upon Plot construction

Now on to style application

• When: styles are applied when the plot is encoded as JSON. In a world where plots are live-updating, this means that styles are applied before the first time it is displayed. This is what I meant by "last possible moment above" -- the last moment before the plot is displayed
• How: because style is an attribute of each plot I can just look up the plot.style attribute and apply it
• What: style attributes are default attributes for plots. To apply a style we recursively go through the style attributes and set them on the plot (traces or layout), only when that attribute is not already set on the plot.

What I am missing here, as @jmmease hinted at, is how styles should be applied to e.g. new traces added to the figure after display. In my current Julia implementation that doesn't happen, which is a bug. Given that I have style attribute on the plot, however, it would be straightforward to apply the style to the new trace before placing it on the figure (I call addtraces!(my_plot, new_traces) so my_plot.style is known at this point and I should be able to apply the style to the new traces.) If in ipyplotly, figures had a theme attribute, we should be able to do the same thing.

Note that In this PR I implemented a different system:

• Figures don't have a style
• There is no global default or current style
• You must call fig.apply_style to apply styles -- nothing is automatic

If we keep those semantics, I think @jackparmer proposal to have fig.apply_theme not mutate fig and instead return a copy is worth thinking about. Maybe at that point it is a function, not a method on Figure (e.g. apply_theme(figure, theme))?

Happy to talk more on this

---

[jackparmer]

Ah, I see, I didn't think about the case of re-applying styles. Thanks for explaining this. Seems like a good argument for storing the theme as a top level object in the Python figure versus a function transform. It could be a strange experience to apply a theme, then add a new trace, and the new trace looks totally different. I just want to make sure this is done in a way that is compatible with the imminent ipyplotly migration, but I'll leave that to you and Jon.

Along similar lines, @alexcjohnson and I were discussing theme "completeness" last night and agreed that if a theme is applied, then a 2nd theme is applied afterwards, the 2nd theme should erase all trace of the 1st theme. This was a big drawback of Plotly's first theme implementation in Chart Studio. CS didn't "clean the slate" between themes so you would get weird leftover artifacts when switching from one theme to another.

Do you think that the former JSON representation of themes is the way we will ultimately want to do it?

I'll leave it to you to decide what to keep and what to change or leave out. It's also a few years old and a bit out-of-date. I agree cart_axis_update (and any other new ontology that the user would have to learn) should go. We're likely going to have to change whatever file format we settle on again when Plotly.js eventually exposes a universal Plotly.theme() method.

I'm sure there are other things I'm not thinking through, happy to keep fleshing this out here for as long as needed. To summarize, the 3 things I'd like to see in the next PR if possible:

1. Themes as files
2. Trace addition to themed figures includes the theme transform
3. Theme "completeness", so that switching between themes "cleans the slate" before applying the next theme transform

[jonmmease]

Thanks for the detailed description of how PlotlyJS.jl handles styles @sglyon, that was very helpful. I think supporting styles as you've described will be feasible in the ipyplotly paradigm, and I'm really excited to have them when we're done!

There will be some subtleties to sort through. Here are a few that come to mind:

1. ipyplotly converts property assignments into restyle and relayout commands, so the entire plot is not redrawn on update (is that how JuliaJS.jl handles things?). So we'll need to think through how these commands might need to be modified, based on the current theme, before being applied on the JS side. One example that comes to mind is that assigning a property the value of None on the Python side is converted into a relayout/restyle command that sets the property value to null on the JS side. This unsets any explicitly set property and allows Plotly.js to once again choose the default value. When the Figure has a theme, this relayout/restyle command would need to be modified to change the null value into the value defined in the theme.

2. The JS side of ipyplotly computes the difference between the JSON structure that has been explicitly defined by the user and the fullData + fullLayout properties of the Figure, and then sends this difference back to the Python side. The Python side keeps track of these default properties and presents them to the user upon field access. With themes in the mix, there will be 3 kinds of properties that will need to be considered during field access: Properties the user has set explicitly, properties not set by the user but defined in the theme, and properties not defined by the user or the theme but computed as defaults by Plotly.js. Finding the right design to keep all of this consistent may prove a bit challenging.

3. When the theme for a Figure is altered, I think we'd want to convert this operation into a Plotly.update command. Also, I think we might want some kind of force=True option when applying a theme to have the option of overriding/unsetting the properties that user has specified explicitly.

[jonmmease]

I just noticed that cufflinks (https://plot.ly/ipython-notebooks/cufflinks/) also has its own theme support. I haven't looked at their implementation, but it would be nice if this also unified what they are doing.

[sglyon]

@jackparmer I like your idea regarding theme completeness. I think to do that we'd need access to something like fullData and fullLayout before any themes are applied. Does plotly.js supply this, or can it somehow be included in plotschema.json? If not, I think the system @jmmease is building in ipyplotly should give us the info we need.

I will keep thinking about how we should represent the theme in JSON, but here are some preliminary thoughts:

• Have a reference key that maps to a JSON object mapping any name into any value. This is just a place for theme-creators to store information that might be repeated. For sake of example, suppose in this section I have a key-value pair: "my_colors": ["red", "blue"]
• Have a global key that contains a JSON object with mappings from trace attributes to their values. Extending our example, this section might look like "global": {"marker": {"color": "my_colors"}}, which would instruct the theme parser to look for "my_colors" in the the reference object and replace it with the associated value (so we end up with "global": {"marker": {"color": ["red", "blue"]}}
• Have a layout key whose value is a JSON object prescribing default values for all layout attributes
• Have one section for each trace, where the key is name of the trace type and the value is

This is different from the current format for themes in (at least) the following ways:

• Besides global and reference , all keys in the theme are known plotly "entities" (be it trace names, the layout, or attributes of one of those). There are no cart_axis_update or layout_update keys
• the attributes for specific trace types are key-value pairs at the top level of the JSON object describing the theme, not under the type_templates key.
• The global section will apply the attributes to traces, whenever possible (similar to my implementation in this PR). this helps us easily develop themes that have a consistent feel across all trace types
• When specifying the attributes that should apply to traces of a specific type, you need to list key-value pairs instead of giving a list of keys found in the references section.

Thoughts on this?

---

@jmmease in Julia the notion of dynamic properties only just recently landed in the master branch. Thus the idea of doing plot.data[0].marker.line.color = "red" hasn't really been a valid option in all released versions of Julia. So, what I decided to do instead was invert the relationship between "property setting" and calls to restyle and relayout. Specifically, I encourage users to call restyle!(plot, 1, marker_line_color="red") which will both update the display and set the attribute on the plot object. If I understand correctly you would have plot.data[0].marker.line.color ="red" trigger a call to restyle, which would both set the property and the trigger an update of the display. I think the result is the same (both the property and display are updated), but the entry point is different

You make a good point about setting a property to None. Right now behavior is similar in PlotlyJS.jl (replacing None with Julia's nothing), but doesn't properly account for the presence of a default value in the theme. Good catch, this is something that should be fixed there and implemented properly here.

I think that having ipyplotly track/report the difference between data and fullData (similar for layout), actually makes the suggestion from @jackparmer to have themes satisfy a notion of completeness easier. With that information in hand, you can identify if a value came from the user, from a plotly.js default, or a theme and choose the right behavior when you want "unapply" a theme (or equivalently erase all properties set by the theme).

Great call on themodify a theme ==> Plotly.update.

I haven't looked into the theme support in cufflinks. If what we build here ends up going back upstream into plotly.js then I suppose cufflinks would start to support it also? If that's the case then I think we can look at their implementation for inspiration, aim to build the most sensible theme story here in plotly.py, and then adjust theming in cufflinks to follow suit.

[jonmmease]

It looks like the recently added layout.colorway option (plotly/plotly.js#2156) can now be used to set the default color order for all trace types. I think this is probably the 90% use case for property cycling in themes.

One option would be to not do any property cycling in the theme and just have the theme set colorway directly. Then if we want to add other cycling properties in the future, we push them to plotly.js as new properties analogous to colorway

[jonmmease]

Hi @sglyon,

I just updated master with Plotly.js 1.39.2, which has the new template (theme) support from plotly/plotly.js#2761. With just rerunning code generation it's already usable:

fig = go.FigureWidget()
fig.layout.template = {
    'layout': {'paper_bgcolor': 'lightgray'},
    'data': {'scatter': [{'mode': 'markers',
                          'marker': {'size': 20, 'symbol': 'square'}},
                         {'mode': 'markers',
                          'marker': {'size': 20, 'symbol': 'triangle'}},
                        ]}}
fig.add_scatter(y=[3, 1, 3])
fig.add_scatter(y=[1, 3, 2])
fig

Do you have any suggestions for improving the Python API? Here are some of my thoughts:

• Write a custom validator for layout.template (right now it accepts anything and Plotly.js just ignores invalid properties)
• Expose the Plotly.makeTemplate function to extract a template from an existing figure
• Add a template kwarg to the Figure/FigureWidget constructor.
• Include some way to set a global template that is the default unless if it overridden in the constructor.
• Include a set of predefined named templates like you have in this PR.
• Figure out what's going on with the new name and templateitemname properties that were added to all of the object array properties (e.g. annotations, images, etc.). I think the name properties should only show up in the template hierarchy and templateitemname should only show up in the normal object hiererahcy.

[sglyon]

That's fantastic. I'm happy that this is a plotly.js feature and that codegen opened it up to python right away.

The one additional feature that PlotlyJS.jl has is the ability to merge themes. So I could have one that handles fonts, another for colors, another for XXX -- then I just call a function like mytheme = create_theme(theme_fonts, theme_colors, theme_XXX) and it creates one with all the fields merged together (precedence is given to the last theme to appear in the function argument list.

[jonmmease]

Going to close this and we can pick up discussion on the the templates integration in #1161