Restructured widget initialization order

[HalfWhitt]

Fixes #2937

As discussed in beeware/travertino#224 (review), I've written this to work with up-to-date Travertino (with beeware/travertino#232 applied), but also be backwards-compatible with Travertino 0.3.0.

The way I've currently set it up, a widget class (in core) has an _IMPL_NAME class attribute that tells Toga what its implementation is named. In all cases here, that's the same as the core/interface class, but this way, subclassing a Toga widget won't suddenly break it.

Could be relevant to #2687, in that I've grouped the factory- and implementation-fetching logic for all widgets into the base Widget init.

PR Checklist:

• All new features have been tested
• All new features have been documented
• I have read the CONTRIBUTING.md file
• I will abide by the code of conduct

[HalfWhitt]

Huh, I hadn't noticed that the Textual base widget (and for that matter the web one too, though that isn't tested here) didn't already call reapply. Now that it's being called from the interface's initializer, it crashes. I currently know nothing about the Textual backend; will have to look more into this.

[HalfWhitt]

Once there's a newer release of Travertino, this will be the unreachable branch... should I preemptively exempt it as well, to make the tests fully forwards-compatible? As weird as it feels excluding both branches from coverage testing...

[freakboy3742]

Yes, it's weird; but this is a bit of a weird situation. As long as it's documented inline why we're not checking coverage for this section, I don't see a problem pre-emptively excluding - and as soon as there's a new Travertino release that we can pin as a minimum, the whole block will disappear.

Also - # pragma: no branch is a better pattern for an else: pass branch.

[HalfWhitt]

A thought occurs to me. Rather than simply change all of TogaApplicator's uses of self.widget to self.node, would you object to a property that aliases widget to node? It would just be syntactic sugar... but it would be nice to keep the more specific / self-explanatory attribute name, since TogaApplicator will presumably only ever be operating on widgets.

[freakboy3742]

I'm not sure I completely follow the motivation for the _IMPL_NAME change - is that a required part of this change, or is this more of a cleanup opportunity that you've identified? If it's a cleanup, I think I'd rather tackle that as part of #2687, rather than have all the code churn on a temporary partial fix.

[HalfWhitt]

Yes and no, I suppose. This whole thing is essentially a cleanup, isn't it? My goal with this PR was to do the Toga half of the reorganization described in beeware/travertino#224. With the reorganization I proposed:

Before

WidgetSubclass.__init__(style)
|   
|   Widget.__init__(style=style)
|   |   
|   |   Node.__init__(style=style if style else Pack(), applicator=TogaApplicator(self))
|   |   |    ...
|   |   |___________
|   |   
|   |   self._impl = None
|   |   self.factory = get_platform_factory()
|   |______
| 
|   self._impl = self.factory.WidgetSubClass(interface=self)
|   |   
|   |   ImplementationWidget.__init__(interface=interface)
|   |   |   
|   |   |   self.interface = interface
|   |   |   self.interface._impl = self
|   |   |   # Unnecessary, since this is already being assigned to self._impl above
|   |   |   
|   |   |   self.create()
|   |   |   self.interface.style.reapply()
|   |   |   # This is where all the styles are actually applied for the first time.
|   |   |___________
|   |_______________
|___________________

After

WidgetSubclass.__init__(style)
|
|   Widget.__init__(style=style)
|   |   
|   |   Node.__init__(style=style if style else Pack(), applicator=None)
|   |   |   ...
|   |   |___________
|   |
|   |   self.factory = get_platform_factory()
|   |   self._impl = getattr(self.factory, self.however_we_store_implementation_name)(interface=self)
|   |   |
|   |   |   ImplementationWidget.__init__(interface=interface)
|   |   |   |   
|   |   |   |   self.interface = interface
|   |   |   |   self.create()
|   |   |   |_______
|   |   |___________
|   |
|   |   self.applicator = TogaApplicator(self)
|   |   # Applicator property will trigger a reapply when set.
|   |_______________
|___________________

Beforehand / currently, each widget subclass calls Node.__init__(), then creates and assigns its implementation. The tail end of the implementation's base __init__ applies the style.

If, as proposed, Widget assigns its applicator at the end of its __init__, then the widget needs to have its implementation before that point; it can't happen after Widget.__init__. I put the implementation creation/assignment inside Widget.__init__ to minimize repetition, but then that means each class needs a way to specify its implementation class name, in a way that:

• Won't break when a user subclasses a widget (which means we can't just use the class name), and
• Allows subclasses to easily override the implementation name if needed (which means we can't supply it as an argument Widget.__init__).

I suppose I could instead try putting it in each subclass __init__, before calling Widget.__init__, and see if it causes any weird problems doing so before the node is initialized.

[freakboy3742]

Yes and no, I suppose. This whole thing is essentially a cleanup, isn't it?

True... except that there are downstream consequences.

In particular, while I can appreciate the "consistent logic" portion of moving the create impl part into the base class, that makes some uses cases difficult (or impossible). The core of the problem is that the widget is no longer responsible for determining how to instantiate it's own _impl... which I'm not sure is a good idea.

For an example in the wild - toga-chart doesn't have an _impl, because it's a composite widget - a wrapper around a toga.Chart.

Another example is a third-party widget like BitmapView - that's a "normal" widget in the sense of it having an _impl and native, but it's a custom widget, so it can't use Toga's factory (at least, not without the rest of #2687); and in the interim, there's no obvious workaround.

I suppose I could instead try putting it in each subclass __init__, before calling Widget.__init__, and see if it causes any weird problems doing so before the node is initialized.

That would be my inclination. There's still going to be a migration issue for subclassed widgets, but at least there will be a viable path for moving forward.

Better still would be to find a way that doesn't have this backwards incompatibility... but I don't have any immediate ideas for what that would look like.

[HalfWhitt]

For an example in the wild - toga-chart doesn't have an _impl, because it's a composite widget - a wrapper around a toga.Chart.

Another example is a third-party widget like BitmapView - that's a "normal" widget in the sense of it having an _impl and native, but it's a custom widget, so it can't use Toga's factory (at least, not without the rest of #2687); and in the interim, there's no obvious workaround.

Thank you for the context. My brain's hurting a little figuring out how and why toga.Chart works the way it does. I would have expected something like that to either subclass Chart, or to subclass something like Box and contain the Chart and/or other widgets; subclassing base Widget while using the _impl of the wrapped Canvas is not something that would have occurred to me. Perhaps it would be worth formalizing or at least documenting the process(es) for making custom widgets... but I guess that's more of #2687's territory, and at some point I have to stop chasing the Matroyshka yaks ever downward.

I've made a first stab at reordering all the _impl creation before calling Widget.__init__. Since I already had to make one tweak in the Cocoa implementation, I pushed the commit here largely just to see if and how the other testbeds crash. Looks like CI hit a snag before even getting that far, though.

[HalfWhitt]

This was getting called when Button's implementation is created, but it presupposes the existence of the button's style — which isn't there until Button calls super.__init__(). It also gets called when the button's font, bounds, or icon are set, and commenting it out here doesn't affect any testbed tests, so it seems it's still getting sufficiently applied. (I'm betting there are a few spots like this in the other implementation layers, though.)

[HalfWhitt]

One thing to note about this arrangement is that it still prevents a subclass of an existing widget from choosing a different implementation. TextInput gets around this by calling self._create() instead of a hard-coded _impl creation/assignment, which lets PasswordInput override _create() to pick its own implementation. I don't think it would break anything existing to add the same mechanism to all other widgets... but if we do so, in order to expose it as a public mechanism for subclassing, perhaps it should lose the underscore and/or get a more descriptive name.

Then again, I suspect this might be replaced entirely by a different mechanism in #2687.

[freakboy3742]

Thank you for the context. My brain's hurting a little figuring out how and why toga.Chart works the way it does. I would have expected something like that to either subclass Chart,

An earlier implementation did subclass Canvas - the problem becomes that the API of Canvas then becomes public API for this new widget (specifically, the redraw method), which isn't desirable. Chart can essentially be thought of as an API façade over the toga.Canvas API.

or to subclass something like Box and contain the Chart and/or other widgets; subclassing base Widget while using the _impl of the wrapped Canvas is not something that would have occurred to me.

This would have been another viable approach - and one that would make sense if there was more than 1 widget being composed. However, while the API surface of toga.Box isn't prone to the same issue as toga.Chart - but it still represents an extra overhead of a widget that isn't actually doing anything (a box that contains... a box).

Perhaps it would be worth formalizing or at least documenting the process(es) for making custom widgets...

That's definitely something worth doing. We haven't formally documented the contract for a custom widget, but if doing so will simplify the work here, then we should consider it.

but I guess that's more of #2687's territory, and at some point I have to stop chasing the Matroyshka yaks ever downward.

That's more than fair :-)

Understanding the direction the widget contract is heading doesn't necessarily mean fully formally defining and documenting it - but I definitely appreciate the desire to close off the scope for this issue at some point.

[HalfWhitt]

That's definitely something worth doing. We haven't formally documented the contract for a custom widget, but if doing so will simplify the work here, then we should consider it.

Understanding the direction the widget contract is heading doesn't necessarily mean fully formally defining and documenting it - but I definitely appreciate the desire to close off the scope for this issue at some point.

I'm not averse to getting into a few more weeds, and it does seem inextricably linked to getting this PR done sensibly.

So far I see five potential paths for making a custom widget:

1. Subclass a container like Box, and include other widgets inside it.
2. Only subclass base Widget, writing an entirely new widget class and implementation.
3. Subclass an existing Toga widget, keeping the same implementation and only modifying the interface level (adding methods, etc.)
4. Subclass an existing Toga widget, and provide a different implementation (like PasswordInput does to TextInput).
5. Wrap an existing Toga widget and self-assign its implementation in order to essentially be it while masking its API, like Chart does to Canvas.

Presuming we want to support all five:

• 1 and 2 are straightforward, as far as supporting them; 1 simply includes internal widgets as-is, and 2 doesn't have any existing specific implementation to interact with.
• 3 and 5 are both reasons we can't infer the class name of the implementation from that of the interface, since these want to use the same implementation as the parent/composed widget.
• 4 means a subclass of a Toga widget needs to be able to override the implementation of its parent class — perhaps the way TextInput does with _create(), or something similar.
  • Satisfying the previous two points was my rationale for adding _IMPL_NAME, but I was overlooking 5...
• I now understand the reasoning for 5, as unintuitive as it was initially. I'm still kind of confused to see that the style is provided to both the widget and canvas initializer... This is both a clearly useful design pattern and also one with a number of odd, probably subtle interactions. That makes me think we should, at minimum, document the heck out of it, or preferably, provide a useable hook that can be trusted to handle such things correctly.

There is, of course, always the option of choosing to explicitly not support one or more of the above, the way PIL.Image explicitly doesn't support inherited subclasses. Not ideal, but worth remembering.

I think I also need to look more into how the platform/factory/implementation selection works in the first place, so I better understand what's being proposed in #2928.

That's all I've got in my head so far. Any thoughts/comments/direction at this point?

[HalfWhitt]

With regards to type 5, with some initial tinkering, pulling the wrapper functionality of Chart out into a separate class seems to work, at least for the included example:

class WidgetWrapper(Widget):
    def __init__(
        self,
        WidgetClass: type[Widget],
        id: str = None,
        style=None,
        *args,
        **kwargs
    ):
        self.wrapped_widget = WidgetClass(*args, **kwargs)
        self._impl = self.wrapped_widget._impl
        super().__init__(id=id, style=style)

    @Widget.app.setter
    def app(self, app):
        # Invoke the superclass property setter
        Widget.app.fset(self, app)
        # Point the canvas to the same app
        self.wrapped_widget.app = app

    @Widget.window.setter
    def window(self, window):
        # Invoke the superclass property setter
        Widget.window.fset(self, window)
        # Point the canvas to the same window
        self.wrapped_widget.window = window


class Chart(WidgetWrapper):
    def __init__(
        self,
        id: str = None,
        style=None,
        on_resize: callable = None,
        on_draw: callable = None,
    ):
        """..."""
        self.on_draw = on_draw
        if on_resize is None:
            on_resize = self._on_resize

        super().__init__(Canvas, id=id, style=style, on_resize=on_resize)

        self.canvas = self.wrapped_widget

(At least, it works as well as the existing code. Resizing the window is identically buggy in both... case in point for this being subtle and hard to get right.)

[freakboy3742]

That's all I've got in my head so far. Any thoughts/comments/direction at this point?

It looks like a good summary of the options/usage patterns to me.

Regarding (5); I agree it's odd, but if locking out that particular design pattern in favor of an alternative approach makes sense, I'm not fundamentally opposed, as long as the broader use case can be satisfied in other ways.

As for your alternate toga-chart - two questions:

1. What's the window resizing bug? That's a new one for me...
2. Is the extra layer required? Couldn't you get the same net effect by setting self._impl = self.canvas._impl?

[HalfWhitt]

2. Is the extra layer required? Couldn't you get the same net effect by setting self._impl = self.canvas._impl?

Huh... apparently that is indeed all you need. What were those app and window setter overrides for? They were in there, so I assumed they were necessary. But taking them out doesn't seem to change anything. I'd already found that there's similarly no need to supply the style to the wrapped widget, either. I guess this is simpler than I thought it was. Except possibly for whatever's causing...

1. What's the window resizing bug? That's a new one for me...

I've submitted it here: beeware/toga-chart#191
Haven't dug into it yet.

[HalfWhitt]

What were those app and window setter overrides for?

On further reflection: if we treat the outer widget as the "real" one — it's the one inserted into layouts, assigned to the window, given style properties, etc. — and the inner widget is an orphan whose methods and _impl are borrowed — then that causes a problem if a method of the inner widget ever tries to determine anything about its app, window, or style. Or if the implementation needs to know any of those things about its interface. I don't know how many times, if any, that currently happens in the codebase. But to ensure this model is trustworthy, we need to ensure it doesn't.

Another option — and I suspect this is what you were going for with the setter overrides I mentioned — is to monkey about with what the inner widget has access to. If we went that route, I'd be inclined to try a WidgetWrapper class to handle it all in one place; however, rather than setting it up so things are set on the inner widget when they're set on the outer, I'd rather monkey patch the inner widget's attributes like app and window to alias directly to the ones on the outer widget. And probably be readonly, since I imagine those shouldn't ever be getting set on the inner widget.

Either way, it would probably be a good idea to parametrize a wrapped widget into some existing tests.

[HalfWhitt]

I suppose I could instead try putting it in each subclass __init__, before calling Widget.__init__, and see if it causes any weird problems doing so before the node is initialized.

Who would've guessed there would indeed be weird problems! I haven't really done any Toga testing on platforms besides macOS, so it may take me a minute to set up emulators and virtual machines and track these all down.

We may need to start formalizing what an implementation is and isn't allowed to reach back up and access on its interface... or at least when it can do so. I don't have any recommendations on that until I look more into this, though.