iced-test crate design

[1024bees]

Hey folks,

With #1176 (an extension of #957 by @derezzedex ) I'd like to add functionality for running headless applications and taking screenshots. The motivation for this PR was the fact that I have an iced application with a few custom widgets that I'd like to test rigorously with end to end tests. In this PR, @hecrj brought up that we should have an external iced-test crate for testing, and that we should try to keep testing APIs in iced as thin as possible.

I think an iced-test crate is a great idea! I'd like to help drive this, I'd like to ask broadly: how would folks like to test their iced applications and what features would y'all like from an iced-test crate? Also, I have very little expierence with iced-web, so any ideas on testing applications that target wasm would be great :)

I'll start with some of my thoughts.

Traces

A "Trace" represents serialized executions of an application. In #1176 we added "Message" Trace (represented by type Vec<(Application::Message, std::time::Duration)>, where each Message is sent to the application after Duration has passed. This could be expanded to event traces as well (instead of Application::Messages, we send winit::event::Events)

With traces defined, we need a way to execute them! a method was added in #1176 called run_with_message_trace. This is a way of executing a trace on an application, and can be done headlessly or non-headlessly.

In the future, we can add interfaces to record, serialize and deserialize Traces. For example, when a user encounters a bug, they could run their application with a "record" flag, that would record a Trace from user actions, serialize it to a file, and allow it to be re-run at a later time

I think a lot of the scaffolding that traces introduce should be offloaded to iced-test, but I'm having trouble drawing the line on where iced should end and iced-test should begin. For example, run_with_message_trace is implemented by spawning a thread that feeds Messages into the Application via the EventLoopProxy.

There are certainly other ways to implement this function, but I think it is important that this interface is provided "for free" without requiring end users add logic to application code.

Lastly I'd like to pose the question: Traces give a mechanism to run an application without user intervention, and lets headless applications "do things". What other ways could we run applications without user input?

Inspecting Application State

After executing a Trace, should we return the Application for users to inspect state? What other ways could we inspect application state?

Inspecting layout

How can we inspect layout of an application?

We have a coarse grained way of doing that with Screenshots, but what interfaces could we use to bring that up? @hecrj mentioned something like elm-html-test (https://package.elm-lang.org/packages/eeue56/elm-html-test/5.2.0/) and I think we could bring up a similar "query" api to get state of particular widgets.

if there is a better place to have this discussion, I'd be happy to move it there.

Cheers!
bees

[hecrj]

I believe the easiest way to drive an Application for testing is by replaying a History of messages (the traces you described). We can do this easily on top of iced itself:

pub fn run<A>(flags: A::Flags, messages: impl Iterator<Item = A::Message>) -> A
where
    A: iced::Application,
{
    let (mut application, _) = A::new(flags);

    for message in messages {
        let _ = application.update(message);
    }

    application
}

Since we just want to drive the Application state forward, we can discard all of the commands produced during this process.

When it comes to recording a history of messages, I think the best way to expose the feature would be through a flag and a trait bound of Deserialize for Application::Message, although I'm still unsure.

iced_test could then provide helpers to inspect and interact with an Application headlessly. Specifically, it would be great to be able to test the presence of particular widgets and trigger interactions (e.g. after pressing button X, there should be text Y). To allow this (and other use cases), I have plans to introduce a visit method to the Widget trait and leverage the Visitor pattern.

[1024bees]

I believe the easiest way to drive an Application for testing is by replaying a History of messages (the traces you described). We can do this easily on top of iced itself:

pub fn run<A>(flags: A::Flags, messages: impl Iterator<Item = A::Message>) -> A
where
    A: iced::Application,
{
    let (mut application, _) = A::new(flags);

    for message in messages {
        let _ = application.update(message);
    }

    application
}

Since we just want to drive the Application state forward, we can discard all of the commands produced during this process.

this sounds great, I'll look to bring this up in an iced-test crate

When it comes to recording a history of messages, I think the best way to expose the feature would be through a flag and a trait bound of Deserialize for Application::Message, although I'm still unsure.

iced_test could then provide helpers to inspect and interact with an Application headlessly. Specifically, it would be great to be able to test the presence of particular widgets and trigger interactions (e.g. after pressing button X, there should be text Y). To allow this (and other use cases), I have plans to introduce a visit method to the Widget trait and leverage the Visitor pattern.

I think this makes sense; a gap in my knowledge on iced internals is how we would really use a visit function like this; does a widget tree exist that we could call visit() on?