[Ingest pipelines] Polish pipeline debugging workflow

[alisonelizabeth]

This PR contains a number of general usability and design improvements.

I tried to break the work up logically by commit (although I didn’t do the best job toward the end 😅). To test, please follow the same instructions in #74964 and ensure there are no regressions.

Notable changes:

• More robust logic to determine the previous processor input (“Data in” accordion on the output tab of the processor flyout)
• Better handling of re-running simulation
  • From the initial flow ("Add documents" button):
    • User can click the “Run the pipeline” button OR click the “Output” tab to rerun simulation.
    • Clicking "Refresh output" will refresh the output AND the verbose output (to refresh processor results)
  • From the processor flyout:
    • Simulation will run when clicking the “Update” button OR the “Output” tab.
    • If configuration changes are made, but the user cancels the submission, simulation will rerun with the original processors.
      • Note: I hit some oddities keeping the form in sync when going back and forth between the configuration and output tabs, so if you could test this area that would be appreciated!
• Created a container component for the test_pipeline_flyout
• Created two components for editing and adding a processor. This was previously one with conditional logic (formerly manage_processor_form). I felt this was getting overly complex, and there was enough divergence that it made sense to create two separate components.
• Various UX improvements
  • New icons to represent statuses
  • New layout for processor output tab
  • New design for the test actions section

What’s missing:

• Better UX for empty prompt. Michael provided a design for this, but it will require additional refactoring. I’m putting it on hold for now and will address separately.
• There is still not a way to cancel/stop simulation (without refreshing page). Design pending.

Screenshots

[mdefazio]

Here are some initial thoughts:

• I think we could make the Output flyout close when clicking outside of it.
• Perhaps the 'Run the pipeline' button in the flyout could be above the code block/document block so it matches the placement of 'Refresh output' on the Output tab?
• Will we be able to provide the list of fields available in the processor configuration?
• Hitting 'Update' closes the flyout, so changing the configuration and viewing the output is difficult
• When updating the configuration, the 'Update' button should be disabled until a change is made. Then perhaps we need to add a status icon next to the Output tab to indicate that the output is not current until 'Update' is clicked. Closing the flyout might then also need an alert modal that asks to save/dismiss changes?
• Do we need a similar alert modal when cancelling the Edit Pipeline flow when there are edits and the user doesn't hit Save?

[alisonelizabeth]

Thanks @mdefazio for the initial feedback!

I think we could make the Output flyout close when clicking outside of it.

👍 I don't have a strong opinion about this, but don't mind implementing it.

Perhaps the 'Run the pipeline' button in the flyout could be above the code block/document block so it matches the placement of 'Refresh output' on the Output tab?

For me, it feels a little cumbersome to first add JSON for the sample documents and then move your mouse back up to click "Run the pipeline". When filling out a form, I typically would expect the submit button to be below. I understand this is only one field though so we might treat it a little differently.

Will we be able to provide the list of fields available in the processor configuration?

Can you clarify what you mean here? JL is working on building out the forms for each processor type.

Hitting 'Update' closes the flyout, so changing the configuration and viewing the output is difficult
When updating the configuration, the 'Update' button should be disabled until a change is made. Then perhaps we need to add a status icon next to the Output tab to indicate that the output is not current until 'Update' is clicked. Closing the flyout might then also need an alert modal that asks to save/dismiss changes?
Do we need a similar alert modal when cancelling the Edit Pipeline flow when there are edits and the user doesn't hit Save?

If you click the "Output" tab, the simulation will rerun with your unsaved changes, so you are able to toggle back and forth between configuration and output and make changes. I understand this may not be obvious though. If you cancel, the output will be reset to the original state. I don't think we've typically disabled forms if the user has not changed any values. Similarly, we don't have confirmation modals for exiting forms with unsaved changes. I think the latter is needed, but just wanted to point out that it is probably an issue we need to solve across several of our apps with forms.

[elasticmachine]

Pinging @elastic/es-ui (Team:Elasticsearch UI)

[jloleysens]

Great work @alisonelizabeth !

I had one code change in view that would be great to get your thoughts on, but I view it as non-blocking for this contribution.

In terms of UX everything worked as expected from your PR description. I think it would be great if we can make the documents dropdown a little bit bigger:

I really like the "Data in"/"Data out" sections added per processor. It gives users a lot of insight into the per-step transformations. I wonder if it is worth add a way to view these on the processor itself? I'd be interested to hear your thoughts @mdefazio !

[jloleysens]

Should be able to remove the !

[jloleysens]

Could we wrap this in a useCallback?

[jloleysens]

If unsavedProcessorDataFormData is only used inside of the form can we push the state further down the component tree? Perhaps the state tracking this could live in the processor settings form component where the tabs get rendered?

Additionally, I think the getDefaultProcessorOptions function that we are passing into the processor settings form fields component could be slightly updated. Maybe something like in processor_form.container:

// ...
const unsavedState = useRef<{} | undefined>();

// we set unsavedState.current whenever a form update happens...

const getProcessor = useCallback((): ProcessorInternal => {
    let options;

    if (unsavedFormData) {
      options = unsavedFormData.current.options;
    } else {
      defaultFields = { fields: processor?.options ?? {} };
    }

    return { ...processor, options: };
  }, [processor]);
// ...

Then we always sync using the form.subscribe callback that is calling onFormUpdate. I think one step further would be to hold "unsaved" state in a useRef. Did not test this 😅 - lmk what you think!

Also, then instead of passing in processor and getDefaultOptions to processor fields form, we just pass in the getProcessor callback since it is a "computed" value.

[alisonelizabeth]

Great suggestion. I went ahead and implemented this, but only update the ref on submit rather than the form.subscribe callback, as the form data ends up getting reset on tab change here.

[jloleysens]

It is a little bit tricky to assess, but I am assuming we have not introduced changes that require test coverage changes.

[alisonelizabeth]

Thanks for the review @jloleysens! I addressed your feedback via 1aa0ef4. I also made the dropdown wider and fixed an alignment issue with the custom icons within the callout.

[mdefazio]

Some comments regarding design and UX:

• Let's change success check to a checkInCircle so it stands out more.
• Change the dot icon color to euiColorLightShade so it doesn't stand out more. My thinking here is that by making more contrast between these it will be more obvious where in the list the pipeline stopped.
• Perhaps change 'Test' to 'Test pipeline' in the updated testing module (also would match the flyout header title) so it's clearer what this area does
• I think we could add a status and timestamp in the flyout that indicates that testing state. I find myself unsure when it ran and what action in the UI will trigger a new run. (maybe a larger UI issue here?)
• Add a play icon to 'Run the pipeline' button in document tab since we have the refresh icon in the output tab
• 'Output' button in test module should read 'View output' so it represents an action
• Maybe not for this PR, but I noticed that the 'Refresh' button in the pipeline list is using the secondary color—this should be primary to match the Create button
  

[alisonelizabeth]

@elasticmachine merge upstream

[alisonelizabeth]

Thanks @mdefazio! Great suggestions.

I addressed most of your comments. Would you mind taking another look? I'm still not 100% on the best approach for adding a status/timestamp to the flyout. As mentioned 1:1, perhaps it's best to punt on this until we get some user testing.

Maybe not for this PR, but I noticed that the 'Refresh' button in the pipeline list is using the secondary color—this should be primary to match the Create button

Great catch. I took a look at some of our other UIs, and we're also using the secondary color 🤔. The button was not introduced as part of this PR so I'm going to leave as-is for now, and it also might be something we need to address across other apps.

Latest screenshots:

[mdefazio]

Looks good. I see we don't have the euiColorLightShade as an option for the icon. Maybe this is something we can look into on the EUI side. But we'd obviously like to remove those hex values in a follow up PR.

[kibanamachine]

💛 Build succeeded, but was flaky

• continuous-integration/kibana-ci/pull-request
• Commit: b03f2d2
• Flaky suites:
  • xpack-kibana-ciGroup7

Build metrics

@kbn/optimizer bundle module count

id	value	diff	baseline
ingestPipelines	447	+16	431

async chunks size

id	value	diff	baseline
ingestPipelines	764.4KB	+20.0KB	744.4KB

History

• 💔 Build #74541 failed 7f87c1a
• 💚 Build #74342 succeeded 1aa0ef4
• 💚 Build #74223 succeeded 53f278b
• 💚 Build #72747 succeeded 9616142
• 💛 Build #71892 was flaky d8107bc

To update your PR or re-run it, just comment with:
@elasticmachine merge upstream