feat: autocomplete data fields in the editor #4062

jessicamcinchak · 2024-12-10T14:25:53Z

Key changes:

Introduces a new store method getFlowSchema which gets all the existing data fields (fn & val) for a given flow split into "nodes" (what you can click on in the graph) and "options" (Question & Checklist "options")
- This allows us to have more refined suggestions when autocompleting and encourages content best practices
- Filters and their flag values are omitted from the schema "suggestions" because they are not exposed to editors to configure
Adds a <DataFieldAutocomplete /> component which we can plug in across the editor like other input rows
Implements the autocomplete across all component types with editor-facing data fields, deriving schema "suggestions" using three broad strategies:
1. Question & Checklist are most complex
  - Top-level fn suggestions reflect flow nodes, options val suggestions reflect other options only
  - If fn is any of proposal.projectType, property.type, or application.type, we'll override default suggested options in favor of ODP Schema enum values
    - When using ODP Schema values, any existing initialValues are additionally supported, as well as option to "Add.." new values, to ensure editors maintain flexibility and are able to use and experiment with data values outside of the current schema release
2. FileUpload & FileUploadAndLabel
  - Suggestions override current flow schema and always suggest based on ODP Schema FileType enums; same initialValues & "Add.." new values handling as above
3. All other component types (inputs, lists, etc)
  - Suggestions reflect other flow nodes only (never options)

Todos:

https://trello.com/c/6OkLrUNE/2668-autofill-passport-variables-when-creating-nodes-in-the-editor

github-actions · 2024-12-10T15:49:11Z

Pizza

Deployed 45c0221 to https://4062.planx.pizza.

Useful links:

…ss/data-field-autocomplete

jessicamcinchak · 2024-12-18T10:43:08Z

editor.planx.uk/src/@planx/components/shared/DataFieldAutocomplete.tsx

+          return formattedOption;
+        }}
+        renderOption={renderOptions}
+        freeSolo


Explicitly passing the freeSolo prop is a bit strange because it's already handled in AutocompleteProps<string, false, false, true, "div">, but explicitly passing here seems to be the Stack Overflow consensus for how to clear up a console error related to isOptionEqualToValue which assumes "new" options will never be added/present

jessicamcinchak · 2024-12-18T10:46:31Z

editor.planx.uk/src/@planx/components/shared/DataFieldAutocomplete.tsx

+        id="data-field-autocomplete"
+        key="data-field-autocomplete"
+        placeholder="Data field"
+        required={Boolean(props.required)}


required prop will simply trigger browser default error handling which is quite consistent with current editor inputs !

Gov UK-looking error wrappers for autocomplete in the editor are larger future thing to revisit (eg would we simply wrap in our existing wrapper or want to account for error-related props here?)

This is what I would have gone for here as well! Currently we have a mix but this works great. We can revisit this when taking another design look at this modal alongside labels and layout.

jessicamcinchak · 2024-12-18T10:48:32Z

editor.planx.uk/src/ui/shared/AutocompleteInput.tsx

+  backgroundColor: theme.palette.background.paper,
+  [`& .${outlinedInputClasses.root}, input`]: {
+    cursor: "pointer",
+    // TODO extract as `format="data"` prop more like `Input` ? 


This TODO is good to note, but I don't think we need to pick it up until there's another use case/iteration of the autocomplete input - any objections?

Yep good point - I don't think we really need a format prop here right now - a TODO works for me!

jessicamcinchak · 2024-12-18T15:57:51Z

editor.planx.uk/src/@planx/components/List/Editor.tsx

@@ -92,6 +94,8 @@ function ListComponent(props: Props) {
    validateOnChange: false,
  });

+  const dataFieldSchema = useStore().getFlowSchema()?.nodes;


List, Page & MapAndLabel components have their own pre-existing concepts of "schemas", so the store function is more explicitly assigned to dataFieldSchema in these cases

jessicamcinchak · 2024-12-18T16:00:18Z

editor.planx.uk/src/pages/FlowEditor/lib/store/editor.ts

+
+      // TODO align to (reuse?) data facets from search
+      if (node.data?.output) nodes.add(node.data.output);
+      if (node.data?.dataFieldBoundary) nodes.add(node.data.dataFieldBoundary);


This TODO goes away with #4083 & #4085 🧹

jessicamcinchak · 2024-12-19T07:31:08Z

e2e/tests/ui-driven/src/helpers/addComponent.ts

-      await page.getByPlaceholder("Data Field").fill(options?.[0] || "");
+      await page
+        .getByRole("combobox", { name: "Data field" })
+        .fill(options?.[0] || "");


‼️ E2E still are ultimately failing because nodes with a required data field are not being correctly populated - I've tried a couple iterations here, but still not landed on the right solution.

Opening for wider review in the meantime and any pointers very welcome!

Here's what I've tried so far:

The placeholder text has changed from "Data Field" to "Data field", but getByPlaceholder() will now match more than one element (MUI Autocompletes have many nested elements)

So, "getByRole("combobox", { name: "Data field" }) should exclusively be matching the input element

On the first component, there will be no flow schema or suggestions yet and the popupIcon arrow is disabled, so there's no dropdown of options to .click() or open/expand - but I'm guessing I'm still using the wrong combo of .fill/.click here to mimic typing input ??

This is working for me locally - 5114475

Running on CI now 🤞

It failed on some still - to be fair I didn't run them all locally just the first few. Will pick this up again but I think adding the .click() is what we're after.

Appreciate second eyes on this - it's a tedious tricky one! Once one node with a data field is successfully added, then future ones will have suggested schema options - so might require combination of fill & click ?

jessicamcinchak · 2024-12-19T09:55:21Z

editor.planx.uk/src/@planx/components/Checklist/Editor/Options.tsx

+
+  const schema = useStore().getFlowSchema()?.options;
+  const initialOptions: Option[] | undefined = formik.initialValues.options || formik.initialValues.groupedOptions?.map((group: Group<Option>) => group.children)?.flat();
+  const initialOptionVals = initialOptions?.map((option) => option.data?.val);


This is a bit of a lazy rebase right now while the exclusive-or work is still in-progress - I think initialOptions should be able to be simplified/derived from exclusiveOptions or nonExclusiveOptions once those are handling "grouped" options

DafyddLlyr

Great! Zero issues - all working as expected, code was clear and easy to follow.

Appreciate the detailed PR description and thought that's gone into this - great feature 😄

DafyddLlyr · 2024-12-19T16:42:26Z

editor.planx.uk/src/@planx/components/shared/DataFieldAutocomplete.tsx

+import InputRow from "ui/shared/InputRow";
+
+interface Props {
+  schema?: string[];


Could be have getStore().getFlowSchema()?.nodes be the default value for this prop so that it doesn't need to be defined in many components?

It also then feel more explicit where there's an exception (e.g. for options).

💡 Good suggestion! Will aim to push this change up this afternoon

Very satisfying one to implement ✂️ thanks again ! 867a6c2

DafyddLlyr · 2024-12-19T16:52:56Z

editor.planx.uk/src/ui/shared/AutocompleteInput.tsx

+  backgroundColor: theme.palette.background.paper,
+  [`& .${outlinedInputClasses.root}, input`]: {
+    cursor: "pointer",
+    // TODO extract as `format="data"` prop more like `Input` ? 


Yep good point - I don't think we really need a format prop here right now - a TODO works for me!

DafyddLlyr · 2024-12-20T08:53:05Z

editor.planx.uk/src/@planx/components/shared/DataFieldAutocomplete.tsx

+        id="data-field-autocomplete"
+        key="data-field-autocomplete"
+        placeholder="Data field"
+        required={Boolean(props.required)}


This is what I would have gone for here as well! Currently we have a mix but this works great. We can revisit this when taking another design look at this modal alongside labels and layout.

DafyddLlyr · 2024-12-20T09:05:04Z

editor.planx.uk/src/@planx/components/shared/utils.ts

@@ -63,3 +64,21 @@ export const getPreviouslySubmittedData = ({

  return data;
 };
+
+export const getOptionsSchemaByFn = (fn?: string, defaultOptionsSchema?: string[], initialOptions?: (string | undefined)[]) => {


Super clear and easy to follow 👍

DafyddLlyr · 2024-12-20T09:23:45Z

editor.planx.uk/src/pages/FlowEditor/lib/store/editor.ts

+        if (node.data?.fn !== "flag") nodes.add(node.data.fn)
+      };
+
+      // TODO align to (reuse?) data facets from search


Great comment!

We could probably use DATA_FACETS here (filtered to strings only) and then use this to get the data values from each node.

However, I think this is actually a more complex solution - there's a very different use case here and we'll have just as many exceptions and clauses. This works really well for now - we just need to make sure that we keep data values for new components, this function, and DATA_FACETS in sync.

Yep - fn only basically felt like right "first" go at this since other DATA_FACETS like FileUploadAndLabel individual file fns are getting "overwritten" by ODP Schema enum suggestions anyways ! So it's a good question of whether or not we actually always want this to be perfectly in sync !

DafyddLlyr · 2024-12-20T09:26:15Z

P.S. Happy to have a crack at Playwright later today and push a commit here 👍

…r component

…ss/data-field-autocomplete

jessicamcinchak and others added 2 commits December 10, 2024 15:24

wip

48d60ce

fix: Autocomplete types

d0e707e

jessicamcinchak added 9 commits December 11, 2024 19:58

Merge branch 'main' of github.com:theopensystemslab/planx-new into je…

3b2c2c5

…ss/data-field-autocomplete

styled and mostly working x3 component types

76676e5

Question options plus ODP enum overrides working

aa13297

hide popup icon

04e19ce

rebase and filter out flag values

e6a8be6

fix add new option

949f090

more component types

cd5f1b8

final component types

5496a47

type fix

6dba5b7

jessicamcinchak commented Dec 18, 2024

View reviewed changes

jessicamcinchak added 5 commits December 18, 2024 20:28

rebase plus store tests

d878159

update e2e placeholders

c27fdf1

try page.locator by id in e2e, placeholder matches multiple elements

cea734f

prettier

8e11f6c

try getByRole('combobox')

83e6e3a

jessicamcinchak commented Dec 19, 2024

View reviewed changes

jessicamcinchak marked this pull request as ready for review December 19, 2024 07:31

jessicamcinchak requested a review from a team December 19, 2024 07:31

jessicamcinchak commented Dec 19, 2024

View reviewed changes

DafyddLlyr approved these changes Dec 20, 2024

View reviewed changes

DafyddLlyr and others added 2 commits December 20, 2024 10:52

fix(e2e): Add click on combobox

5114475

set default schema options and remove getFlowSchema()?.nodes calls pe…

867a6c2

…r component

DafyddLlyr and others added 4 commits December 20, 2024 15:49

fix(e2e): Missing clicks and enters

e57ba7f

Merge branch 'main' of github.com:theopensystemslab/planx-new into je…

0e822be

…ss/data-field-autocomplete

rebase

a24345a

fix(e2e): Drop index, use last. Match by value without using fill()

ec9c832

DafyddLlyr force-pushed the jess/data-field-autocomplete branch from 710156f to ec9c832 Compare December 20, 2024 16:34

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

feat: autocomplete data fields in the editor #4062

feat: autocomplete data fields in the editor #4062

jessicamcinchak commented Dec 10, 2024 •

edited

Loading

github-actions bot commented Dec 10, 2024 •

edited

Loading

jessicamcinchak Dec 18, 2024

jessicamcinchak Dec 18, 2024 •

edited

Loading

DafyddLlyr Dec 20, 2024

jessicamcinchak Dec 18, 2024

DafyddLlyr Dec 19, 2024

jessicamcinchak Dec 18, 2024

jessicamcinchak Dec 18, 2024 •

edited

Loading

jessicamcinchak Dec 19, 2024 •

edited

Loading

DafyddLlyr Dec 20, 2024

DafyddLlyr Dec 20, 2024

jessicamcinchak Dec 20, 2024

jessicamcinchak Dec 19, 2024

DafyddLlyr left a comment

DafyddLlyr Dec 19, 2024

jessicamcinchak Dec 20, 2024

jessicamcinchak Dec 20, 2024

DafyddLlyr Dec 19, 2024

DafyddLlyr Dec 20, 2024

DafyddLlyr Dec 20, 2024

DafyddLlyr Dec 20, 2024

jessicamcinchak Dec 20, 2024

DafyddLlyr commented Dec 20, 2024

feat: autocomplete data fields in the editor #4062

Are you sure you want to change the base?

feat: autocomplete data fields in the editor #4062

Conversation

jessicamcinchak commented Dec 10, 2024 • edited Loading

github-actions bot commented Dec 10, 2024 • edited Loading

Pizza

Choose a reason for hiding this comment

jessicamcinchak Dec 18, 2024 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

jessicamcinchak Dec 18, 2024 • edited Loading

Choose a reason for hiding this comment

jessicamcinchak Dec 19, 2024 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

DafyddLlyr left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

DafyddLlyr commented Dec 20, 2024

jessicamcinchak commented Dec 10, 2024 •

edited

Loading

github-actions bot commented Dec 10, 2024 •

edited

Loading

jessicamcinchak Dec 18, 2024 •

edited

Loading

jessicamcinchak Dec 18, 2024 •

edited

Loading

jessicamcinchak Dec 19, 2024 •

edited

Loading