-
Notifications
You must be signed in to change notification settings - Fork 4.2k
Commit
This commit does not belong to any branch on this repository, and may belong to a fork outside of the repository.
[Connector Builder] Add stream slicer (#20748)
* move connector builder components into the same shared components/connectorBuilder directory * move diff over from poc branch * save current progress * add modal for adding streams * focus stream after adding and reset button style * add reset confirm modal and select view on add * style global config and streams buttons * styling improvements * handle long stream names better * pull in connector manifest schema directly * add box shadows to resizable panels * upgrade orval and use connector manifest schema directly * remove airbyte protocol from connector builder api spec * generate python models from openapi change * fix position of yaml toggle * handle no stream case with better looking message * group global fields into single object and fix console error * confirmation modal on toggling dirty form + cleanup * fix connector name display * undo change to manifest schema * remove commented code * remove unnecessary change * fix spacing * use shadow mixin for connector img * add comment about connector img * change onSubmit to no-op * remove console log * clean up styling * simplify sidebar to remove StreamSelectButton component * swap colors of toggle * move FormikPatch to src/core/form * move types up to connectorBuilder/ level * use grid display for ui yaml toggle button * use spread instead of setting array index directly * add intl in missing places * pull connector manifest schema in through separate openapi spec * use correct intl string id * throttle setting json manifest in yaml editor * use button prop instead of manually styling * consolidate AddStreamButton styles * fix sidebar flex styles * use specific flex properties instead of flex * clean up download and reset button styles * use row-reverse for yaml editor download button * fix stream selector styles to remove margins * give connector setup guide panel same corner and shadow styles * remove blur from page display * set view to stream when selected in test panel * add placeholder when stream name is empty * switch to index-based stream selection to preserve testing panel selected stream on rename * handle empty name in stream selector * make connector form work in connector builder * wip * fix small stuff * add basic input UI * user inputs * make most of inputs configuration work * fix a bunch of stuff * handle unknown config types * add warning label * fix label * fix some styling * review comments * improve state management and error handling * allow auth configuration * check for conflicts with the inferred inputs * fix invisible inputs * handle stored form values that don't contain new fields properly * session token and oauth authentication * fill in session token variable * fix merge of default values * add primaryKey and cursorField to builder types, and consolidate default valeues to types.ts * add cursor and primary key fields to ui * save * add page size and token option inputs * fixes after rebase * add pagination * fix pagination types * handle empty field_name better * Update airbyte-webapp/src/locales/en.json Co-authored-by: Lake Mossman <lake@airbyte.io> * Update airbyte-webapp/src/components/connectorBuilder/Builder/InputsView.tsx Co-authored-by: Lake Mossman <lake@airbyte.io> * inputs editing weirdness * input form reset * using the Label component * reduce redundancy and hide advanced input options for inferred inputs * unnecessary validation * typo * unnecessary effect hook * build spec even for invalid forms but do not update stream list * typos * make sure validation error does not go away * make primary key and cursor optional, and reorder * save toggle group progress * fix style of toggle label * handle empty values better * fix page size/token option field validation and rendering * handle cursor pagination page size option correctly * save stream slicer progress * finish stream slicer * fix stream slicer fields and validation Co-authored-by: Joe Reuter <joe@airbyte.io>
- Loading branch information
Showing
4 changed files
with
263 additions
and
9 deletions.
There are no files selected for viewing
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
182 changes: 182 additions & 0 deletions
182
airbyte-webapp/src/components/connectorBuilder/Builder/StreamSlicerSection.tsx
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,182 @@ | ||
| import { useField } from "formik"; | ||
|
|
||
| import { ControlLabels } from "components/LabeledControl"; | ||
|
|
||
| import { timeDeltaRegex } from "../types"; | ||
| import { BuilderCard } from "./BuilderCard"; | ||
| import { BuilderField } from "./BuilderField"; | ||
| import { BuilderOneOf } from "./BuilderOneOf"; | ||
| import { BuilderOptional } from "./BuilderOptional"; | ||
| import { InjectRequestOptionFields } from "./InjectRequestOptionFields"; | ||
| import { ToggleGroupField } from "./ToggleGroupField"; | ||
|
|
||
| interface StreamSlicerSectionProps { | ||
| streamFieldPath: (fieldPath: string) => string; | ||
| } | ||
|
|
||
| export const StreamSlicerSection: React.FC<StreamSlicerSectionProps> = ({ streamFieldPath }) => { | ||
| const [field, , helpers] = useField(streamFieldPath("streamSlicer")); | ||
|
|
||
| const handleToggle = (newToggleValue: boolean) => { | ||
| if (newToggleValue) { | ||
| helpers.setValue({ | ||
| type: "ListStreamSlicer", | ||
| slice_values: [], | ||
| cursor_field: "", | ||
| }); | ||
| } else { | ||
| helpers.setValue(undefined); | ||
| } | ||
| }; | ||
| const toggledOn = field.value !== undefined; | ||
|
|
||
| return ( | ||
| <BuilderCard | ||
| toggleConfig={{ | ||
| label: ( | ||
| <ControlLabels | ||
| label="Stream slicer" | ||
| infoTooltipContent="Configure how to partition a stream into subsets of records and iterate over the data" | ||
| /> | ||
| ), | ||
| toggledOn, | ||
| onToggle: handleToggle, | ||
| }} | ||
| > | ||
| <BuilderOneOf | ||
| path={streamFieldPath("streamSlicer")} | ||
| label="Mode" | ||
| tooltip="Stream slicer method to use on this stream" | ||
| options={[ | ||
| { | ||
| label: "List", | ||
| typeValue: "ListStreamSlicer", | ||
| children: ( | ||
| <> | ||
| <BuilderField | ||
| type="array" | ||
| path={streamFieldPath("streamSlicer.slice_values")} | ||
| label="Slice values" | ||
| tooltip="List of values to iterate over" | ||
| /> | ||
| <BuilderField | ||
| type="string" | ||
| path={streamFieldPath("streamSlicer.cursor_field")} | ||
| label="Cursor field" | ||
| tooltip="Field on record to use as the cursor" | ||
| /> | ||
| <ToggleGroupField | ||
| label="Slice request option" | ||
| tooltip="Optionally configures how the slice values will be sent in requests to the source API" | ||
| fieldPath={streamFieldPath("streamSlicer.request_option")} | ||
| initialValues={{ | ||
| inject_into: "request_parameter", | ||
| field_name: "", | ||
| }} | ||
| > | ||
| <InjectRequestOptionFields | ||
| path={streamFieldPath("streamSlicer.request_option")} | ||
| descriptor="slice value" | ||
| excludeInjectIntoValues={["path"]} | ||
| /> | ||
| </ToggleGroupField> | ||
| </> | ||
| ), | ||
| }, | ||
| { | ||
| label: "Datetime", | ||
| typeValue: "DatetimeStreamSlicer", | ||
| children: ( | ||
| <> | ||
| <BuilderField | ||
| type="string" | ||
| path={streamFieldPath("streamSlicer.datetime_format")} | ||
| label="Datetime format" | ||
| tooltip="Specify the format of the start and end time, e.g. %Y-%m-%d" | ||
| /> | ||
| <BuilderField | ||
| type="string" | ||
| path={streamFieldPath("streamSlicer.start_datetime")} | ||
| label="Start datetime" | ||
| tooltip="Start time to start slicing" | ||
| /> | ||
| <BuilderField | ||
| type="string" | ||
| path={streamFieldPath("streamSlicer.end_datetime")} | ||
| label="End datetime" | ||
| tooltip="End time to end slicing" | ||
| /> | ||
| <BuilderField | ||
| type="string" | ||
| path={streamFieldPath("streamSlicer.step")} | ||
| label="Step" | ||
| tooltip="Time interval for which to break up stream into slices, e.g. 1d" | ||
| pattern={timeDeltaRegex} | ||
| /> | ||
| <BuilderField | ||
| type="string" | ||
| path={streamFieldPath("streamSlicer.cursor_field")} | ||
| label="Cursor field" | ||
| tooltip="Field on record to use as the cursor" | ||
| /> | ||
| <BuilderOptional> | ||
| <BuilderField | ||
| type="string" | ||
| path={streamFieldPath("streamSlicer.lookback_window")} | ||
| label="Lookback window" | ||
| tooltip="How many days before the start_datetime to read data for, e.g. 31d" | ||
| optional | ||
| /> | ||
| <ToggleGroupField | ||
| label="Start time request option" | ||
| tooltip="Optionally configures how the start datetime will be sent in requests to the source API" | ||
| fieldPath={streamFieldPath("streamSlicer.start_time_option")} | ||
| initialValues={{ | ||
| inject_into: "request_parameter", | ||
| field_name: "", | ||
| }} | ||
| > | ||
| <InjectRequestOptionFields | ||
| path={streamFieldPath("streamSlicer.start_time_option")} | ||
| descriptor="start datetime" | ||
| excludeInjectIntoValues={["path"]} | ||
| /> | ||
| </ToggleGroupField> | ||
| <ToggleGroupField | ||
| label="End time request option" | ||
| tooltip="Optionally configures how the end datetime will be sent in requests to the source API" | ||
| fieldPath={streamFieldPath("streamSlicer.end_time_option")} | ||
| initialValues={{ | ||
| inject_into: "request_parameter", | ||
| field_name: "", | ||
| }} | ||
| > | ||
| <InjectRequestOptionFields | ||
| path={streamFieldPath("streamSlicer.end_time_option")} | ||
| descriptor="end datetime" | ||
| excludeInjectIntoValues={["path"]} | ||
| /> | ||
| </ToggleGroupField> | ||
| <BuilderField | ||
| type="string" | ||
| path={streamFieldPath("streamSlicer.stream_state_field_start")} | ||
| label="Stream state field start" | ||
| tooltip="Set which field on the stream state to use to determine the starting point" | ||
| optional | ||
| /> | ||
| <BuilderField | ||
| type="string" | ||
| path={streamFieldPath("streamSlicer.stream_state_field_end")} | ||
| label="Stream state field end" | ||
| tooltip="Set which field on the stream state to use to determine the ending point" | ||
| optional | ||
| /> | ||
| </BuilderOptional> | ||
| </> | ||
| ), | ||
| }, | ||
| ]} | ||
| /> | ||
| </BuilderCard> | ||
| ); | ||
| }; |
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters