Skip to content

Commit

Permalink
Remove duplicate paragraph (#18662)
Browse files Browse the repository at this point in the history
Removes a duplicate of the paragraph that begins "Note that we're also setting the `stream_cursor_field` in the stream's `$options`..." which followed an outdated schema.
  • Loading branch information
scottleechua authored and nataly committed Nov 3, 2022
1 parent f3cc56d commit df0ffa9
Showing 1 changed file with 7 additions and 19 deletions.
Original file line number Diff line number Diff line change
Expand Up @@ -115,13 +115,15 @@ The start time is defined in the config file, while the end time is defined by t
Note that we're also setting the `stream_cursor_field` in the stream's `$options` so it can be accessed by the `StreamSlicer`:

```yaml
streams:
- type: DeclarativeStream
definitions:
<...>
rates_stream:
$ref: "*ref(definitions.base_stream)"
$options:
name: "rates"
primary_key: "date"
path: "/exchangerates_data/{{config['start_date'] or 'latest'}}"
stream_cursor_field: "date"
primary_key: "rates"
<...>
```

We'll also update the retriever to user the stream slicer:
Expand All @@ -138,20 +140,6 @@ definitions:
This will generate slices from the start time until the end time, where each slice is exactly one day.
The start time is defined in the config file, while the end time is defined by the `now_utc()` macro, which will evaluate to the current date in the current timezone at runtime. See the section on [string interpolation](../advanced-topics.md#string-interpolation) for more details.

Note that we're also setting the `stream_cursor_field` in the stream's `$options` so it can be accessed by the `StreamSlicer`:

```yaml
definitions:
<...>
rates_stream:
$ref: "*ref(definitions.base_stream)"
$options:
name: "rates"
primary_key: "date"
path: "/exchangerates_data/{{config['start_date'] or 'latest'}}"
stream_cursor_field: "date"
```

Finally, we'll update the path to point to the `stream_slice`'s start_time

```yaml
Expand Down Expand Up @@ -299,4 +287,4 @@ Next, we'll run the [Source Acceptance Tests suite to ensure the connector invar

- [Incremental reads](../../cdk-python/incremental-stream.md)
- [Stream slicers](../understanding-the-yaml-file/stream-slicers.md)
- [Stream slices](../../cdk-python/stream-slices.md)
- [Stream slices](../../cdk-python/stream-slices.md)

0 comments on commit df0ffa9

Please sign in to comment.