Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Feature/enable unstructured #173

Merged
merged 89 commits into from
Oct 10, 2024
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
89 commits
Select commit Hold shift + click to select a range
605a7fb
send to bk
fivetran-jamie Jun 3, 2024
c58b4b6
try this out
fivetran-jamie Jun 3, 2024
5f2113b
first try
fivetran-jamie Jun 3, 2024
662f3f2
use mcro
fivetran-jamie Jun 3, 2024
5bfead9
Testing
fivetran-jamie Jun 3, 2024
d36dc49
puhs
fivetran-jamie Jun 3, 2024
700589f
redshift
fivetran-jamie Jun 3, 2024
f6ada0c
try nullif
fivetran-jamie Jun 4, 2024
f07271c
feature/historical-schedules
fivetran-catfritz Sep 11, 2024
f15b4b1
Merge branch 'feature/historical-schedules' into explore/audit-log-spike
fivetran-catfritz Sep 11, 2024
fd53978
Merge pull request #170 from fivetran/explore/audit-log-spike
fivetran-catfritz Sep 11, 2024
cf89429
rework schedule_history
fivetran-catfritz Sep 12, 2024
54cdf21
update unnest logic
fivetran-catfritz Sep 13, 2024
505f343
update schedule_history
fivetran-catfritz Sep 23, 2024
4ca4099
complete schedule_history
fivetran-catfritz Sep 23, 2024
7806d83
revise holidays
fivetran-catfritz Sep 26, 2024
a54bee2
add macro
fivetran-catfritz Sep 26, 2024
0477543
typeo
fivetran-catfritz Sep 26, 2024
f980433
updates
fivetran-catfritz Sep 26, 2024
016e449
updates
fivetran-catfritz Sep 26, 2024
45fbcfe
updates
fivetran-catfritz Sep 26, 2024
3907813
updates
fivetran-catfritz Sep 26, 2024
8b32c45
try out in buildkite
fivetran-jamie Sep 27, 2024
9c49399
remove schedule days during holiday
fivetran-catfritz Sep 27, 2024
e4bf09f
revise
fivetran-catfritz Sep 27, 2024
4523d2c
revise
fivetran-catfritz Sep 27, 2024
cceae3f
add config
fivetran-catfritz Sep 27, 2024
62c5fa9
add config
fivetran-catfritz Sep 27, 2024
1db42a7
allow disable holidays
fivetran-catfritz Sep 27, 2024
f730908
streamline bk run
fivetran-jamie Sep 27, 2024
ee99616
adjust for multiple holidays in a week
fivetran-catfritz Sep 27, 2024
4f395de
add casting
fivetran-catfritz Sep 27, 2024
8613f8a
add casting
fivetran-catfritz Sep 27, 2024
f762ca1
fixes
fivetran-catfritz Sep 27, 2024
00a98cb
fixes
fivetran-catfritz Sep 27, 2024
96be5ae
fixes
fivetran-catfritz Sep 27, 2024
7517057
fixes
fivetran-catfritz Sep 27, 2024
184e639
fixes
fivetran-catfritz Sep 28, 2024
c594f3f
fix multiyear schedules
fivetran-catfritz Sep 28, 2024
e613480
add longer holiday support
fivetran-catfritz Sep 30, 2024
a49f0d6
revert multiweek
fivetran-catfritz Oct 1, 2024
e1f9d32
adjust multiweek
fivetran-catfritz Oct 1, 2024
98e6872
account for non sunday week starts
fivetran-catfritz Oct 2, 2024
0321581
updates
fivetran-catfritz Oct 2, 2024
6163942
update weeks spanned calc
fivetran-catfritz Oct 2, 2024
c774f6d
update naming
fivetran-catfritz Oct 2, 2024
b8a9f4a
update to dbt_date
fivetran-catfritz Oct 3, 2024
cfcd106
update comments
fivetran-catfritz Oct 3, 2024
781f4fd
update to dbt date weekstart
fivetran-catfritz Oct 3, 2024
28457e8
modernize calendar spine
fivetran-catfritz Oct 3, 2024
c117df9
make sure we're working with strings when replacing
fivetran-jamie Oct 3, 2024
933d62e
let's see if bk works
fivetran-jamie Oct 4, 2024
0aacff4
postgres?
fivetran-jamie Oct 4, 2024
e39919e
i think schedule history may actually be workingggg
fivetran-catfritz Oct 4, 2024
8a9d84d
get ready to merge into catherines branch
fivetran-jamie Oct 4, 2024
f691904
Merge branch 'feature/historical-schedules' into feature/historical-s…
fivetran-jamie Oct 4, 2024
b3c97ab
postgres revert
fivetran-jamie Oct 4, 2024
94b11a2
Merge branch 'feature/historical-schedules-jamie-redshift' of https:/…
fivetran-jamie Oct 4, 2024
eff6401
add comments
fivetran-catfritz Oct 4, 2024
763ca33
redshift fixes
fivetran-catfritz Oct 4, 2024
238bd96
redshift fixes
fivetran-catfritz Oct 4, 2024
4f57203
redshift fixes
fivetran-catfritz Oct 4, 2024
a6f3536
Merge branch 'feature/historical-schedules' into feature/historical-s…
fivetran-catfritz Oct 4, 2024
e342c79
Merge pull request #172 from fivetran/feature/historical-schedules-ja…
fivetran-catfritz Oct 4, 2024
5c720b5
updates
fivetran-catfritz Oct 4, 2024
6d8ff79
validation update
fivetran-catfritz Oct 6, 2024
c100e51
split models
fivetran-catfritz Oct 6, 2024
56bb954
update ymls
fivetran-catfritz Oct 7, 2024
c4929cf
revise scchedule groups
fivetran-catfritz Oct 8, 2024
1d1b2e1
updates
fivetran-catfritz Oct 9, 2024
05fefd5
add inline comments
fivetran-catfritz Oct 9, 2024
d97f714
update decision log
fivetran-catfritz Oct 9, 2024
6573314
update changelog
fivetran-catfritz Oct 9, 2024
0b0a05d
regen docs
fivetran-catfritz Oct 9, 2024
246624b
fix yml
fivetran-catfritz Oct 9, 2024
888f109
update changelog
fivetran-catfritz Oct 9, 2024
af951b1
address review comments and regen docs
fivetran-catfritz Oct 9, 2024
855dc77
address review comments
fivetran-catfritz Oct 9, 2024
73845a9
Apply suggestions from code review
fivetran-catfritz Oct 10, 2024
b21e492
feature/enable-unstructured
fivetran-catfritz Oct 10, 2024
8a8d92f
update packages
fivetran-catfritz Oct 10, 2024
d234ef9
release review updates
fivetran-catfritz Oct 10, 2024
fa2c2e6
merge
fivetran-catfritz Oct 10, 2024
03172b9
Apply suggestions from code review
fivetran-catfritz Oct 10, 2024
60afd2b
regen docs
fivetran-catfritz Oct 10, 2024
c99c36b
update readme
fivetran-catfritz Oct 10, 2024
30d7af9
update readme
fivetran-catfritz Oct 10, 2024
0be5631
update readme
fivetran-catfritz Oct 10, 2024
7cc92a6
update quickstartyml
fivetran-catfritz Oct 10, 2024
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
2 changes: 1 addition & 1 deletion .buildkite/scripts/run_models.sh
Original file line number Diff line number Diff line change
Expand Up @@ -20,7 +20,7 @@ dbt seed --target "$db" --full-refresh
dbt run --target "$db" --full-refresh
dbt run --target "$db"
dbt test --target "$db"
dbt run --vars '{zendesk__unstructured_enabled: true, using_schedules: false, using_domain_names: false, using_user_tags: false, using_ticket_form_history: false, using_organization_tags: false}' --target "$db" --full-refresh
dbt run --vars '{zendesk__unstructured_enabled: false, using_schedules: false, using_domain_names: false, using_user_tags: false, using_ticket_form_history: false, using_organization_tags: false}' --target "$db" --full-refresh
dbt test --target "$db"
dbt run --vars '{using_schedule_histories: true, using_holidays: false}' --target "$db" --full-refresh
dbt test --target "$db"
Expand Down
3 changes: 2 additions & 1 deletion .quickstart/quickstart.yml
Original file line number Diff line number Diff line change
Expand Up @@ -33,5 +33,6 @@ public_models: [
"zendesk__ticket_summary",
"zendesk__ticket_field_history",
"zendesk__sla_policies",
"zendesk__ticket_backlog"
"zendesk__ticket_backlog",
"zendesk__document"
]
7 changes: 6 additions & 1 deletion CHANGELOG.md
Original file line number Diff line number Diff line change
@@ -1,3 +1,8 @@
# dbt_zendesk v0.19.0-a1
[PR #173](https://github.com/fivetran/dbt_zendesk/pull/173) includes the following changes:
## Breaking Changes
- Model `zendesk__document` is now enabled by default. To disable it, set the `zendesk__unstructured_enabled` variable to `false` in your `dbt_project.yml`.

# dbt_zendesk v0.18.0
[PR #171](https://github.com/fivetran/dbt_zendesk/pull/171) includes the following changes:

Expand Down Expand Up @@ -381,4 +386,4 @@ Tiny release ahead!
- [csaroff](https://github.com/csaroff) ([#47](https://github.com/fivetran/dbt_zendesk/pull/47))
- [jackiexsun](https://github.com/jackiexsun) ([#42](https://github.com/fivetran/dbt_zendesk/pull/42))
- [emiliedecherney](https://github.com/emiliedecherney) ([#50](https://github.com/fivetran/dbt_zendesk/pull/50))
- [gareginordyan](https://github.com/gareginordyan) ([#44](https://github.com/fivetran/dbt_zendesk/pull/44))
- [gareginordyan](https://github.com/gareginordyan) ([#44](https://github.com/fivetran/dbt_zendesk/pull/44))
10 changes: 5 additions & 5 deletions README.md
Original file line number Diff line number Diff line change
Expand Up @@ -65,7 +65,7 @@ Include the following zendesk package version in your `packages.yml` file:
```yml
packages:
- package: fivetran/zendesk
version: [">=0.18.0", "<0.19.0"]
version: 0.19.0-a1

```
> **Note**: Do not include the Zendesk Support source package. The Zendesk Support transform package already has a dependency on the source in its own `packages.yml` file.
Expand Down Expand Up @@ -94,14 +94,14 @@ vars:

### (Optional) Step 5: Additional configurations

#### Enabling the unstructured document model for NLP
#### Disabling the unstructured document model for NLP
This package includes the `zendesk__document` model, which processes and segments Zendesk text data for vectorization, making it suitable for NLP workflows. The model outputs structured chunks of text with associated document IDs, segment indices, and token counts. For definitions and more information, refer to [zendesk__document](https://fivetran.github.io/dbt_zendesk/#!/model/model.zendesk.zendesk__document) in our dbt docs.

By default, this model is disabled. To enable it, update the `zendesk__unstructured_enabled` variable to true in your dbt_project.yml:
By default, this model is enabled. To disable it, update the `zendesk__unstructured_enabled` variable to true in your dbt_project.yml:

```yml
vars:
zendesk__unstructured_enabled: true # false by default.
zendesk__unstructured_enabled: false # true by default.
```

##### Customizing Chunk Size for Vectorization
Expand Down Expand Up @@ -261,4 +261,4 @@ This dbt package takes an opinionated stance on how business time metrics are ca

## Are there any resources available?
- If you have questions or want to reach out for help, see the [GitHub Issue](https://github.com/fivetran/dbt_zendesk/issues/new/choose) section to find the right avenue of support for you.
- If you would like to provide feedback to the dbt package team at Fivetran or would like to request a new dbt package, fill out our [Feedback Form](https://www.surveymonkey.com/r/DQ7K7WW).
- If you would like to provide feedback to the dbt package team at Fivetran or would like to request a new dbt package, fill out our [Feedback Form](https://www.surveymonkey.com/r/DQ7K7WW).
2 changes: 1 addition & 1 deletion dbt_project.yml
Original file line number Diff line number Diff line change
@@ -1,5 +1,5 @@
name: 'zendesk'
version: '0.18.0'
version: '0.19.0'

config-version: 2
require-dbt-version: [">=1.3.0", "<2.0.0"]
Expand Down
2 changes: 1 addition & 1 deletion docs/catalog.json

Large diffs are not rendered by default.

2 changes: 1 addition & 1 deletion docs/manifest.json

Large diffs are not rendered by default.

9 changes: 4 additions & 5 deletions integration_tests/dbt_project.yml
Original file line number Diff line number Diff line change
@@ -1,11 +1,14 @@
config-version: 2

name: 'zendesk_integration_tests'
version: '0.18.0'
version: '0.19.0'

profile: 'integration_tests'

vars:
## Uncomment for docs generation
# using_schedule_histories: True

zendesk_schema: zendesk_integration_tests_50
zendesk_source:
zendesk_organization_identifier: "organization_data"
Expand All @@ -27,10 +30,6 @@ vars:
zendesk_user_tag_identifier: "user_tag_data"
zendesk_audit_log_identifier: "audit_log_data"

## Uncomment for docs generation
# zendesk__unstructured_enabled: True
# using_schedule_histories: True

## For validation testing. To be commented out before release.
# zendesk_schema: zendesk_test_env
# using_domain_names: false
Expand Down
Original file line number Diff line number Diff line change
@@ -1,4 +1,4 @@
{{ config(enabled=var('zendesk__unstructured_enabled', False)) }}
{{ config(enabled=var('zendesk__unstructured_enabled', True)) }}

with ticket_comments as (
select *
Expand Down
Original file line number Diff line number Diff line change
@@ -1,4 +1,4 @@
{{ config(enabled=var('zendesk__unstructured_enabled', False)) }}
{{ config(enabled=var('zendesk__unstructured_enabled', True)) }}

with filtered_comment_documents as (
select *
Expand Down
Original file line number Diff line number Diff line change
@@ -1,4 +1,4 @@
{{ config(enabled=var('zendesk__unstructured_enabled', False)) }}
{{ config(enabled=var('zendesk__unstructured_enabled', True)) }}

with tickets as (
select *
Expand Down
2 changes: 1 addition & 1 deletion models/unstructured/zendesk__document.sql
Original file line number Diff line number Diff line change
@@ -1,4 +1,4 @@
{{ config(enabled=var('zendesk__unstructured_enabled', False)) }}
{{ config(enabled=var('zendesk__unstructured_enabled', True)) }}

with ticket_document as (
select *
Expand Down