Skip to content

Commit

Permalink
Add slugify macro to utils, use in pivot macro (#314)
Browse files Browse the repository at this point in the history
  • Loading branch information
Claire Carroll authored and clrcrl committed May 19, 2021
1 parent c02c009 commit 445dbfd
Show file tree
Hide file tree
Showing 11 changed files with 61 additions and 6 deletions.
1 change: 1 addition & 0 deletions CHANGELOG.md
Original file line number Diff line number Diff line change
Expand Up @@ -36,6 +36,7 @@ If you were relying on the position to match up your optional arguments, this ma

## Features
* Add new argument, `order_by`, to `get_column_values` (code originally in [#289](https://github.com/fishtown-analytics/dbt-utils/pull/289/) from [@clausherther](https://github.com/clausherther), merged via [#349](https://github.com/fishtown-analytics/dbt-utils/pull/349/))
* Add `slugify` macro, and use it in the pivot macro. :rotating_light: This macro uses the `re` module, which is only available in dbt v0.19.0+. As a result, this feature introduces a breaking change. ([#314](https://github.com/fishtown-analytics/dbt-utils/pull/314))

## Fixes

Expand Down
42 changes: 38 additions & 4 deletions README.md
Original file line number Diff line number Diff line change
Expand Up @@ -974,8 +974,8 @@ When an expression falls outside the range, the function returns:


---
### Logger
#### pretty_time ([source](macros/logger/pretty_time.sql))
### Jinja Helpers
#### pretty_time ([source](macros/jinja_helpers/pretty_time.sql))
This macro returns a string of the current timestamp, optionally taking a datestring format.
```sql
{#- This will return a string like '14:50:34' -#}
Expand All @@ -985,7 +985,7 @@ This macro returns a string of the current timestamp, optionally taking a datest
{{ dbt_utils.pretty_time(format='%Y-%m-%d %H:%M:%S') }}
```

#### pretty_log_format ([source](macros/logger/pretty_log_format.sql))
#### pretty_log_format ([source](macros/jinja_helpers/pretty_log_format.sql))
This macro formats the input in a way that will print nicely to the command line when you `log` it.
```sql
{#- This will return a string like:
Expand All @@ -994,7 +994,7 @@ This macro formats the input in a way that will print nicely to the command line

{{ dbt_utils.pretty_log_format("my pretty message") }}
```
#### log_info ([source](macros/logger/log_info.sql))
#### log_info ([source](macros/jinja_helpers/log_info.sql))
This macro logs a formatted message (with a timestamp) to the command line.
```sql
{{ dbt_utils.log_info("my pretty message") }}
Expand All @@ -1005,6 +1005,40 @@ This macro logs a formatted message (with a timestamp) to the command line.
11:07:31 + my pretty message
```

#### slugify ([source](macros/jinja_helpers/slugify.sql))
This macro is useful for transforming Jinja strings into "slugs", and can be useful when using a Jinja object as a column name, especially when that Jinja object is not hardcoded.

For this example, let's pretend that we have payment methods in our payments table like `['venmo App', 'ca$h-money']`, which we can't use as a column name due to the spaces and special characters. This macro does its best to strip those out in a sensible way: `['venmo_app',
'cah_money']`.

```sql
{%- set payment_methods = dbt_utils.get_column_values(
table=ref('raw_payments'),
column='payment_method'
) -%}

select
order_id,
{%- for payment_method in payment_methods %}
sum(case when payment_method = '{{ payment_method }}' then amount end)
as {{ slugify(payment_method) }}_amount,

{% endfor %}
...
```

```sql
select
order_id,

sum(case when payment_method = 'Venmo App' then amount end)
as venmo_app_amount,

sum(case when payment_method = 'ca$h money' then amount end)
as cah_money_amount,
...
```

### Materializations
#### insert_by_period ([source](macros/materializations/insert_by_period_materialization.sql))
`insert_by_period` allows dbt to insert records into a table one period (i.e. day, week) at a time.
Expand Down
3 changes: 2 additions & 1 deletion dbt_project.yml
Original file line number Diff line number Diff line change
@@ -1,7 +1,8 @@
name: 'dbt_utils'
version: '0.1.0'

require-dbt-version: [">=0.18.0", "<0.20.0"]
require-dbt-version: [">=0.19.0", "<0.20.0"]

config-version: 2

target-path: "target"
Expand Down
7 changes: 7 additions & 0 deletions integration_tests/tests/jinja_helpers/test_slugify.sql
Original file line number Diff line number Diff line change
@@ -0,0 +1,7 @@
{% if dbt_utils.slugify('!Hell0 world-hi') == 'hell0_world_hi' %}
{# Return 0 rows for the test to pass #}
select 1 limit 0
{% else %}
{# Return >0 rows for the test to fail #}
select 1
{% endif %}
File renamed without changes.
File renamed without changes.
File renamed without changes.
12 changes: 12 additions & 0 deletions macros/jinja_helpers/slugify.sql
Original file line number Diff line number Diff line change
@@ -0,0 +1,12 @@
{% macro slugify(string) %}

{#- Lower case the string -#}
{% set string = string | lower %}
{#- Replace spaces and dashes with underscores -#}
{% set string = modules.re.sub('[ -]+', '_', string) %}
{#- Only take letters, numbers, and underscores -#}
{% set string = modules.re.sub('[^a-z0-9_]+', '', string) %}

{{ return(string) }}

{% endmacro %}
2 changes: 1 addition & 1 deletion macros/sql/pivot.sql
Original file line number Diff line number Diff line change
Expand Up @@ -78,7 +78,7 @@ Arguments:
{% if quote_identifiers %}
as {{ adapter.quote(prefix ~ v ~ suffix) }}
{% else %}
as {{prefix ~ v ~ suffix }}
as {{ dbt_utils.slugify(prefix ~ v ~ suffix) }}
{% endif %}
{% endif %}
{% if not loop.last %},{% endif %}
Expand Down

0 comments on commit 445dbfd

Please sign in to comment.