Added new "filters" python file, along with a "hash" filter. This can… #18000

alexander-marquardt · 2022-10-14T16:07:18Z

Added new Jinja filter for calculating the hash of fields in the low code connector/transformation/AddFields functionality.

I added a new filters.py python file, in which I have declared a "hash" filter. This can be extended to include other custom filters in the future. I also modified jinja.py so that the new hash functionality is available from the low code yaml declaration.

This can be used in the calculation of values in the AddFields transformation as follows: value: "{{ record['rates']['CAD'] | hash('md5', 'mysalt') }}"

Addresses : Creation of a Jinja filter that allows hashing of sensitive data before arriving at a destinationn

What

Adds a custom jinja filter for hashing within the low-code transform functionality.

How

Wrapper around hashlib

🚨 User Impact 🚨

Will allow additional functionality that users did not previously have, and will better support GDPR compliance.

Pre-merge Checklist

Not sure .. will need to discuss with engineering

Community member or Airbyter

Airbyter

… be extended to include other custom filters in the future.

girarda

Love it! Thanks for contributing @alexander-marquardt

girarda · 2022-10-15T01:13:06Z

airbyte-cdk/python/airbyte_cdk/sources/declarative/interpolation/filters.py

+      - type: AddFields
+        fields:
+          - path: ["some_new_path"]
+            value: "{{ record['rates']['CAD'] | hash('md5', 'mysalt')  }}"


can we add an example as a unit test?

I'll add a unit test that verifies the functionality

…xists

alexander-marquardt · 2022-10-15T16:22:50Z

Note: the md5 hash, when executed without a salt is not matching the expected md5 hash , from command line, which can be validated as follows: echo -n foobar |md5 versus the jinja code: "{{ 'xxx' | hash('md5') }}".

I will figure out the reason for this difference and update the code once I understand the reason this is happening.

…sh unless it is actually passed in.

alexander-marquardt · 2022-10-15T16:28:24Z

Oh, that was caused the application of a "None" salt when it was not passed in ... this was fixed by placing a condition around the application of the salt so that it does not modify the hash unless it is explicitly used. -- the reason is because I am converting all input values into string before encoding, and str(None) is "None", which was effectively being concatenated to the value before the digest was created.

alexander-marquardt · 2022-10-17T11:56:19Z

@girarda - I have added unit tests to validate the hash functionality. I think it is ready to merge?

girarda · 2022-10-17T16:13:33Z

@alexander-marquardt this looks good. Only outstanding issue is the flake check failing because of unused imports in test_filters .py

./unit_tests/sources/declarative/interpolation/test_filters.py:7:1: F401 'pytest' imported but unused
See https://github.com/airbytehq/airbyte/actions/runs/3265231904/jobs/5367150554

Next step will be update the changelog and bump the version in setup.py

alexander-marquardt · 2022-10-17T19:49:24Z

@girarda - I've updated the changelog and setup to reflect the newest version 0.1.102, and removed the import of pytest from the unit test.

girarda · 2022-10-18T03:55:50Z

/publish-cdk dry-run=false

🕑 https://github.com/airbytehq/airbyte/actions/runs/3270473079
✅ https://github.com/airbytehq/airbyte/actions/runs/3270473079

#18000) * Added new "filters" python file, along with a "hash" filter. This can be extended to include other custom filters in the future. * Added additional comments * Moved usage of the hash_obj inside the conditional that confirms it exists * Moved the hash function call inside a condition to ensure that it exists * Fixed the application of the salt , so that it does not modify the hash unless it is actually passed in. * Added unit tests to validate new jinja hash functionality * Updated unit test to pass numeric value as a float instead of string * Removed unreferenced import to pytest * Updated version * format * format * format * format * format Co-authored-by: Alexandre Girard <alexandre@airbyte.io>

* Implement ColumnSortButton component * Updates component name; Moves component to ui/Table folder; Refactors formattedMessageId property into using render content as children directly; Removes minor SortIcon component * Update airbyte-webapp/src/App.tsx Co-authored-by: Edmundo Ruiz Ghanem <168664+edmundito@users.noreply.github.com> * Updates next properties: wasActive -> isActive, lowToLarge -> isAscending * Skip psql stop in acceptance test for gke (#18023) * Checks for iterator hasNext element (#18041) * Checks for iterator hasNext element * Fix linter with newline * Add Message Migration to Destination Connection Checks (#17954) * Add Message Migration to Destination Connection Checks * Fix test setup * Update helm release workflow (#18048) * Update workflow * Update trigger rules * fix: Update release workflow with abillity to add tags * Update workflow * Remove unused `airbyte-cli` (#18009) * 🐛 [low-code] $options shouldn't overwrite values that are already defined (#18060) * fix * Add missing test * remove prints * extract to method * rename * Add missing test * rename * bump * Update helm chart comments (#18072) * Update helm charts (#18073) * add test * fix chart.yaml * 16250 Destination Redis: Add SSH support (#17951) * 16250 Destination Redis: Add SSH support * 16250 Resolve port issue * 11679 Bump version * auto-bump connector version Co-authored-by: Octavia Squidington III <octavia-squidington-iii@users.noreply.github.com> * Bump helm chart version reference to 0.40.20 (#18074) * Bump helm chart version reference to 0.40.20 * remove binary Co-authored-by: xpuska513 <xpuska513@users.noreply.github.com> Co-authored-by: Kyryl Skobylko <xpuska513@gmail.com> * Helm Chart: Create service annotations for airbyte-server (#17932) * Support annotations for airbyte-server as well, update version and update docs. * Fix auto-indent. Co-authored-by: Kyryl Skobylko <xpuska513@gmail.com> * Bmoric/remove dep server worker (#17894) * test [ci skip] * Autogenerated files * Add missing annotation * Remove unused json2Schema block from worker * Move tess * Missing deps and format * Fix test build * TMP * Add missing dependencies * PR comments * Tmp * [ci skip] Tmp * Fix acceptance test and add the seed dependency * Fix build * For diff * tmp * Build pass * make the worker to be on the platform only * fix setting.yaml * Fix pmd * Fix Cron * Add chart * Fix cron * Fix server build.gradle * Fix jar conflict * PR comments * Add cron micronaut environemnt * Updated connector catalog page (#18076) * Move the port forward outside of the main docker-compose (#17864) * Bump Airbyte version from 0.40.14 to 0.40.15 (#17970) Co-authored-by: benmoriceau <benmoriceau@users.noreply.github.com> * 🎉 Source Shopify: Add metafield streams (#17962) * 🎉 Source Shopify: Add metafield streams * Source Shopify: fix unittest * Source Shopify: docs update * Source Shopify: fix backward compatibility test * Source Shopify: fix schemas * Source Shopify: fix state filter * Source Shopify: refactor & optimize * Source Shopify: fix test privileges * Source Shopify: fix stream filter * Source Shopify: fix streams * Source Shopify: update abnormal state * Source Shopify: fix abnormal state streams * Source Shopify: fix streams * updated methods, formated code * Source Shopify: typo fix * auto-bump connector version Co-authored-by: Oleksandr Bazarnov <oleksandr.bazarnov@globallogic.com> Co-authored-by: Octavia Squidington III <octavia-squidington-iii@users.noreply.github.com> * fix check for streams that do not use a stream slicer (#18080) * fix check for streams that do not use a stream slicer * increment version and changelog before publish * tolerate database nulls in webhook operation configs (#18084) * Implement webhook operation in the sync workflow (#18022) Implements the webhook operation as part of the sync workflow. - Introduces the new activity implementation - Updates the various interfaces that pass input to get the relevant configs to the sync workflow - Hooks the new activity into the sync workflow - Passes the webhook configs along into the sync workflow job * Bump helm chart version reference to 0.40.22 (#18077) * Added new "filters" python file, along with a "hash" filter. This can… (#18000) * Added new "filters" python file, along with a "hash" filter. This can be extended to include other custom filters in the future. * Added additional comments * Moved usage of the hash_obj inside the conditional that confirms it exists * Moved the hash function call inside a condition to ensure that it exists * Fixed the application of the salt , so that it does not modify the hash unless it is actually passed in. * Added unit tests to validate new jinja hash functionality * Updated unit test to pass numeric value as a float instead of string * Removed unreferenced import to pytest * Updated version * format * format * format * format * format Co-authored-by: Alexandre Girard <alexandre@airbyte.io> * Bump helm chart version reference to 0.40.24 (#18081) * Bump helm chart version reference to 0.40.24 * Update .gitignore Co-authored-by: benmoriceau <benmoriceau@users.noreply.github.com> Co-authored-by: Kyryl Skobylko <xpuska513@gmail.com> * SATs: allow new records in a sequential read for full refresh test (#17660) * SATs: allow new records in a sequential read for full refresh test * SATs: upd changelog * SATs: change the output when failing full refresh test * SATs: upd according to code review * Source facebook-marketing: remove `pixel` from custom conversions stream (#18045) * #744 source facebook-marketing: rm pixel from custom conversions stream * #744 source fb marketing: upd changelog * #744 source facebook-marketing - add custom_conversions to the test catalog * auto-bump connector version Co-authored-by: Octavia Squidington III <octavia-squidington-iii@users.noreply.github.com> * #17506 fix klaviyo & marketo expected_records (#18101) Co-authored-by: Edmundo Ruiz Ghanem <168664+edmundito@users.noreply.github.com> Co-authored-by: terencecho <3916587+terencecho@users.noreply.github.com> Co-authored-by: Ryan Fu <ryan.fu@airbyte.io> Co-authored-by: Jimmy Ma <gosusnp@users.noreply.github.com> Co-authored-by: Kyryl Skobylko <xpuska513@gmail.com> Co-authored-by: Evan Tahler <evan@airbyte.io> Co-authored-by: Alexandre Girard <alexandre@airbyte.io> Co-authored-by: Yevhen Sukhomud <suhomud@gmail.com> Co-authored-by: Octavia Squidington III <octavia-squidington-iii@users.noreply.github.com> Co-authored-by: github-actions[bot] <41898282+github-actions[bot]@users.noreply.github.com> Co-authored-by: xpuska513 <xpuska513@users.noreply.github.com> Co-authored-by: Prasanth <72515998+sfc-gh-pkommini@users.noreply.github.com> Co-authored-by: Benoit Moriceau <benoit@airbyte.io> Co-authored-by: Amruta Ranade <11484018+Amruta-Ranade@users.noreply.github.com> Co-authored-by: Octavia Squidington III <90398440+octavia-squidington-iii@users.noreply.github.com> Co-authored-by: benmoriceau <benmoriceau@users.noreply.github.com> Co-authored-by: Artem Inzhyyants <36314070+artem1205@users.noreply.github.com> Co-authored-by: Oleksandr Bazarnov <oleksandr.bazarnov@globallogic.com> Co-authored-by: Brian Lai <51336873+brianjlai@users.noreply.github.com> Co-authored-by: Michael Siega <109092231+mfsiega-airbyte@users.noreply.github.com> Co-authored-by: Alexander Marquardt <alexander.marquardt@gmail.com> Co-authored-by: Denys Davydov <davydov.den18@gmail.com>

airbytehq#18000) * Added new "filters" python file, along with a "hash" filter. This can be extended to include other custom filters in the future. * Added additional comments * Moved usage of the hash_obj inside the conditional that confirms it exists * Moved the hash function call inside a condition to ensure that it exists * Fixed the application of the salt , so that it does not modify the hash unless it is actually passed in. * Added unit tests to validate new jinja hash functionality * Updated unit test to pass numeric value as a float instead of string * Removed unreferenced import to pytest * Updated version * format * format * format * format * format Co-authored-by: Alexandre Girard <alexandre@airbyte.io>

* Implement ColumnSortButton component * Updates component name; Moves component to ui/Table folder; Refactors formattedMessageId property into using render content as children directly; Removes minor SortIcon component * Update airbyte-webapp/src/App.tsx Co-authored-by: Edmundo Ruiz Ghanem <168664+edmundito@users.noreply.github.com> * Updates next properties: wasActive -> isActive, lowToLarge -> isAscending * Skip psql stop in acceptance test for gke (airbytehq#18023) * Checks for iterator hasNext element (airbytehq#18041) * Checks for iterator hasNext element * Fix linter with newline * Add Message Migration to Destination Connection Checks (airbytehq#17954) * Add Message Migration to Destination Connection Checks * Fix test setup * Update helm release workflow (airbytehq#18048) * Update workflow * Update trigger rules * fix: Update release workflow with abillity to add tags * Update workflow * Remove unused `airbyte-cli` (airbytehq#18009) * 🐛 [low-code] $options shouldn't overwrite values that are already defined (airbytehq#18060) * fix * Add missing test * remove prints * extract to method * rename * Add missing test * rename * bump * Update helm chart comments (airbytehq#18072) * Update helm charts (airbytehq#18073) * add test * fix chart.yaml * 16250 Destination Redis: Add SSH support (airbytehq#17951) * 16250 Destination Redis: Add SSH support * 16250 Resolve port issue * 11679 Bump version * auto-bump connector version Co-authored-by: Octavia Squidington III <octavia-squidington-iii@users.noreply.github.com> * Bump helm chart version reference to 0.40.20 (airbytehq#18074) * Bump helm chart version reference to 0.40.20 * remove binary Co-authored-by: xpuska513 <xpuska513@users.noreply.github.com> Co-authored-by: Kyryl Skobylko <xpuska513@gmail.com> * Helm Chart: Create service annotations for airbyte-server (airbytehq#17932) * Support annotations for airbyte-server as well, update version and update docs. * Fix auto-indent. Co-authored-by: Kyryl Skobylko <xpuska513@gmail.com> * Bmoric/remove dep server worker (airbytehq#17894) * test [ci skip] * Autogenerated files * Add missing annotation * Remove unused json2Schema block from worker * Move tess * Missing deps and format * Fix test build * TMP * Add missing dependencies * PR comments * Tmp * [ci skip] Tmp * Fix acceptance test and add the seed dependency * Fix build * For diff * tmp * Build pass * make the worker to be on the platform only * fix setting.yaml * Fix pmd * Fix Cron * Add chart * Fix cron * Fix server build.gradle * Fix jar conflict * PR comments * Add cron micronaut environemnt * Updated connector catalog page (airbytehq#18076) * Move the port forward outside of the main docker-compose (airbytehq#17864) * Bump Airbyte version from 0.40.14 to 0.40.15 (airbytehq#17970) Co-authored-by: benmoriceau <benmoriceau@users.noreply.github.com> * 🎉 Source Shopify: Add metafield streams (airbytehq#17962) * 🎉 Source Shopify: Add metafield streams * Source Shopify: fix unittest * Source Shopify: docs update * Source Shopify: fix backward compatibility test * Source Shopify: fix schemas * Source Shopify: fix state filter * Source Shopify: refactor & optimize * Source Shopify: fix test privileges * Source Shopify: fix stream filter * Source Shopify: fix streams * Source Shopify: update abnormal state * Source Shopify: fix abnormal state streams * Source Shopify: fix streams * updated methods, formated code * Source Shopify: typo fix * auto-bump connector version Co-authored-by: Oleksandr Bazarnov <oleksandr.bazarnov@globallogic.com> Co-authored-by: Octavia Squidington III <octavia-squidington-iii@users.noreply.github.com> * fix check for streams that do not use a stream slicer (airbytehq#18080) * fix check for streams that do not use a stream slicer * increment version and changelog before publish * tolerate database nulls in webhook operation configs (airbytehq#18084) * Implement webhook operation in the sync workflow (airbytehq#18022) Implements the webhook operation as part of the sync workflow. - Introduces the new activity implementation - Updates the various interfaces that pass input to get the relevant configs to the sync workflow - Hooks the new activity into the sync workflow - Passes the webhook configs along into the sync workflow job * Bump helm chart version reference to 0.40.22 (airbytehq#18077) * Added new "filters" python file, along with a "hash" filter. This can… (airbytehq#18000) * Added new "filters" python file, along with a "hash" filter. This can be extended to include other custom filters in the future. * Added additional comments * Moved usage of the hash_obj inside the conditional that confirms it exists * Moved the hash function call inside a condition to ensure that it exists * Fixed the application of the salt , so that it does not modify the hash unless it is actually passed in. * Added unit tests to validate new jinja hash functionality * Updated unit test to pass numeric value as a float instead of string * Removed unreferenced import to pytest * Updated version * format * format * format * format * format Co-authored-by: Alexandre Girard <alexandre@airbyte.io> * Bump helm chart version reference to 0.40.24 (airbytehq#18081) * Bump helm chart version reference to 0.40.24 * Update .gitignore Co-authored-by: benmoriceau <benmoriceau@users.noreply.github.com> Co-authored-by: Kyryl Skobylko <xpuska513@gmail.com> * SATs: allow new records in a sequential read for full refresh test (airbytehq#17660) * SATs: allow new records in a sequential read for full refresh test * SATs: upd changelog * SATs: change the output when failing full refresh test * SATs: upd according to code review * Source facebook-marketing: remove `pixel` from custom conversions stream (airbytehq#18045) * airbytehq#744 source facebook-marketing: rm pixel from custom conversions stream * airbytehq#744 source fb marketing: upd changelog * airbytehq#744 source facebook-marketing - add custom_conversions to the test catalog * auto-bump connector version Co-authored-by: Octavia Squidington III <octavia-squidington-iii@users.noreply.github.com> * #17506 fix klaviyo & marketo expected_records (airbytehq#18101) Co-authored-by: Edmundo Ruiz Ghanem <168664+edmundito@users.noreply.github.com> Co-authored-by: terencecho <3916587+terencecho@users.noreply.github.com> Co-authored-by: Ryan Fu <ryan.fu@airbyte.io> Co-authored-by: Jimmy Ma <gosusnp@users.noreply.github.com> Co-authored-by: Kyryl Skobylko <xpuska513@gmail.com> Co-authored-by: Evan Tahler <evan@airbyte.io> Co-authored-by: Alexandre Girard <alexandre@airbyte.io> Co-authored-by: Yevhen Sukhomud <suhomud@gmail.com> Co-authored-by: Octavia Squidington III <octavia-squidington-iii@users.noreply.github.com> Co-authored-by: github-actions[bot] <41898282+github-actions[bot]@users.noreply.github.com> Co-authored-by: xpuska513 <xpuska513@users.noreply.github.com> Co-authored-by: Prasanth <72515998+sfc-gh-pkommini@users.noreply.github.com> Co-authored-by: Benoit Moriceau <benoit@airbyte.io> Co-authored-by: Amruta Ranade <11484018+Amruta-Ranade@users.noreply.github.com> Co-authored-by: Octavia Squidington III <90398440+octavia-squidington-iii@users.noreply.github.com> Co-authored-by: benmoriceau <benmoriceau@users.noreply.github.com> Co-authored-by: Artem Inzhyyants <36314070+artem1205@users.noreply.github.com> Co-authored-by: Oleksandr Bazarnov <oleksandr.bazarnov@globallogic.com> Co-authored-by: Brian Lai <51336873+brianjlai@users.noreply.github.com> Co-authored-by: Michael Siega <109092231+mfsiega-airbyte@users.noreply.github.com> Co-authored-by: Alexander Marquardt <alexander.marquardt@gmail.com> Co-authored-by: Denys Davydov <davydov.den18@gmail.com>

Added new "filters" python file, along with a "hash" filter. This can…

cb576f8

… be extended to include other custom filters in the future.

alexander-marquardt requested a review from a team as a code owner October 14, 2022 16:07

github-actions bot added the CDK Connector Development Kit label Oct 14, 2022

Added additional comments

470a14c

girarda reviewed Oct 15, 2022

View reviewed changes

alex.marquardt@airbyte.io added 2 commits October 15, 2022 18:00

Moved usage of the hash_obj inside the conditional that confirms it e…

b979352

…xists

Moved the hash function call inside a condition to ensure that it exists

139db62

Fixed the application of the salt , so that it does not modify the ha…

2359a51

…sh unless it is actually passed in.

Added unit tests to validate new jinja hash functionality

dabe91e

Updated unit test to pass numeric value as a float instead of string

983bb85

alex.marquardt@airbyte.io added 3 commits October 17, 2022 18:22

Removed unreferenced import to pytest

8fb64e5

Merge branch 'master' into custom-jinja-filters-alexm

1a5e10d

Updated version

2936305

girarda added 7 commits October 17, 2022 12:51

Merge branch 'master' into custom-jinja-filters-alexm

907e911

merge

d7eba5f

format

8b5fb64

format

f5c332e

format

b2b86d9

format

1055f69

format

addfb06

Merge branch 'master' into custom-jinja-filters-alexm

daf78b7

girarda approved these changes Oct 18, 2022

View reviewed changes

girarda merged commit 26b6317 into master Oct 18, 2022

girarda deleted the custom-jinja-filters-alexm branch October 18, 2022 06:19

octavia-squidington-iii mentioned this pull request Oct 20, 2022

Bump Airbyte version from 0.40.15 to 0.40.16 #18268

Merged

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Added new "filters" python file, along with a "hash" filter. This can… #18000

Added new "filters" python file, along with a "hash" filter. This can… #18000

alexander-marquardt commented Oct 14, 2022 •

edited

Loading

girarda left a comment

girarda Oct 15, 2022

alexander-marquardt Oct 17, 2022

alexander-marquardt commented Oct 15, 2022

alexander-marquardt commented Oct 15, 2022 •

edited

Loading

alexander-marquardt commented Oct 17, 2022 •

edited

Loading

girarda commented Oct 17, 2022

alexander-marquardt commented Oct 17, 2022 •

edited

Loading

girarda commented Oct 18, 2022 •

edited by github-actions bot

Loading

Added new "filters" python file, along with a "hash" filter. This can… #18000

Added new "filters" python file, along with a "hash" filter. This can… #18000

Conversation

alexander-marquardt commented Oct 14, 2022 • edited Loading

What

How

🚨 User Impact 🚨

Pre-merge Checklist

Community member or Airbyter

girarda left a comment

Choose a reason for hiding this comment

girarda Oct 15, 2022

Choose a reason for hiding this comment

alexander-marquardt Oct 17, 2022

Choose a reason for hiding this comment

alexander-marquardt commented Oct 15, 2022

alexander-marquardt commented Oct 15, 2022 • edited Loading

alexander-marquardt commented Oct 17, 2022 • edited Loading

girarda commented Oct 17, 2022

alexander-marquardt commented Oct 17, 2022 • edited Loading

girarda commented Oct 18, 2022 • edited by github-actions bot Loading

alexander-marquardt commented Oct 14, 2022 •

edited

Loading

alexander-marquardt commented Oct 15, 2022 •

edited

Loading

alexander-marquardt commented Oct 17, 2022 •

edited

Loading

alexander-marquardt commented Oct 17, 2022 •

edited

Loading

girarda commented Oct 18, 2022 •

edited by github-actions bot

Loading