TDL-19957 Update dict based to class based implementation #64

namrata270998 · 2022-07-22T10:39:14Z

Description of change

TDL 19957 update dict based implementation to class based

Changed dict-based implementation to class-based implementation.
Created functionality-wise separate files - schema, discover, sync, streams.
Moved logic of REST API call and error handling to client.py.
Created class for incremental, full_table sync.
Created parent class and class for each stream.
Added details code comments for each function.
Added detailed unit test case.
Removed unnecessary unit test case.
Added implementation to write separate bookmark for child and parent stream.
Tap use a minimum bookmark among the selected parent and its child stream.
Tap write maximum bookmark for all selected parents and its child's stream
Tap write all child records of newly updated parent records.
Tap write replication key value of parent's record in the state of child stream.
Split landings into submitted_landings and unsubmitted_landings and introduced a new replication key for both.

Manual QA steps

Risks

Rollback steps

revert this branch

…ub.com/singer-io/tap-typeform into TDL-19957-update-dict-based-to-class-based

…github.com/singer-io/tap-typeform into TDL-19957-update-dict-based-to-class-based

…ub.com/singer-io/tap-typeform into TDL-19957-update-dict-based-to-class-based

NevilParikh14 · 2022-07-28T07:14:14Z

tap_typeform/__init__.py


-    mismatched_forms = _compare_forms(config_forms, api_forms)
+    mismatched_forms = config_forms.difference(api_forms)

    if len(mismatched_forms) > 0:
        LOGGER.fatal(f"FormMistmatchError: forms {mismatched_forms} not returned by API")


Can you please add an form mismatch message in the FormMistmatchError exception raised below instead of defining Logger for the same?

Added the message

NevilParikh14 · 2022-07-28T08:18:53Z

tap_typeform/streams.py

+    key_properties = ['form_id', 'question_id']
+    endpoint = 'forms/{}'
+    params = {
+                'since': '',


Can you please remove this 'since' param? As it isn't used for the FULL_TABLE stream.

Removed the param

dbshah1212 · 2022-08-01T11:02:51Z

tap_typeform/streams.py

+    def add_fields_at_1st_level(self, record, additional_data = {}):
+        data_type = record.get('type')
+
+        if data_type in ['choice', 'choices', 'payment']:


Add code comments

Added comments

dbshah1212 · 2022-08-01T11:03:30Z

tap_typeform/streams.py

+        else:
+            answer_value = record.get(data_type)
+
+        record.update({


Add code comment

Added comments

dbshah1212 · 2022-08-01T11:20:33Z

tests/unittests/test_exception_handling.py

-    return Mocksession(json_data, status_code, content, headers, raise_error)
-
+from tap_typeform.client import ERROR_CODE_EXCEPTION_MAPPING
+import tap_typeform.client as client_



unit tests for the TypeformError for method raise_for_error are missing.

Added unittests

dbshah1212 · 2022-08-01T11:23:30Z

tests/unittests/test_exception_handling.py

-        self.assertEqual(mocked_session.call_count, 3)
-
+        with self.assertRaises(error):
+            client.request(url)


unittests of raw data items Logger is missing for the request method.

Added unittests

dbshah1212 · 2022-08-01T11:26:45Z

tests/unittests/test_sync_obj.py

+class TestAddFieldAt1StLevel(unittest.TestCase):
+    """
+    Test `add_fields_at_1st_level` method for all streams.
+    """


Below lines aren't covered in test cases:

answer_value = json.dumps(record.get(data_type))

answer_value = str(record.get(data_type))

Added unittests

…ub.com/singer-io/tap-typeform into TDL-19957-update-dict-based-to-class-based

bryantgray · 2022-08-09T17:36:36Z

tap_typeform/client.py

+        try:
+            if page_size is None:
+                pass
+            elif int(float(page_size)) > 0:


Could you add a comment explaining the int(float()). It's not immediately clear why there is a conversion to float before int.

We are typecasting first to float to handle the corner scenarios such as string float values i.e. "10.6" which would throw an exception when directly typecasted to int

* TDL-19964 Added missing tap-tester commits * updated tap-tester tests * updated bookmarks tap-tester test * Updated and removed extra comments * updated start date as per format * updated pagination test case * resolved review comments * added new stream in tap-tester * fixed the cci issues * added missing assertion for all fields * removed get_logger() * added logger instead of print * TDL-19957 Update dict based to class based implementation (#64) * TDL-19957 update dict based to class based * updated while condition * updated while condition * removed incremental_range from REQUIRED_CONFIG_KEYS * updated discover and schema file * updated to replication_key instead of keys * updated schemas and added comments * added unittests for code coverage * added unittests for sync.py * added parameterized in setup.py * added parameterized in config.yml and updated unittests * updated the replication key to list instead of a single key * fixed the issue for Keyerror of form_id in answers stream * updated unittests * added configurable page size param * handled page_size for 0 and updated unittests * resolved bug fixes for pagination and removed incompleted_forms_only param * added new stream unsubmitted landings * resolved PR review comments * added page_size in example config * added back incremental_range in the tap-tester * raised exc instead of fatal error message and updated unittests * resolved PR comments * fixed the issue when page_size not passed in config

* Tdl 19964 add missing tap tester tests (#65) * TDL-19964 Added missing tap-tester commits * updated tap-tester tests * updated bookmarks tap-tester test * Updated and removed extra comments * updated start date as per format * updated pagination test case * resolved review comments * added new stream in tap-tester * fixed the cci issues * added missing assertion for all fields * removed get_logger() * added logger instead of print * TDL-19957 Update dict based to class based implementation (#64) * TDL-19957 update dict based to class based * updated while condition * updated while condition * removed incremental_range from REQUIRED_CONFIG_KEYS * updated discover and schema file * updated to replication_key instead of keys * updated schemas and added comments * added unittests for code coverage * added unittests for sync.py * added parameterized in setup.py * added parameterized in config.yml and updated unittests * updated the replication key to list instead of a single key * fixed the issue for Keyerror of form_id in answers stream * updated unittests * added configurable page size param * handled page_size for 0 and updated unittests * resolved bug fixes for pagination and removed incompleted_forms_only param * added new stream unsubmitted landings * resolved PR review comments * added page_size in example config * added back incremental_range in the tap-tester * raised exc instead of fatal error message and updated unittests * resolved PR comments * fixed the issue when page_size not passed in config * TDL-19801: Tap fetch data for sub-questions (#62) * TDL-19801: Tap does not support fetching data for the questions nested within a Question Group. * addressed the comments * add unittest * modify funciton comment * formated test_case value in unittest * formated expected_case * Updated schema in questions.json * resolved build fail error * resolve build fail error * updated unittest * updated setup.py * add parameterized * change start date in start_date_test * make change in bookmark test * Updated unit test case. * Updated schemas and keyerror. Co-authored-by: Jay Tilala <jay.tilala@CDSYS.LOCAL> Co-authored-by: prijendev <prijen.khokhani@crestdatasys.com> * TDL-19959 added missing fields (#63) * added missing fields * TDL-19957 update dict based to class based * updated while condition * updated while condition * removed incremental_range from REQUIRED_CONFIG_KEYS * TDL-19964 Added missing tap-tester commits * updated discover and schema file * updated tap-tester tests * updated to replication_key instead of keys * updated schemas and added comments * added unittests for code coverage * updated bookmarks tap-tester test * added unittests for sync.py * added parameterized in setup.py * added parameterized in config.yml and updated unittests * updated the replication key to list instead of a single key * Updated and removed extra comments * fixed the issue for Keyerror of form_id in answers stream * updated unittests * updated start date as per format * added configurable page size param * updated pagination test case * handled page_size for 0 and updated unittests * resolved review comments * resolved bug fixes for pagination and removed incompleted_forms_only param * updated array type schema * added new stream unsubmitted landings * added new stream in tap-tester * updated indentation * resolved PR review comments * updated indentation to use 2 spaces * added page_size in example config * added back incremental_range in the tap-tester * fixed the cci issues * added missing fields to a dict * added missing assertion for all fields * updated comment * raised exc instead of fatal error message and updated unittests * removed get_logger() * added logger instead of print * Updated schema for questions. * Removed duplicate assertion in all_fields test. Co-authored-by: prijendev <prijen.khokhani@crestdatasys.com> * Updated schema for answers stream. Co-authored-by: namrata270998 <75604662+namrata270998@users.noreply.github.com> Co-authored-by: jtilala <104966482+jtilala@users.noreply.github.com> Co-authored-by: Jay Tilala <jay.tilala@CDSYS.LOCAL>

* Tdl 19964 add missing tap tester tests (#65) * TDL-19964 Added missing tap-tester commits * updated tap-tester tests * updated bookmarks tap-tester test * Updated and removed extra comments * updated start date as per format * updated pagination test case * resolved review comments * added new stream in tap-tester * fixed the cci issues * added missing assertion for all fields * removed get_logger() * added logger instead of print * TDL-19957 Update dict based to class based implementation (#64) * TDL-19957 update dict based to class based * updated while condition * updated while condition * removed incremental_range from REQUIRED_CONFIG_KEYS * updated discover and schema file * updated to replication_key instead of keys * updated schemas and added comments * added unittests for code coverage * added unittests for sync.py * added parameterized in setup.py * added parameterized in config.yml and updated unittests * updated the replication key to list instead of a single key * fixed the issue for Keyerror of form_id in answers stream * updated unittests * added configurable page size param * handled page_size for 0 and updated unittests * resolved bug fixes for pagination and removed incompleted_forms_only param * added new stream unsubmitted landings * resolved PR review comments * added page_size in example config * added back incremental_range in the tap-tester * raised exc instead of fatal error message and updated unittests * resolved PR comments * fixed the issue when page_size not passed in config * TDL-19801: Tap fetch data for sub-questions (#62) * TDL-19801: Tap does not support fetching data for the questions nested within a Question Group. * addressed the comments * add unittest * modify funciton comment * formated test_case value in unittest * formated expected_case * Updated schema in questions.json * resolved build fail error * resolve build fail error * updated unittest * updated setup.py * add parameterized * change start date in start_date_test * make change in bookmark test * Updated unit test case. * Updated schemas and keyerror. Co-authored-by: Jay Tilala <jay.tilala@CDSYS.LOCAL> Co-authored-by: prijendev <prijen.khokhani@crestdatasys.com> * TDL-19959 added missing fields (#63) * added missing fields * TDL-19957 update dict based to class based * updated while condition * updated while condition * removed incremental_range from REQUIRED_CONFIG_KEYS * TDL-19964 Added missing tap-tester commits * updated discover and schema file * updated tap-tester tests * updated to replication_key instead of keys * updated schemas and added comments * added unittests for code coverage * updated bookmarks tap-tester test * added unittests for sync.py * added parameterized in setup.py * added parameterized in config.yml and updated unittests * updated the replication key to list instead of a single key * Updated and removed extra comments * fixed the issue for Keyerror of form_id in answers stream * updated unittests * updated start date as per format * added configurable page size param * updated pagination test case * handled page_size for 0 and updated unittests * resolved review comments * resolved bug fixes for pagination and removed incompleted_forms_only param * updated array type schema * added new stream unsubmitted landings * added new stream in tap-tester * updated indentation * resolved PR review comments * updated indentation to use 2 spaces * added page_size in example config * added back incremental_range in the tap-tester * fixed the cci issues * added missing fields to a dict * added missing assertion for all fields * updated comment * raised exc instead of fatal error message and updated unittests * removed get_logger() * added logger instead of print * Updated schema for questions. * Removed duplicate assertion in all_fields test. Co-authored-by: prijendev <prijen.khokhani@crestdatasys.com> * Updated schema for answers stream. * update version and changelog * changelog update for v2.0.0 * changelog update for v2.0.0 * minor changelog update Co-authored-by: namrata270998 <75604662+namrata270998@users.noreply.github.com> Co-authored-by: jtilala <104966482+jtilala@users.noreply.github.com> Co-authored-by: Jay Tilala <jay.tilala@CDSYS.LOCAL> Co-authored-by: prijendev <prijen.khokhani@crestdatasys.com> Co-authored-by: kethan1122 <kcherukuri@talend.com> Co-authored-by: rdeshmukh15 <107538720+rdeshmukh15@users.noreply.github.com>

namrata270998 added 15 commits July 22, 2022 05:19

TDL-19957 update dict based to class based

39dbb50

updated while condition

8e00cbd

updated while condition

f94f68e

removed incremental_range from REQUIRED_CONFIG_KEYS

f299b8d

updated discover and schema file

5f8e433

Merge branch 'TDL-19964-add-missing-tap-tester-tests' of https://gith…

b76f9cf

…ub.com/singer-io/tap-typeform into TDL-19957-update-dict-based-to-class-based

updated to replication_key instead of keys

4353def

Merge branch 'TDL-19964-add-missing-tap-tester-tests' of https://gith…

e0f5029

…ub.com/singer-io/tap-typeform into TDL-19957-update-dict-based-to-class-based

updated schemas and added comments

69692b7

added unittests for code coverage

89d4e4a

added unittests for sync.py

140cd09

Merge branch 'TDL-19964-add-missing-tap-tester-tests' of https://gith…

9e27ff2

…ub.com/singer-io/tap-typeform into TDL-19957-update-dict-based-to-class-based

added parameterized in setup.py

c3e40e2

added parameterized in config.yml and updated unittests

3249d74

Merge branch 'TDL-19957-update-dict-based-to-class-based' of https://…

c025e28

…github.com/singer-io/tap-typeform into TDL-19957-update-dict-based-to-class-based

namrata270998 changed the base branch from crest-master to TDL-19964-add-missing-tap-tester-tests July 27, 2022 05:41

updated the replication key to list instead of a single key

8b7163c

namrata270998 requested review from prijendev, dbshah1212 and savan-chovatiya July 27, 2022 08:57

namrata270998 added 4 commits July 27, 2022 09:11

fixed the issue for Keyerror of form_id in answers stream

f9989bd

Merge branch 'TDL-19964-add-missing-tap-tester-tests' of https://gith…

bc91c4b

…ub.com/singer-io/tap-typeform into TDL-19957-update-dict-based-to-class-based

updated unittests

3a63e1e

Merge branch 'TDL-19964-add-missing-tap-tester-tests' of https://gith…

8888391

…ub.com/singer-io/tap-typeform into TDL-19957-update-dict-based-to-class-based

namrata270998 changed the title ~~Tdl 19957 Update dict based to class based implementation~~ TDK 19957 Update dict based to class based implementation Jul 28, 2022

namrata270998 changed the title ~~TDK 19957 Update dict based to class based implementation~~ TDL 19957 Update dict based to class based implementation Jul 28, 2022

namrata270998 changed the title ~~TDL 19957 Update dict based to class based implementation~~ TDL-19957 Update dict based to class based implementation Jul 28, 2022

NevilParikh14 reviewed Jul 28, 2022

View reviewed changes

added configurable page size param

4289539

dbshah1212 reviewed Aug 1, 2022

View reviewed changes

namrata270998 added 5 commits August 1, 2022 13:31

resolved PR review comments

e63152f

added page_size in example config

a15099f

added back incremental_range in the tap-tester

728da16

Merge branch 'TDL-19964-add-missing-tap-tester-tests' of https://gith…

651f0cc

…ub.com/singer-io/tap-typeform into TDL-19957-update-dict-based-to-class-based

Merge branch 'TDL-19964-add-missing-tap-tester-tests' of https://gith…

5cc0a73

…ub.com/singer-io/tap-typeform into TDL-19957-update-dict-based-to-class-based

namrata270998 requested review from dbshah1212 and savan-chovatiya August 2, 2022 12:36

raised exc instead of fatal error message and updated unittests

c92dd35

savan-chovatiya approved these changes Aug 2, 2022

View reviewed changes

dbshah1212 approved these changes Aug 3, 2022

View reviewed changes

Merge branch 'TDL-19964-add-missing-tap-tester-tests' of https://gith…

1eb6324

…ub.com/singer-io/tap-typeform into TDL-19957-update-dict-based-to-class-based

dbshah1212 requested review from nitingaikwad1, bryantgray, RushiT0122 and KrisPersonal August 9, 2022 04:21

namrata270998 added 2 commits August 10, 2022 12:50

resolved PR comments

159b6d7

fixed the issue when page_size not passed in config

7c5960e

bryantgray approved these changes Aug 11, 2022

View reviewed changes

namrata270998 requested a review from dmosorast August 12, 2022 10:43

prijendev merged commit 1c636fd into TDL-19964-add-missing-tap-tester-tests Aug 29, 2022

prijendev mentioned this pull request Sep 2, 2022

Crest master #66

Merged

kethan1122 deleted the TDL-19957-update-dict-based-to-class-based branch April 28, 2023 09:40

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

TDL-19957 Update dict based to class based implementation #64

TDL-19957 Update dict based to class based implementation #64

namrata270998 commented Jul 22, 2022 •

edited by dbshah1212

Loading

NevilParikh14 Jul 28, 2022

namrata270998 Jul 29, 2022

NevilParikh14 Jul 28, 2022

namrata270998 Jul 29, 2022

dbshah1212 Aug 1, 2022

namrata270998 Aug 1, 2022

dbshah1212 Aug 1, 2022

namrata270998 Aug 1, 2022

dbshah1212 Aug 1, 2022

namrata270998 Aug 1, 2022

dbshah1212 Aug 1, 2022

namrata270998 Aug 1, 2022

dbshah1212 Aug 1, 2022

namrata270998 Aug 1, 2022

bryantgray Aug 9, 2022

namrata270998 Aug 12, 2022

TDL-19957 Update dict based to class based implementation #64

TDL-19957 Update dict based to class based implementation #64

Conversation

namrata270998 commented Jul 22, 2022 • edited by dbshah1212 Loading

Description of change

Manual QA steps

Risks

Rollback steps

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

namrata270998 commented Jul 22, 2022 •

edited by dbshah1212

Loading