Tdl 19964 add missing tap tester tests #65

namrata270998 · 2022-07-27T05:41:16Z

Description of change

Updated several test cases according to the code changes
Skipped missing fields from the all_fields test case
Added assertion for primary keys in the automatic_fields test
Added an assertion that replication value is greater than the bookmark in bookmarks test case
Added an assertion for no duplicate metadata entry in the discover test case
Added an assertion for actual replication that matches our expected replication method in the discover test case
Added pagination test case
Added replication key is greater or equal to start_date in start_date test
Added primary keys assertion in start_date test

Note: Test cases will pass once after all the code of dict-to-class will be merged

Manual QA steps

Risks

Rollback steps

revert this branch

savan-chovatiya · 2022-07-27T16:23:59Z

tests/test_typeform_bookmarks.py

@@ -107,7 +58,7 @@ def test_run(self):
        ##########################################################################

        new_states = {'bookmarks': dict()}
-        simulated_states = self.calculated_states_by_stream(first_sync_bookmarks)
+        simulated_states = {'forms': {'last_updated_at': '2022-07-22T16:00:47Z'}, 'landings': {'mVn2wE55': {'landed_at': '2022-07-21T00:00:00Z'}, 'xJ8emTTy': {'landed_at': '2022-07-24T16:49:54Z'}}, 'answers': {'mVn2wE55': {'landed_at': '2022-07-21T00:00:00Z'}, 'xJ8emTTy': {'landed_at': '2022-07-24T16:49:54Z'}}}


Divide this into multiline for better readability.

Added multiline for better readability

savan-chovatiya · 2022-07-27T16:24:52Z

tests/test_typeform_bookmarks.py

+                                for record in second_sync_messages:
+
+                                    # Verify the second sync records respect the previous (simulated) bookmark value
+                                    # replication_key_value = [record.get(replication_key) if record.get('_sdc_form_id') == form_key]


Remove commented code from the test.

removed commented code

savan-chovatiya · 2022-07-27T16:26:02Z

tests/test_typeform_pagination.py

@@ -0,0 +1,75 @@
+#for tap-typeform pagination


Remove the above the line.

removed unwanted comments

savan-chovatiya · 2022-07-27T16:27:05Z

tests/test_typeform_start_date.py

+        • verify that a sync with a later start date has at least one record synced
+        and less records than the 1st sync with a previous start date
+        • verify that each stream has less records than the earlier start date sync
+        • verify all data from later start data has bookmark values >= start_date


Keep the first character of a sentence in upper case.

Updated to uppercase

savan-chovatiya · 2022-07-27T16:39:55Z

tests/test_typeform_bookmarks.py

+                                    else:
+                                        continue


Remove this else part as there is nothing after it inside a for a loop.

removed the else

savan-chovatiya · 2022-07-27T16:45:22Z

tests/test_typeform_bookmarks.py

-
-
-                            # Verify the number of records in the 2nd sync is less then the first
+                            if stream != 'answers':


Add code comment here like why the above stream is excluded for some of the assertions.

Added comments

* TDL-19957 update dict based to class based * updated while condition * updated while condition * removed incremental_range from REQUIRED_CONFIG_KEYS * updated discover and schema file * updated to replication_key instead of keys * updated schemas and added comments * added unittests for code coverage * added unittests for sync.py * added parameterized in setup.py * added parameterized in config.yml and updated unittests * updated the replication key to list instead of a single key * fixed the issue for Keyerror of form_id in answers stream * updated unittests * added configurable page size param * handled page_size for 0 and updated unittests * resolved bug fixes for pagination and removed incompleted_forms_only param * added new stream unsubmitted landings * resolved PR review comments * added page_size in example config * added back incremental_range in the tap-tester * raised exc instead of fatal error message and updated unittests * resolved PR comments * fixed the issue when page_size not passed in config

* Tdl 19964 add missing tap tester tests (#65) * TDL-19964 Added missing tap-tester commits * updated tap-tester tests * updated bookmarks tap-tester test * Updated and removed extra comments * updated start date as per format * updated pagination test case * resolved review comments * added new stream in tap-tester * fixed the cci issues * added missing assertion for all fields * removed get_logger() * added logger instead of print * TDL-19957 Update dict based to class based implementation (#64) * TDL-19957 update dict based to class based * updated while condition * updated while condition * removed incremental_range from REQUIRED_CONFIG_KEYS * updated discover and schema file * updated to replication_key instead of keys * updated schemas and added comments * added unittests for code coverage * added unittests for sync.py * added parameterized in setup.py * added parameterized in config.yml and updated unittests * updated the replication key to list instead of a single key * fixed the issue for Keyerror of form_id in answers stream * updated unittests * added configurable page size param * handled page_size for 0 and updated unittests * resolved bug fixes for pagination and removed incompleted_forms_only param * added new stream unsubmitted landings * resolved PR review comments * added page_size in example config * added back incremental_range in the tap-tester * raised exc instead of fatal error message and updated unittests * resolved PR comments * fixed the issue when page_size not passed in config * TDL-19801: Tap fetch data for sub-questions (#62) * TDL-19801: Tap does not support fetching data for the questions nested within a Question Group. * addressed the comments * add unittest * modify funciton comment * formated test_case value in unittest * formated expected_case * Updated schema in questions.json * resolved build fail error * resolve build fail error * updated unittest * updated setup.py * add parameterized * change start date in start_date_test * make change in bookmark test * Updated unit test case. * Updated schemas and keyerror. Co-authored-by: Jay Tilala <jay.tilala@CDSYS.LOCAL> Co-authored-by: prijendev <prijen.khokhani@crestdatasys.com> * TDL-19959 added missing fields (#63) * added missing fields * TDL-19957 update dict based to class based * updated while condition * updated while condition * removed incremental_range from REQUIRED_CONFIG_KEYS * TDL-19964 Added missing tap-tester commits * updated discover and schema file * updated tap-tester tests * updated to replication_key instead of keys * updated schemas and added comments * added unittests for code coverage * updated bookmarks tap-tester test * added unittests for sync.py * added parameterized in setup.py * added parameterized in config.yml and updated unittests * updated the replication key to list instead of a single key * Updated and removed extra comments * fixed the issue for Keyerror of form_id in answers stream * updated unittests * updated start date as per format * added configurable page size param * updated pagination test case * handled page_size for 0 and updated unittests * resolved review comments * resolved bug fixes for pagination and removed incompleted_forms_only param * updated array type schema * added new stream unsubmitted landings * added new stream in tap-tester * updated indentation * resolved PR review comments * updated indentation to use 2 spaces * added page_size in example config * added back incremental_range in the tap-tester * fixed the cci issues * added missing fields to a dict * added missing assertion for all fields * updated comment * raised exc instead of fatal error message and updated unittests * removed get_logger() * added logger instead of print * Updated schema for questions. * Removed duplicate assertion in all_fields test. Co-authored-by: prijendev <prijen.khokhani@crestdatasys.com> * Updated schema for answers stream. Co-authored-by: namrata270998 <75604662+namrata270998@users.noreply.github.com> Co-authored-by: jtilala <104966482+jtilala@users.noreply.github.com> Co-authored-by: Jay Tilala <jay.tilala@CDSYS.LOCAL>

* Tdl 19964 add missing tap tester tests (#65) * TDL-19964 Added missing tap-tester commits * updated tap-tester tests * updated bookmarks tap-tester test * Updated and removed extra comments * updated start date as per format * updated pagination test case * resolved review comments * added new stream in tap-tester * fixed the cci issues * added missing assertion for all fields * removed get_logger() * added logger instead of print * TDL-19957 Update dict based to class based implementation (#64) * TDL-19957 update dict based to class based * updated while condition * updated while condition * removed incremental_range from REQUIRED_CONFIG_KEYS * updated discover and schema file * updated to replication_key instead of keys * updated schemas and added comments * added unittests for code coverage * added unittests for sync.py * added parameterized in setup.py * added parameterized in config.yml and updated unittests * updated the replication key to list instead of a single key * fixed the issue for Keyerror of form_id in answers stream * updated unittests * added configurable page size param * handled page_size for 0 and updated unittests * resolved bug fixes for pagination and removed incompleted_forms_only param * added new stream unsubmitted landings * resolved PR review comments * added page_size in example config * added back incremental_range in the tap-tester * raised exc instead of fatal error message and updated unittests * resolved PR comments * fixed the issue when page_size not passed in config * TDL-19801: Tap fetch data for sub-questions (#62) * TDL-19801: Tap does not support fetching data for the questions nested within a Question Group. * addressed the comments * add unittest * modify funciton comment * formated test_case value in unittest * formated expected_case * Updated schema in questions.json * resolved build fail error * resolve build fail error * updated unittest * updated setup.py * add parameterized * change start date in start_date_test * make change in bookmark test * Updated unit test case. * Updated schemas and keyerror. Co-authored-by: Jay Tilala <jay.tilala@CDSYS.LOCAL> Co-authored-by: prijendev <prijen.khokhani@crestdatasys.com> * TDL-19959 added missing fields (#63) * added missing fields * TDL-19957 update dict based to class based * updated while condition * updated while condition * removed incremental_range from REQUIRED_CONFIG_KEYS * TDL-19964 Added missing tap-tester commits * updated discover and schema file * updated tap-tester tests * updated to replication_key instead of keys * updated schemas and added comments * added unittests for code coverage * updated bookmarks tap-tester test * added unittests for sync.py * added parameterized in setup.py * added parameterized in config.yml and updated unittests * updated the replication key to list instead of a single key * Updated and removed extra comments * fixed the issue for Keyerror of form_id in answers stream * updated unittests * updated start date as per format * added configurable page size param * updated pagination test case * handled page_size for 0 and updated unittests * resolved review comments * resolved bug fixes for pagination and removed incompleted_forms_only param * updated array type schema * added new stream unsubmitted landings * added new stream in tap-tester * updated indentation * resolved PR review comments * updated indentation to use 2 spaces * added page_size in example config * added back incremental_range in the tap-tester * fixed the cci issues * added missing fields to a dict * added missing assertion for all fields * updated comment * raised exc instead of fatal error message and updated unittests * removed get_logger() * added logger instead of print * Updated schema for questions. * Removed duplicate assertion in all_fields test. Co-authored-by: prijendev <prijen.khokhani@crestdatasys.com> * Updated schema for answers stream. * update version and changelog * changelog update for v2.0.0 * changelog update for v2.0.0 * minor changelog update Co-authored-by: namrata270998 <75604662+namrata270998@users.noreply.github.com> Co-authored-by: jtilala <104966482+jtilala@users.noreply.github.com> Co-authored-by: Jay Tilala <jay.tilala@CDSYS.LOCAL> Co-authored-by: prijendev <prijen.khokhani@crestdatasys.com> Co-authored-by: kethan1122 <kcherukuri@talend.com> Co-authored-by: rdeshmukh15 <107538720+rdeshmukh15@users.noreply.github.com>

namrata270998 added 4 commits July 22, 2022 13:30

TDL-19964 Added missing tap-tester commits

999d32d

updated tap-tester tests

86ffe72

updated bookmarks tap-tester test

348262e

Updated and removed extra comments

cd435d4

namrata270998 requested review from prijendev, dbshah1212 and savan-chovatiya July 27, 2022 08:57

updated start date as per format

6621815

savan-chovatiya suggested changes Jul 28, 2022

View reviewed changes

namrata270998 added 2 commits July 28, 2022 09:51

updated pagination test case

677eea7

resolved review comments

0dc281b

namrata270998 requested a review from savan-chovatiya July 29, 2022 05:05

savan-chovatiya approved these changes Jul 29, 2022

View reviewed changes

namrata270998 added 3 commits July 29, 2022 13:23

added new stream in tap-tester

e645182

fixed the cci issues

d0089aa

added missing assertion for all fields

e580805

dbshah1212 approved these changes Aug 3, 2022

View reviewed changes

namrata270998 added 2 commits August 3, 2022 10:16

removed get_logger()

75e4566

added logger instead of print

7924ef9

dbshah1212 requested review from RushiT0122, KrisPersonal and kspeer825 August 9, 2022 04:19

kspeer825 approved these changes Aug 19, 2022

View reviewed changes

prijendev merged commit d06cb08 into crest-master Aug 29, 2022

prijendev mentioned this pull request Sep 2, 2022

Crest master #66

Merged

kethan1122 deleted the TDL-19964-add-missing-tap-tester-tests branch April 28, 2023 09:40

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Tdl 19964 add missing tap tester tests #65

Tdl 19964 add missing tap tester tests #65

namrata270998 commented Jul 27, 2022 •

edited

Loading

savan-chovatiya Jul 27, 2022

namrata270998 Jul 29, 2022

savan-chovatiya Jul 27, 2022

namrata270998 Jul 29, 2022

savan-chovatiya Jul 27, 2022

namrata270998 Jul 29, 2022

savan-chovatiya Jul 27, 2022

namrata270998 Jul 29, 2022

savan-chovatiya Jul 27, 2022

namrata270998 Jul 29, 2022

savan-chovatiya Jul 27, 2022

namrata270998 Jul 29, 2022



		# Verify the number of records in the 2nd sync is less then the first
		if stream != 'answers':

Tdl 19964 add missing tap tester tests #65

Tdl 19964 add missing tap tester tests #65

Conversation

namrata270998 commented Jul 27, 2022 • edited Loading

Description of change

Manual QA steps

Risks

Rollback steps

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

namrata270998 commented Jul 27, 2022 •

edited

Loading