Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

TDL 19530 update dict based implementation to class based #168

Merged
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
70 commits
Select commit Hold shift + click to select a range
e059953
added backoff for 5xx error and fixed indexerror when only passed org…
namrata270998 Jul 1, 2022
66f2f2b
added missing test cases and updated test cases according to code change
NevilParikh14 Jul 6, 2022
3549178
removed tags stream as it is not yet added
NevilParikh14 Jul 6, 2022
dc82926
Updated inturrupted sync assertion
NevilParikh14 Jul 6, 2022
79d8209
Removed bookmark conversion to utc
NevilParikh14 Jul 8, 2022
a2d2b0d
updated inerrupted sync assertions
NevilParikh14 Jul 8, 2022
5442073
resolved review comments
NevilParikh14 Jul 12, 2022
ad1e6fd
updated sreams in parent-child test case
NevilParikh14 Jul 12, 2022
e55bba9
Initial commit.
prijendev Jul 12, 2022
709fdcd
Removed unnecessary unit test cases.
prijendev Jul 12, 2022
f5245f9
updated code comments
namrata270998 Jul 12, 2022
713ccee
Resolved pylint errors.
prijendev Jul 12, 2022
b724ea0
disabled pylint for get_child_records.
prijendev Jul 12, 2022
654b7b1
Merge branch 'TDl-19530-update-dict-based-implementation-to-class-bas…
namrata270998 Jul 12, 2022
4fd734b
Added unittest cases.
prijendev Jul 13, 2022
b72464e
Added filter params for commits and comments stream.
prijendev Jul 13, 2022
9c05ebe
Updated unit test cases.
prijendev Jul 13, 2022
88a7eb3
resolved review comments
NevilParikh14 Jul 13, 2022
8235dbb
resolved review comments
NevilParikh14 Jul 13, 2022
e474ed5
Updated start_date test
NevilParikh14 Jul 13, 2022
32064ad
updated date format conversion in test
NevilParikh14 Jul 13, 2022
7b9bd5b
Updated bookmark logic for parent child streams.
prijendev Jul 14, 2022
517ba1a
Updated credentials check logic.
prijendev Jul 14, 2022
267a435
Updated unittest case.
prijendev Jul 14, 2022
df3361e
Resolved PR comments and handled error for / in repository in config
namrata270998 Jul 14, 2022
deae57a
added unittests for only / in config
namrata270998 Jul 14, 2022
3f6576a
Updated client.py to raise error for wrong org.
prijendev Jul 15, 2022
db4f3cb
Added back creds check logic during object creation time.
prijendev Jul 15, 2022
26e4d42
handled the /repo and raised exception with a list
namrata270998 Jul 15, 2022
25b3dd7
unskipped incremental child streams from assertion
NevilParikh14 Jul 15, 2022
5578009
Make pk of child automatic in parent stream.
prijendev Jul 15, 2022
34ef898
Merge branch 'TDL-19555-add-missing-tap-tester-tests' of https://gith…
namrata270998 Jul 15, 2022
3d88bcf
Merge branch 'TDl-19530-update-dict-based-implementation-to-class-bas…
namrata270998 Jul 15, 2022
c7676b5
updated unittests
namrata270998 Jul 16, 2022
19a14b2
Added logger message to print url for all streams.
prijendev Jul 18, 2022
98adfdf
updated base.py
NevilParikh14 Jul 18, 2022
489f267
Skipped 404 error for all streams.
prijendev Jul 18, 2022
2715ad7
Merge branch 'TDL-19555-add-missing-tap-tester-tests' into TDl-19530-…
prijendev Jul 18, 2022
28cd4af
Removed unnecessary __exit__ method from client
prijendev Jul 18, 2022
d279d09
Updated doc link for each stream.
prijendev Jul 18, 2022
7f82e05
Merge branch 'TDl-19530-update-dict-based-implementation-to-class-bas…
namrata270998 Jul 18, 2022
0161a79
removed duplicates from extract_repos_from_config() and thrown warnin…
namrata270998 Jul 18, 2022
ff6367c
resolved PR comments and fixed pylint
namrata270998 Jul 18, 2022
a33b6b2
Added __exit__ method in the Client object.
prijendev Jul 19, 2022
1e68ea5
Added new line in the end of client.py
prijendev Jul 19, 2022
9de3174
Updated schema for pr_commits.
prijendev Jul 19, 2022
21bab80
resolved PR comments
namrata270998 Jul 20, 2022
a4f1207
Give warning message for invalid repo.
prijendev Jul 20, 2022
6fe16b4
Added comments in the code.
prijendev Jul 20, 2022
fae0288
resolved PR comments
namrata270998 Jul 21, 2022
a199442
resolved review comments
NevilParikh14 Jul 22, 2022
61d0c71
Raised error for invalid organization.
prijendev Jul 22, 2022
7b270ef
Merge branch 'TDl-19530-update-dict-based-implementation-to-class-bas…
namrata270998 Jul 22, 2022
ca8fdae
Updated logger message in client.py
prijendev Jul 22, 2022
80d0080
Merge remote-tracking branch 'origin/TDL-19527-update-backoff-error-h…
prijendev Jul 22, 2022
e2c1840
Updated unittest case to use parameterized.
prijendev Jul 25, 2022
b18f699
added a new TooManyRequests class error for 429
namrata270998 Jul 26, 2022
9fbe579
resolved review comment
NevilParikh14 Aug 1, 2022
67adc29
Merge branch 'TDL-19555-add-missing-tap-tester-tests' into TDl-19530-…
prijendev Aug 1, 2022
a024fba
update variable name
NevilParikh14 Aug 1, 2022
9e47b6c
Merge branch 'TDL-19555-add-missing-tap-tester-tests' into TDl-19530-…
prijendev Aug 1, 2022
83b1e5b
resolved review comment and updated print - LOGGER
NevilParikh14 Aug 3, 2022
b1edb4f
Merge branch 'TDL-19555-add-missing-tap-tester-tests' into TDl-19530-…
prijendev Aug 4, 2022
b8dcd14
TDL 16573 sync teams at organization level (#173)
prijendev Aug 8, 2022
2b7cbd8
TDL-5961 Support of custom domain (#172)
prijendev Aug 8, 2022
084069a
TDL-12323 Implemented currently syncing for repos and streams (#171)
namrata270998 Aug 8, 2022
8620298
Updated verify_access_for_repo method in client.
prijendev Aug 8, 2022
b0c9d6b
Resolved pylint.
prijendev Aug 8, 2022
affbfc0
Updated currently syncing for teams streams. (#174)
prijendev Aug 9, 2022
35f65dc
backmerged main branch
prijendev Aug 10, 2022
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
2 changes: 1 addition & 1 deletion .circleci/config.yml
Original file line number Diff line number Diff line change
Expand Up @@ -29,7 +29,7 @@ jobs:
name: 'Unit Tests'
command: |
source /usr/local/share/virtualenvs/tap-github/bin/activate
pip install nose coverage
pip install nose coverage parameterized
nosetests --with-coverage --cover-erase --cover-package=tap_github --cover-html-dir=htmlcov tests/unittests
coverage html
when: always
Expand Down
15 changes: 9 additions & 6 deletions README.md
Original file line number Diff line number Diff line change
Expand Up @@ -53,15 +53,18 @@ This tap:
3. Create the config file

Create a JSON file containing the start date, access token you just created
and the path to one or multiple repositories that you want to extract data from. Each repo path should be space delimited. The repo path is relative to
`https://github.com/`. For example the path for this repository is
and the path to one or multiple repositories that you want to extract data from. Each repo path should be space delimited. The repo path is relative to `"base_url"`
(Default: `https://github.com/`). For example the path for this repository is
`singer-io/tap-github`. You can also add request timeout to set the timeout for requests which is an optional parameter with default value of 300 seconds.

```json
{"access_token": "your-access-token",
"repository": "singer-io/tap-github singer-io/getting-started",
"start_date": "2021-01-01T00:00:00Z",
"request_timeout": 300}
{
"access_token": "your-access-token",
"repository": "singer-io/tap-github singer-io/getting-started",
"start_date": "2021-01-01T00:00:00Z",
"request_timeout": 300,
"base_url": "https://api.github.com"
}
```
4. Run the tap in discovery mode to get properties.json file

Expand Down
3 changes: 2 additions & 1 deletion config.sample.json
Original file line number Diff line number Diff line change
Expand Up @@ -2,5 +2,6 @@
"access_token": "abcdefghijklmnopqrstuvwxyz1234567890ABCD",
"repository": "singer-io/target-stitch",
"start_date": "2021-01-01T00:00:00Z",
"request_timeout": 300
"request_timeout": 300,
"base_url": "https://api.github.com"
}
Loading