New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

MINOR: Dynamodb sample data #15264

Merged

sushi30 merged 33 commits into main from dynamodb-sample-data

Apr 22, 2024

Contributor

sushi30 commented Feb 20, 2024

Describe your changes:

Add support for DynamoDB sample data.

Builds on:

Type of change:

Improvement

Checklist:

I have read the CONTRIBUTING document.
My PR title is Fixes <issue-number>: <short explanation>
I have commented on my code, particularly in hard-to-understand areas.
For JSON Schema changes: I updated the migration scripts or explained why it is not needed.

I have added tests around the new logic.
For connector/ingestion changes: I updated the documentation.

sushi30 added 19 commits

February 15, 2024 11:37


          feat(nosql-profiler): row count

ee63fc9

1. Implemented the NoSQLProfilerInterface as an entrypoint for the nosql profiler.
2. Added the NoSQLMetric as an abstract class.
3. Implemented the interface for the MongoDB database source.
4. Implemented an e2e test using testcontainers.


          added profiler support for mongodb connection

doc


          Merge remote-tracking branch 'origin/main' into nosql-row-count

91e103e


          use int_admin_ometa in test setup

74c2abd


          - fixed linting issue in gx

ba8e872

- removed unused inheritance


          moved the nosql function into the metric class

d261553


          feat(profiler): add dynamodb row count

53d5210


          feat(profiler): add dynamodb row count

7b15ab8


          formatting

ffca0e6


          validate_compose: raise exception for bad status code.

215ac9c


          fixed import

1733b9a


          format

e4d794d


          feat(nosql-profiler): added sample data

3677d57

1. Implemented the NoSQL sampler.
2. Some naming changes to the NoSQL adaptor to avoid fixing names with the profiler interface.
3. Tests.


          added default sample limit

2a28fad


          formatting

9ad7213


          fixed import

6d06d3e


          Merge branch 'nosql-profiler-sample-data' into dynamodb-sample-data

8af9676

# Conflicts:
#	ingestion/src/metadata/profiler/metrics/core.py


          feat(profiler): dynamodb sample data

0f3770c

github-actions bot added UI Ingestion backend devops labels

sushi30 added cleanup and removed UI Ingestion backend devops labels

github-actions bot added Ingestion safe to test labels


          tests for dynamo db sample data

sushi30 marked this pull request as ready for review

April 18, 2024 12:38

sushi30 requested a review from a team as a code owner

April 18, 2024 12:38

github-actions bot added the backend label

Contributor

github-actions bot commented Apr 18, 2024

The Python checkstyle failed.

Please run make py_format and py_format_check in the root of your repository and commit the changes to this PR.
You can also use pre-commit to automate the Python code formatting.

You can install the pre-commit hooks with make install_test precommit_install.


          format

a2e2e76

Contributor

github-actions bot commented Apr 18, 2024

The Python checkstyle failed.

Please run make py_format and py_format_check in the root of your repository and commit the changes to this PR.
You can also use pre-commit to automate the Python code formatting.

You can install the pre-commit hooks with make install_test precommit_install.


          format

23b3f30

Contributor

github-actions bot commented Apr 18, 2024

The Python checkstyle failed.

Please run make py_format and py_format_check in the root of your repository and commit the changes to this PR.
You can also use pre-commit to automate the Python code formatting.

You can install the pre-commit hooks with make install_test precommit_install.

IceS2 previously approved these changes

View reviewed changes

ingestion/src/metadata/profiler/adaptors/factory.py Outdated Show resolved Hide resolved


          use service connection for nosql adaptor factory

bb56abd

sushi30 dismissed IceS2’s stale review via

bb56abd

April 19, 2024 10:31

Contributor

github-actions bot commented Apr 19, 2024

The Python checkstyle failed.

Please run make py_format and py_format_check in the root of your repository and commit the changes to this PR.
You can also use pre-commit to automate the Python code formatting.

You can install the pre-commit hooks with make install_test precommit_install.


          fixed tests

41e8e90

Contributor

github-actions bot commented Apr 19, 2024

The Python checkstyle failed.

Please run make py_format and py_format_check in the root of your repository and commit the changes to this PR.
You can also use pre-commit to automate the Python code formatting.

You can install the pre-commit hooks with make install_test precommit_install.

sushi30 added 5 commits

April 19, 2024 15:34


          format

d398f77


          Merge remote-tracking branch 'refs/remotes/origin/main' into dynamodb…

06ee6d8

…-sample-data


          Merge branch 'main' into dynamodb-sample-data

937b415


          Merge remote-tracking branch 'refs/remotes/origin/main' into dynamodb…

03cde70

…-sample-data

# Conflicts:
#	ingestion/src/metadata/profiler/adaptors/dynamodb.py


          fixed after merge

8d33fa2

sushi30 requested a review from IceS2

April 22, 2024 08:20

sushi30 enabled auto-merge (squash)

April 22, 2024 08:23

sushi30 added 3 commits

April 22, 2024 13:23


          Merge branch 'main' into dynamodb-sample-data

97c8616


          set boto3 credentials for localstack

4bc764c


          set dynamodb credentials for localstack

0303ceb

sonarqubecloud bot commented Apr 22, 2024

Quality Gate passed for 'open-metadata-ingestion'

Issues
0 New issues
0 Accepted issues

Measures
0 Security Hotspots
65.0% Coverage on New Code
0.0% Duplication on New Code

See analysis details on SonarCloud

Contributor Author

sushi30 commented Apr 22, 2024

@IceS2 please re-approve after merging and fixing tests

IceS2 approved these changes

View reviewed changes

sushi30 merged commit 93ec391 into main

11 checks passed

sushi30 deleted the dynamodb-sample-data branch

April 22, 2024 15:46

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

backend cleanup Ingestion safe to test