Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

MINOR: fix snowflake system metrics #17989

Merged
merged 7 commits into from
Sep 26, 2024
Merged
Show file tree
Hide file tree
Changes from 2 commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
Original file line number Diff line number Diff line change
Expand Up @@ -25,6 +25,7 @@
SNOWFLAKE_QUERY_LOG_QUERY,
)
from metadata.profiler.metrics.system.dml_operation import DatabaseDMLOperations
from metadata.utils.dict import CustomDict
from metadata.utils.logger import ingestion_logger
from metadata.utils.profiler_utils import QueryResult

Expand Down Expand Up @@ -133,7 +134,7 @@ def get_for_table(session: Session, tablename: str):
)
)
return TypeAdapter(List[SnowflakeQueryLogEntry]).validate_python(
map(dict, rows)
[CustomDict(r).lower_case_keys() for r in rows]
)


Expand Down
3 changes: 3 additions & 0 deletions ingestion/src/metadata/utils/dict.py
Original file line number Diff line number Diff line change
@@ -0,0 +1,3 @@
class CustomDict(dict):
def lower_case_keys(self):
return {k.lower(): v for k, v in self.items()}
Copy link
Contributor

@TeddyCr TeddyCr Sep 25, 2024

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I don't think we need to make that an object right? I think it is a small overhead we can remove -- i.e. the object creation.

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Can you clarify? what needs to be an object?

Original file line number Diff line number Diff line change
Expand Up @@ -17,8 +17,8 @@
No sample data is required beforehand
"""
import pytest

from ingestion.tests.integration.datalake.conftest import BUCKET_NAME

from metadata.generated.schema.entity.data.table import ColumnProfile, Table
from metadata.utils.time_utils import (
get_beginning_of_day_timestamp_mill,
Expand Down
2 changes: 1 addition & 1 deletion ingestion/tests/integration/datalake/test_ingestion.py
Original file line number Diff line number Diff line change
Expand Up @@ -12,8 +12,8 @@
"""Datalake ingestion integration tests"""

import pytest

from ingestion.tests.integration.datalake.conftest import BUCKET_NAME

from metadata.generated.schema.entity.data.table import DataType, Table
from metadata.ingestion.ometa.models import EntityList
from metadata.ingestion.ometa.ometa_api import OpenMetadata
Expand Down
Loading