Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

fix JSON bug with data reading #691

Merged
merged 5 commits into from
Oct 17, 2022
Merged

fix JSON bug with data reading #691

merged 5 commits into from
Oct 17, 2022

Conversation

JGSweets
Copy link
Contributor

@JGSweets JGSweets commented Oct 14, 2022 •
edited
Loading

Previously: we could not read line separated JSON arrays.
Now we can read:

[1, 2]
[2, 3]
[3, 3]

becomes a JSONData which is wrapping pandas as:

   0   1
0  1   2
1  2   3
2  3   3

@JGSweets JGSweets added Bug Something isn't working High Priority Dramatic improvement, inaccurate calculation(s) or bug / feature making the library unusable labels Oct 14, 2022
JGSweets
JGSweets commented Oct 14, 2022
View reviewed changes
dataprofiler/data_readers/avro_data.py
@@ -1,6 +1,6 @@
"""Contains class for saving and loading spreadsheet data."""
from io import BytesIO, StringIO
from typing import Any, Dict, List, Optional, Union, cast
from typing import Any, Dict, List, Optional, Union
Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

the proper way in the other func for a boolean which checks type is to return a TypeGuard

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

no need to cast now

JGSweets
JGSweets commented Oct 14, 2022
View reviewed changes
dataprofiler/data_readers/avro_data.py
@@ -92,18 +92,12 @@ def is_match(

# get current position of stream
if data_utils.is_stream_buffer(file_path):
file_path = cast(
Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

static typing, not functional change, removed bc of TypeGuard

JGSweets
JGSweets commented Oct 14, 2022
View reviewed changes
dataprofiler/data_readers/avro_data.py
starting_location = file_path.tell()

is_valid_avro = fastavro.is_avro(file_path)

# return to original position in stream
if data_utils.is_stream_buffer(file_path):
file_path = cast(
Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

static typing, not functional change, removed bc of TypeGuard

JGSweets
JGSweets commented Oct 14, 2022
View reviewed changes
dataprofiler/data_readers/filepath_or_buffer.py

def is_stream_buffer(filepath_or_buffer: Any) -> bool:

def is_stream_buffer(filepath_or_buffer: Any) -> TypeGuard[Union[StringIO, BytesIO]]:
Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

fix to use TypeGuard

JGSweets
JGSweets commented Oct 14, 2022
View reviewed changes
dataprofiler/data_readers/filepath_or_buffer.py
Comment on lines -30 to +35
open_method: str ="r",
encoding: Optional[str]=None,
seek_offset: Optional[int]=None,
seek_whence: int=0,
open_method: str = "r",
encoding: Optional[str] = None,
seek_offset: Optional[int] = None,
seek_whence: int = 0,
Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

formatting

JGSweets
JGSweets commented Oct 14, 2022
View reviewed changes
dataprofiler/data_readers/filepath_or_buffer.py
Comment on lines -54 to +58
self.original_type: Union[Type[str], Type[StringIO], Type[BytesIO], Type[IO]] = type(filepath_or_buffer)
self.original_type: Union[
Type[str], Type[StringIO], Type[BytesIO], Type[IO]
] = type(filepath_or_buffer)
Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

format

JGSweets
JGSweets commented Oct 14, 2022
View reviewed changes
dataprofiler/data_readers/filepath_or_buffer.py
Comment on lines -87 to +93
self._filepath_or_buffer = cast(TextIOWrapper, self._filepath_or_buffer) # guaranteed by self._is_wrapped
self._filepath_or_buffer = cast(
TextIOWrapper, self._filepath_or_buffer
) # guaranteed by self._is_wrapped
Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

format

JGSweets
JGSweets commented Oct 14, 2022
View reviewed changes
dataprofiler/data_readers/filepath_or_buffer.py
Comment on lines -95 to +103
self._filepath_or_buffer = cast(IO, self._filepath_or_buffer) # can't be str due to conversion in __enter__
self._filepath_or_buffer = cast(
IO, self._filepath_or_buffer
) # can't be str due to conversion in __enter__
Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

format

JGSweets
JGSweets commented Oct 14, 2022
View reviewed changes
dataprofiler/data_readers/json_data.py
_data = _data.to_dict(orient="records", into=OrderedDict)
for i, sample in enumerate(_data):
_data[i] = json.dumps(
data = self._get_data_as_df(data)
Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

no longer having two variables of the data with _data
fixes type acceptance at the top

JGSweets
JGSweets commented Oct 14, 2022
View reviewed changes
dataprofiler/data_readers/json_data.py
"""
Extract the data as a json format.

:param data: raw data
:type data: list
:return: dataframe in json format
"""
_data: Union[pd.DataFrame, List]
Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

no longer having two variables of the data with _data
fixes type acceptance at the top

JGSweets
JGSweets commented Oct 14, 2022
View reviewed changes
dataprofiler/data_readers/json_data.py
@@ -349,6 +349,8 @@ def _convert_flat_to_nested_cols(cls, dic: Dict, separator: str = ".") -> Dict:
:return:
"""
for key in list(dic.keys()):
if not isinstance(key, str):
Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

fix for allowing the [1] format in json reading

JGSweets
JGSweets commented Oct 14, 2022
View reviewed changes
dataprofiler/data_readers/json_data.py
@@ -392,14 +394,16 @@ def is_match(
return True
except (json.JSONDecodeError, UnicodeDecodeError):
data_file.seek(0)

json_identifier_re = re.compile(r"(:|\[)")
Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

allows both : and [ as a JSON identifier differentiating from just a string.

JGSweets
JGSweets commented Oct 14, 2022
View reviewed changes
dataprofiler/tests/data_readers/test_json_data.py
@@ -12,11 +12,12 @@

class TestNestedJSON(unittest.TestCase):
def test_flat_to_nested_json(self):
dic = {"a.b": "ab", "a.c": "ac", "a.d.f": "adf", "b": "b"}
dic = {"a.b": "ab", "a.c": "ac", "a.d.f": "adf", "b": "b", 1: 3}
Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

updates test to check for keys that aren't strings

JGSweets
JGSweets commented Oct 14, 2022
View reviewed changes
dataprofiler/tests/data_readers/test_json_data.py
@@ -56,6 +57,11 @@ def setUpClass(cls):
encoding="utf-8",
count=14,
),
dict(
Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

adds test which includes the new json format

JGSweets
JGSweets commented Oct 14, 2022
View reviewed changes
dataprofiler/tests/data/json/simple-list.json
@@ -0,0 +1,3 @@
[1]
Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

new data for testing json reading

@taylorfturner taylorfturner enabled auto-merge (squash) October 17, 2022 12:39
@taylorfturner taylorfturner merged commit d16b5c8 into capitalone:main Oct 17, 2022
@taylorfturner taylorfturner mentioned this pull request Oct 17, 2022
Merged
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Reviewers

@taylorfturner taylorfturner taylorfturner approved these changes

@ksneab7 ksneab7 ksneab7 approved these changes

@micdavis micdavis Awaiting requested review from micdavis

@tyfarnan tyfarnan Awaiting requested review from tyfarnan

Assignees

@taylorfturner taylorfturner

Labels
Bug Something isn't working High Priority Dramatic improvement, inaccurate calculation(s) or bug / feature making the library unusable
Projects
None yet
Milestone
No milestone
Development

Successfully merging this pull request may close these issues.
3 participants
@JGSweets @taylorfturner @ksneab7