Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

make JSON files smaller by limiting the precision #499

Merged
merged 8 commits into from
Oct 25, 2021

Conversation

jgriesfeller
Copy link
Member

@jgriesfeller jgriesfeller commented Oct 19, 2021

attempt to fix #366

@jgriesfeller jgriesfeller self-assigned this Oct 19, 2021
@codecov
Copy link

codecov bot commented Oct 19, 2021

Codecov Report

Merging #499 (2272172) into main-dev (25aea64) will increase coverage by 0.01%.
The diff coverage is 100.00%.

❗ Current head 2272172 differs from pull request most recent head 5ee4b3e. Consider uploading reports for the commit 5ee4b3e to get more accurate results
Impacted file tree graph

@@             Coverage Diff              @@
##           main-dev     #499      +/-   ##
============================================
+ Coverage     76.64%   76.65%   +0.01%     
============================================
  Files            97       97              
  Lines         17357    17365       +8     
============================================
+ Hits          13304    13312       +8     
  Misses         4053     4053              
Flag Coverage Δ
unittests 76.65% <100.00%> (+0.01%) ⬆️

Flags with carried forward coverage won't be shown. Click here to find out more.

Impacted Files Coverage Δ
pyaerocom/_lowlevel_helpers.py 82.26% <100.00%> (+0.37%) ⬆️

Continue to review full report at Codecov.

Legend - Click here to learn more
Δ = absolute <relative> (impact), ø = not affected, ? = missing data
Powered by Codecov. Last update 25aea64...5ee4b3e. Read the comment docs.

@jgriesfeller jgriesfeller marked this pull request as ready for review October 22, 2021 07:19
@jgriesfeller jgriesfeller requested a review from avaldebe October 22, 2021 07:20
Copy link
Collaborator

@avaldebe avaldebe left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

please conside the following changes

  • use np.round instead of round
  • use comprehensions instead if create/modify loops
  • add a test tat covers this new test that asserts the round_floats functionality
    assert round_floats(16.055, 2) == 16.055
    assert round_floats([16.055], 2) == [16.055]
    assert round_floats({"value":16.055}, 2) == {"value":16.055}

pyaerocom/_lowlevel_helpers.py Outdated Show resolved Hide resolved
pyaerocom/_lowlevel_helpers.py Outdated Show resolved Hide resolved
pyaerocom/_lowlevel_helpers.py Outdated Show resolved Hide resolved
pyaerocom/_lowlevel_helpers.py Outdated Show resolved Hide resolved
pyaerocom/_lowlevel_helpers.py Outdated Show resolved Hide resolved
pyaerocom/_lowlevel_helpers.py Outdated Show resolved Hide resolved
pyaerocom/_lowlevel_helpers.py Outdated Show resolved Hide resolved
pyaerocom/_lowlevel_helpers.py Outdated Show resolved Hide resolved
pyaerocom/_lowlevel_helpers.py Outdated Show resolved Hide resolved
Copy link
Collaborator

@avaldebe avaldebe left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This is what I mean regarding the usage of comprehensions instead of create/modify loops

    if isinstance(in_data, (float, np.float)):
        return np.round(in_data, precision)
    if isinstance(in_data, (list, tuple)):
        return [round_floats(v, precision=precision) for v in in_data]
    if isinstance(in_data, dict):
        return {k:round_floats(v, precision=precision) for k, v in in_data.items()}
    return in_data # or raise a ValueError

I never meant to say that the if isinstance logic should be folded into the comprehension

Copy link
Collaborator

@avaldebe avaldebe left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thanks for the changes.
Could you also ads some tests that assert on the desired functionality.

assert round_floats(16.055, 2) == 16.06
assert round_floats([16.055], 2) == [16.06]
assert round_floats({"value":16.055}, 2) == {"value":16.06}

Copy link
Collaborator

@avaldebe avaldebe left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

👍 ship it

@jgriesfeller
Copy link
Member Author

Will merge as soon as CI is finished

@jgriesfeller jgriesfeller added this to the v0.12.2 milestone Oct 25, 2021
@jgriesfeller jgriesfeller merged commit 5d9db3a into main-dev Oct 25, 2021
@jgriesfeller jgriesfeller deleted the griesie_fix_366 branch October 25, 2021 13:14
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

Reduce all json files size
2 participants