Reduce all json files size

[AugustinMortier]

Te json files currenthly have 17 decimal numbers. By reducing it down to 4-5?, we could save a lot of space. This can be applied to all of the json files: time series, glob_stats, map, and scat. Only the contour files already have a limited decimal number.

e.g

"mnmb": 0.10131313755341606

to be changed to

"mnmb": 0.1013

[jgliss]

Good point @AugustinMortier. We should do that together with the upcoming AeroVal updates for v0.12.0.

[jgliss]

Would be good to look into this soon, perhaps @dulte, if you have time. Will not go into v0.12.0 though.

[jgriesfeller]

I tumbled over this and looked a bit around:

It seems that all json writing is based on the method

pyaerocom/pyaerocom/_lowlevel_helpers.py

Line 29 in 25aea64

def write_json(data_dict, file_path, **kwargs):

which just calls simplejson.dump

According to my findings one has to round the floats to a given precision to get these into the json file.

like (from above):

def round_floats(o):
    if isinstance(o, float):
        return round(o, 5)
    if isinstance(o, dict):
        return {k: round_floats(v) for k, v in o.items()}
    if isinstance(o, (list, tuple)):
        return [round_floats(x) for x in o]
    return o

data = json.dumps(round_floats(data))

[avaldebe]

I would lean into np.round and resolve the dict recurrence on the list clause

import numpy as np
import simplejson as json

def round_floats(o, *, decimals: int = 5):
    if isinstance(o, float):
        return np.round(o, decimals)
    if isinstance(o, (list, tuple)):
        return np.array(o).round(decimals).tolist()
    if isinstance(o, dict):
        return dict(zip(o, round_floats(o.values(), decimals=decimals))
    return o

data = json.dumps(round_floats(data))

And maybe subclass the simplejson.JSONEncoder to handle the float encoding there instead calling json.dumps(round_floats(data))

[avaldebe]

    if isinstance(o, (list, tuple)):
        return np.array(o).round(decimals).tolist()

I guess that this would only work for a list of floats...

[jgriesfeller]

The code was just a snippet I found and not meant to be used as it is. My real code will be in the linked PR

[jgriesfeller]

just another comment on this since I am testing the PR.
The precision on the geojson files is already limited to 5 digits after the decimal point, so this PR will not help with getting the geojson files smaller. But it does help with the normal json files.
Unfortunately the Aeronet processing changed on version 0.12.1 so it's not straight forward to compare different results.