Better handling of typed values in JSON

[jakirkham]

There has been a fair bit of discussion in issues ( #216 ), ( #156 ), and ( #244 ) about trying to store data that does not strictly have a JSON equivalent. There's a pretty good listing of interesting types in this comment specifically. Wanted to try to address this discussion at a higher level than specific failure modes observed.

Namely how do want to represent this non-JSON serializable data. Some suggestions are listed below. Others are certainly welcome.

1. Base64 encode the binary data and label it as such ( Object array fill value #216 (comment) ) ( Complex dtype error  #244 (comment) )
2. Dump the data into strings with a label indicating extraction type
3. Store the metadata in Arrays and link them
4. Allow the use of codecs on these values (e.g. users specify how to make it JSON friendly)
5. Learn from and/or use other OSS libraries in this space (e.g. jsonpickle, json-tricks, etc.)

Thoughts?

Side Note: Raised issue ( numpy/numpy#12481 ) with NumPy to ask for a (blessed) way to export/import NumPy arrays to/from JSON as that seems to be at the core of this issue. It may not be solved before we address this issue ourselves, but hopefully getting involved in that discussion early will shape what we end up with and make it easier to migrate to anything NumPy may offer in the future.

[alimanfoo]

Thanks @jakirkham. FWIW it might be worth breaking down all the different use cases as it may be that different solutions are required. Here's the ones I'm aware of:

• Storing a numpy dtype object as an attribute value (can't have numpy datatypes in attributes #156)
• Storing a numpy array as an attribute value (not sure we have a specific issue for that, but IIRC it has come up as a requirement?)
• Storing any other non-JSON-serializable object as an attribute value (related to can't have numpy datatypes in attributes #156, but not sure we have a specific issue?)
• Setting a fill value for an array with object dtype (Object array fill value #216)
• Setting a fill value for an array with complex dtype (Complex dtype error  #244)

[jakirkham]

Thanks @alimanfoo.

IIUC issue ( #156 ) was discussing storing NumPy scalars as attributes as opposed to dtype objects themselves. Is that your understanding as well or did I miss something?

As to the other use cases, they seem to always have been about storing NumPy values in JSON of various types. Whether these be NumPy scalars or perhaps NumPy arrays. Also whether they were in fill values or attributes. So that's probably the best set of cases for us to address first.