Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Serialization protocol in C++ #136

Closed
jpivarski opened this issue Mar 4, 2020 · 2 comments
Closed

Serialization protocol in C++ #136

jpivarski opened this issue Mar 4, 2020 · 2 comments
Assignees
Labels
feature New feature or request

Comments

@jpivarski
Copy link
Member

Since #68 is being scaled back to a Python function, C++ will need some serialization layer. awkward0.persist should be ported to C++.

Again, this is not for performance, but for accessibility.

@jpivarski jpivarski added the feature New feature or request label Mar 4, 2020
@jpivarski jpivarski self-assigned this Mar 13, 2020
@jpivarski
Copy link
Member Author

Similar to the old one, but it doesn't have to be exactly the same. It really should be in C++, and that already limits the possibility of compatibility.

Also, Awkward1 has a different set of node types than Awkward0.

@jpivarski
Copy link
Member Author

PR #348 added a serialization protocol in Python. There are two new functions, ak.to_arrayset and ak.from_arrayset, which turn an Awkward Array into a schema and a collection of flat arrays (an "arrayset"):

>>> original = ak.Array([[1, 2, 3], [], [4, 5]])
>>> form, container, num_partitions = ak.to_arrayset(original)
>>> form
{
    "class": "ListOffsetArray64",
    "offsets": "i64",
    "content": {
        "class": "NumpyArray",
        "itemsize": 8,
        "format": "l",
        "primitive": "int64",
        "form_key": "node1"
    },
    "form_key": "node0"
}
>>> container
{'node0-offsets': array([0, 3, 3, 5], dtype=int64),
 'node1': array([1, 2, 3, 4, 5])}
>>> print(num_partitions)
None

Unless there's a strong call for a C++ version, the serialization protocol in Python will be considered good enough.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
feature New feature or request
Projects
None yet
Development

No branches or pull requests

1 participant