@@ -32,12 +32,54 @@ memory format remains stable since the 0.3.x release.
3232See the [ Install Page] [ 2 ] to learn how to get the libraries for your
3333platform. The [ complete changelog] [ 5 ] is also available.
3434
35- ## Plasma Object Store
35+ ## Plasma Shared Memory Object Store
3636
37+ This release includes the [ Plasma Store] [ 7 ] , which you can read more about in
38+ the linked blog post. This system was originally developed as part of the [ Ray
39+ Project] [ 8 ] at the [ UC Berkeley RISELab] [ 9 ] . We recognized that Plasma would be
40+ highly valuable to the Arrow community as a tool for shared memory management
41+ and zero-copy deserialization. Additionally, we believe we will be able to
42+ develop a stronger software stack through sharing of IO and buffer management
43+ code.
3744
45+ The Plasma store is a server application which runs as a separate process. A
46+ reference C++ client, with Python bindings, is made available in this
47+ release. Clients can be developed in Java or other languages in the future to
48+ enable simple sharing of complex datasets through shared memory.
3849
39- ## Arrow Format Additions
50+ ## Arrow Format Addition: Map type
4051
52+ We added a Map logical type to represent ordered and unordered maps
53+ in-memory. This corresponds to the ` MAP ` logical type annotation in the Parquet
54+ format (where maps are represented as repeated structs).
55+
56+ Map is represented as a list of structs. It is the first example of a logical
57+ type whose physical representation is a nested type. We have not yet created
58+ implementations of Map containers in any of the implementations, but this can
59+ be done in a future release.
60+
61+ As an example, the Python data:
62+
63+ ```
64+ data = [{'a': 1, 'bb': 2, 'cc': 3}, {'dddd': 4}]
65+ ```
66+
67+ Could be represented in an Arrow ` Map<String, Int32> ` as:
68+
69+ ```
70+ Map<String, Int32> = List<Struct<keys: String, values: Int32>>
71+ is_valid: [true, true]
72+ offsets: [0, 3, 4]
73+ values: Struct<keys: String, values: Int32>
74+ children:
75+ - keys: String
76+ is_valid: [true, true, true, true]
77+ offsets: [0, 1, 3, 5, 9]
78+ data: abbccdddd
79+ - values: Int32
80+ is_valid: [true, true, true, true]
81+ data: [1, 2, 3, 4]
82+ ```
4183## Python Changes
4284
4385Some highlights of Python development outside of bug fixes and general API
@@ -55,10 +97,6 @@ improvements include:
5597
5698## Toward Arrow 1.0.0 and Beyond
5799
58- We have much work ahead of us to build out Arrow integrations in other data
59- systems to improve their processing performance and interoperability with other
60- systems.
61-
62100We are still discussing the roadmap to 1.0.0 release on the [ developer mailing
63101list] [ 6 ] . The focus of the 1.0.0 release will likely be memory format stability
64102and hardening integration tests across the remaining data types implemented in
@@ -69,3 +107,6 @@ Java and C++. Please join the discussion there.
69107[ 3 ] : http://github.com/apache/parquet-cpp
70108[ 5 ] : http://arrow.apache.org/release/0.6.0.html
71109[ 6 ] : http://mail-archives.apache.org/mod_mbox/arrow-dev/
110+ [ 7 ] : http://arrow.apache.org/blog/2017/08/08/plasma-in-memory-object-store/
111+ [ 8 ] : https://ray-project.github.io/ray/
112+ [ 9 ] : https://rise.cs.berkeley.edu/
0 commit comments