Spec Formalization: Extensions Approach

[jacques-n]

The current extensions concepts are roughly defined as follows:

• Multiple extension types (user defined types, type variations, scalar functions, aggregate functions, window functions, etc). This list is likely to expand.
• Extensions belong to an organization. Currently conceived as a global list of organizations maintained in substrait with pointers to a publically available toc defined as a github url. A portion of organization ids are considered private extension ids that can be used for internal/private resources and implementations.
• Each organization manages a TOC, which then lists out 0 or more locations for each extension type to get the definition of the extension.
• Documents are all encoded is relatively simple yaml documents (with schemas that can/should be formalized).

The model is trying to achieve two things:

• Make extensions as independent as possible: once someone registers an organization, the specifics of what they house in that organization are entirely independent of any central control.
• Maintain a root list of organizations so that people can find things that are being used and collaborate rather than always reinvent.

As I think through the initial sketch, one of the things that I dislike about it is that really isn't a good reason to have a numeric id associated with each organization. This was done for encoding reasons but the reality is that we can come up with a rudimentary plan level scheme to achieve the same thing without having to keep a list of canonical numbers.

A couple key questions asked by others include:

• Do we need a central list? Why add that overhead? If we want to have a list of commonly used things, we can maintain that but it shouldn't be used as a blocker for extension (I have to request to be registered before I can begin extending things).
• Can we do harder coupling of git uris so that organizations, list types, etc. Basically are well-known. In general moving more towards further towards convention over configuration. For example, one would simply source github.com/package-url/packageurl-substrait and then all of the assets (functions, types, etc) would be available for plan inclusion.

I'll add breakouts for these and some other similar questions.

[jacques-n]

Key Question 1: Should we have a central list that is part of programmatic extension discovery?

PROS: easier to find what is available, may reduce duplication of efforts
CONS: less organic/heavier process, may reduce people's likelihood to use substrait

[jacques-n]

@carlsverre would love your feedback on this specifically and the extension topic overall.

[westonpace]

It might help to think through some of the motivations for people developing extensions. Just brainstorming:

• Query engine implements support for a type: For example, someone might create a GeoJSON analytics database that can perform operations like DISTANCE(src, dest) => double. They want to make it easy for someone to use this database.
• Same as above but it isn't a query engine but a team working on GeoJSON and they implement the entire thing as extensions. First, extensions to an existing query engine (e.g. custom kernels for Arrow C++ compute) and then also as extensions for substrait.

So I don't know how much extensions need to be "discoverable" as much as engines need a way to broadcast what extensions they have currently installed / support?

[jacques-n]

Key Question 2: What do people think about using something more similar to Go packages where git urls are extension "packages"?

[jacques-n]

Key Question 3: Does it make sense to have a pointer structure e.g. the toc in the current sketch or should we just define well-known file names/paths for each object type? Related: does yaml make sense as a way to express the different extension concepts?

[westonpace]

One potential source of value for substrait extensions is if multiple implementations or systems can agree on a common extension definition. For example, substraits core definition will hopefully be a place for people to agree on common behavior of data types and eliminate quirks. How do extension type authors go about this process.

If a group of interested systems get together and want to define a common definition for complex types and arithmetic do they do this completely outside of substrait and simply list their package with substrait?

Or does substrait serve as a place to vote on "bless" official extensions for certain types?

[jacques-n]

I'm much more inclined to the former (no centralized blessings). For example, the foo and bar communities decide they want to collaborate via substrait plans with a shared set of custom extensions. They define the appropriate extensions in a shared place and both use them. Anyone else in the community should be able to use them/leverage them but it shouldn't require the extensions to actually live in substrait proper. At least that is the way my thinking is trending.