Add a vector engine #4925
Merged
Add a vector engine #4925
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
The dag.Vectorize node tells the compiler to execute its body using the vector engine.
|
Tests in runtime/vcache/ztests/ are broken because "zed dev vcache copy" no longer works. I'd like to fix it rather than disable the tests. |
We're going to fix "zed dev vcache copy" (and "zed dev vcache project", which is also broken) later, so I've disabled the corresponding tests. |
nwt
added a commit
that referenced
this pull request
Dec 20, 2023
Do that by plugging some holes left by #4925. * In runtime/vcache/ztests/, re-enable skipped "zed dev vcache copy" tests. * In cmd/zed/dev/vcache/copy.Command.Run, resurrect the call to zio.Copy. * In package runtime/vcache: * Resurrect the Reader type and update its Read method. * Resurrect Object.NewReader. * Add loader type to hold the *zed.Context needed when loading a vector of type values and convert existing loadXXX functions to loader methods. * In loadArray, fix bad nil argument to loadVector. * In loadMap, load lengths vector. * In loadNulls, create slots slice and load values vector. * In loadPrimitive, remove null handling (vector.Nulls now does that) and add support for missing Zed types (float16, float32, float64, bytes, ip, net, and type). * In loadRecord, load all fields when no specific path is requested. * In loadUnion, load tags vector. * In loadVector, assign to *any when loading a constant vector. * In package vector: * Add missing Byte, Float, IP, Net, and Type types. * Add missing length field to Map type. * Add missing tags field to Union type. * Implement NewBuilder method for Array, Map, and Union types. * Handle nulls for all vector types by replacing the Nullmask type with Nulls, which wraps a vector.Any.
nwt
added a commit
that referenced
this pull request
Dec 20, 2023
Do that by plugging some holes left by #4925. * In runtime/vcache/ztests/, re-enable skipped "zed dev vcache copy" tests. * In cmd/zed/dev/vcache/copy.Command.Run, resurrect the call to zio.Copy. * In package runtime/vcache: * Resurrect the Reader type and update its Read method. * Resurrect Object.NewReader. * Add loader type to hold the *zed.Context needed when loading a vector of type values. * Convert existing loadXXX functions to loader methods. * In loadArray, fix bad nil argument to loadVector. * In loadMap, load lengths vector. * In loadNulls, create slots slice and load values vector. * In loadPrimitive, remove null handling (vector.Nulls now does that) and add support for missing Zed types (float16, float32, float64, bytes, ip, net, and type). * In loadRecord, load all fields when no specific path is requested. * In loadUnion, load tags vector. * In loadVector, assign to *any when loading a constant vector. * In package vector: * Add missing Byte, Float, IP, Net, and Type types. * Add missing length field to Map type. * Add missing tags field to Union type. * Implement NewBuilder method for Array, Map, and Union types. * Handle nulls for all vector types by replacing the Nullmask type with Nulls, which wraps a vector.Any.
nwt
added a commit
that referenced
this pull request
Dec 22, 2023
Do that by plugging some holes left by #4925. * In runtime/vcache/ztests/, re-enable skipped "zed dev vcache copy" tests. * In cmd/zed/dev/vcache/copy.Command.Run, resurrect the call to zio.Copy. * In package runtime/vcache: * Resurrect the Reader type and update its Read method. * Resurrect Object.NewReader. * Add loader type to hold the *zed.Context needed when loading a vector of type values. * Convert existing loadXXX functions to loader methods. * In loadArray, fix bad nil argument to loadVector. * In loadMap, load lengths vector. * In loadNulls, create slots slice and load values vector. * In loadPrimitive, remove null handling (vector.Nulls now does that) and add support for missing Zed types (float16, float32, float64, bytes, ip, net, and type). * In loadRecord, load all fields when no specific path is requested. * In loadUnion, load tags vector. * In loadVector, assign to *any when loading a constant vector. * In package vector: * Add missing Byte, Float, IP, Net, and Type types. * Add missing length field to Map type. * Add missing tags field to Union type. * Implement NewBuilder method for Array, Map, and Union types. * Handle nulls for all vector types by replacing the Nullmask type with Nulls, which wraps a vector.Any.
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Add a vector engine in compiler/vam (for vector abstract machine) with a few initial operations
Add dag.Vectorize, which tells the compiler to execute operations in its body using the vector engine
Add compiler/optimize.Optimizer.Vectorize, which identifies a very limited set of vectorizable operations and wraps them in dag.Vectorize
Add complier/kernel.Builder.compileVectorize, which compiles dag.Vectorize
Co-authored-by: @mccanne