JDK 16 Vector API

[paladin8]

JDK 16 introduces the Vector API for SIMD parallelism directly from Java code: https://docs.oracle.com/en/java/javase/16/docs/api/jdk.incubator.vector/jdk/incubator/vector/package-summary.html

My understanding is that since the original FastPFOR library takes advantage of SIMD, it should be possible to make this one as well. Is that correct?

[lemire]

That’s correct. Many schemes should benefit.

[Fang-Xie]

@lemire, We have implemented bit unpacking with JDK 17/18 Vector API, below are the test results compared with Parquet-mr bit pack implementation shared with you

After integrating with parquet bit pack decoding, and tested from Spark VectorizedParquetRecordReader, we could see around 2x performance gain when reading the parquet file

[Fang-Xie]

If we could submit our code to this community?

[lemire]

Yes. Please prepare a pull request if you can.

[Fang-Xie]

Thanks @lemire, For this project, we are under the internal procedure of open sourcing, once it's done, we will prepare a pull request.

[lemire]

@Fang-Xie I am looking forward to reading your code.