[WIP][DO-NOT-MERGE] Reference PR for flatMapGroupsWithState in PySpark#37863

Closed

HeartSaVioR wants to merge 44 commits intoapache:masterfrom

HeartSaVioR:WIP-applyinpandaswithstate-jungtaek-pipeline-binpack

Commits on Sep 14, 2022

Basework
HeartSaVioR
committed
Add Python implementation

HyukjinKwon
authored and
HeartSaVioR
committed
Reorder key attributes from deduplicated data attributes

HyukjinKwon
authored and
HeartSaVioR
committed
Apply suggestions from code review

HyukjinKwon
authored and
HeartSaVioR
committed
Refactoring a bit to respect the column order

HyukjinKwon
authored and
HeartSaVioR
committed
WIP Changes to execute in pipelined manner
HeartSaVioR
committed
WIP further optimization
HeartSaVioR
committed
WIP comments for more tunes
HeartSaVioR
committed
WIP further tune...
HeartSaVioR
committed
WIP done more tune! didn't do any of pandas/arrow side tunes
HeartSaVioR
committed
WIP avoid adding additional empty row for state, empty row will be added only when there is no data
HeartSaVioR
committed
WIP remove debug log
HeartSaVioR
committed
WIP hack around to see the possibility of perf gain on binpacking
HeartSaVioR
committed
WIP proper work to apply binpacking on python worker -> executor
HeartSaVioR
committed
WIP fix silly bug
HeartSaVioR
committed
WIP another silly bugfix on migration
HeartSaVioR
committed
WIP apply binpacking for executor -> python worker as well
HeartSaVioR
committed
WIP fix silly bug
HeartSaVioR
committed
WIP fix another silly bug
HeartSaVioR
committed
WIP batching per specified size, with sampling
HeartSaVioR
committed
WIP introduce DBR-only change
HeartSaVioR
committed
WIP debugging now...
HeartSaVioR
committed
WIP still debugging... weirdness happened
HeartSaVioR
committed
WIP small fix
HeartSaVioR
committed
WIP fix a serious bug... make sure all columns in Arrow RecordBatch have same number of elements
HeartSaVioR
committed
WIP strengthen test
HeartSaVioR
committed
WIP documenting the changes for pipelining and bin-packing... not yet done
HeartSaVioR
committed
WIP sync
HeartSaVioR
committed
WIP start with is_last_chunk since it's easier to implement... several tests are failing yet (and the existing test code is hard to make it work with multiple calls)
HeartSaVioR
committed
WIP adjust the test code to make test pass with multiple calls
HeartSaVioR
committed
WIP refactor a bit... just extract the abstract classes to explicit ones
HeartSaVioR
committed
WIP iterator of DatFrame done! updated tests and they all passed
HeartSaVioR
committed
WIP FIX pyspark side test failure
HeartSaVioR
committed
WIP sort out codebase a bit
HeartSaVioR
committed
WIP no batch query support in applyInPandasWithState
HeartSaVioR
committed
WIP address some missed things
HeartSaVioR
committed
WIP remove comments which are obsolete or won't be addressed
HeartSaVioR
committed
WIP change the return type of user function to Iterator[DataFrame]
HeartSaVioR
committed
WIP remove unnecessary interface/implementation changes on GroupState as it bothers MiMa
HeartSaVioR
committed
WIP refine out some code
HeartSaVioR
committed
WIP fix scalastyle
HeartSaVioR
committed
WIP remove obsolete class
HeartSaVioR
committed
WIP remove the temp fix
HeartSaVioR
committed
remove unused code
HeartSaVioR
committed