Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

GH-35344: [C++][Format] Implementation of the LIST_VIEW and LARGE_LIST_VIEW array formats #35345

Merged
merged 91 commits into from
Nov 22, 2023
Merged

GH-35344: [C++][Format] Implementation of the LIST_VIEW and LARGE_LIST_VIEW array formats #35345

merged 91 commits into from
Nov 22, 2023

Commits on Oct 26, 2023

  1. [Large]ListViewType: Add the list-view type classes and [LARGE_]LIST_… 

    …VIEW type IDs
    @felipecrv
    felipecrv committed Oct 26, 2023
    Configuration menu
    Copy the full SHA
    614ab2c View commit details
    Browse the repository at this point in the history

  2. [Large]ListViewArray: Add the list-view array classes

    @felipecrv
    felipecrv committed Oct 26, 2023
    Configuration menu
    Copy the full SHA
    03ab072 View commit details
    Browse the repository at this point in the history

  3. [Large]ListViewScalar: Add list-view scalar classes

    @felipecrv
    felipecrv committed Oct 26, 2023
    Configuration menu
    Copy the full SHA
    0e70f57 View commit details
    Browse the repository at this point in the history

  4. Parquet: [Large]ListViewArray: Add placeholder for list-view writing … 

    …into Parquet
    @felipecrv
    felipecrv committed Oct 26, 2023
    Configuration menu
    Copy the full SHA
    7cbf4f1 View commit details
    Browse the repository at this point in the history

  5. Python: [Large]ListViewArray: Disable writing list-view data into Pandas

    @felipecrv
    felipecrv committed Oct 26, 2023
    Configuration menu
    Copy the full SHA
    eb1ce8d View commit details
    Browse the repository at this point in the history

  6. visitor_generate.h: Add [Large]ListView to generated visitor code 

    And all the placeholders necessary to make all the code compile
    @felipecrv
    felipecrv committed Oct 26, 2023
    Configuration menu
    Copy the full SHA
    a97cf86 View commit details
    Browse the repository at this point in the history

  7. [Large]ListViewType: Implement type Compare + tests

    @felipecrv
    felipecrv committed Oct 26, 2023
    Configuration menu
    Copy the full SHA
    0578770 View commit details
    Browse the repository at this point in the history

  8. BaseListBuilder: Make base builder mostly compatible with ListViews

    @felipecrv
    felipecrv committed Oct 26, 2023
    Configuration menu
    Copy the full SHA
    fe96002 View commit details
    Browse the repository at this point in the history

  9. BaseVarLengthListLikeBuilder: Add a version of Append() that takes a … 

    …size as well
    @felipecrv
    felipecrv committed Oct 26, 2023
    Configuration menu
    Copy the full SHA
    ad39a9d View commit details
    Browse the repository at this point in the history

  10. [Large]ListViewArrayBuilder: Add list-view builder classes

    @felipecrv
    felipecrv committed Oct 26, 2023
    Configuration menu
    Copy the full SHA
    06f9072 View commit details
    Browse the repository at this point in the history

  11. [Large]ListViewArray: Buffers validation, creation from JSON, and bas… 

    …ic tests
    @felipecrv
    felipecrv committed Oct 26, 2023
    Configuration menu
    Copy the full SHA
    08d541b View commit details
    Browse the repository at this point in the history

  12. [Large]ListViewScalar: Implement all operations

    @felipecrv
    felipecrv committed Oct 26, 2023
    Configuration menu
    Copy the full SHA
    1d04ed8 View commit details
    Browse the repository at this point in the history

  13. [Large]ListViewArray: Implement Validate

    @felipecrv
    felipecrv committed Oct 26, 2023
    Configuration menu
    Copy the full SHA
    5bfd169 View commit details
    Browse the repository at this point in the history

  14. [Large]ListViewArray: Implement Flatten()

    @felipecrv
    felipecrv committed Oct 26, 2023
    Configuration menu
    Copy the full SHA
    40e9e5f View commit details
    Browse the repository at this point in the history

  15. [Large]ListViewArray: Implement Compare + most of the unit tests

    @felipecrv
    felipecrv committed Oct 26, 2023
    Configuration menu
    Copy the full SHA
    7e3f538 View commit details
    Browse the repository at this point in the history

  16. [Large]ListViewArray: Implement PrettyPrint + tests

    @felipecrv
    felipecrv committed Oct 26, 2023
    Configuration menu
    Copy the full SHA
    21cf422 View commit details
    Browse the repository at this point in the history

  17. type_traits.h: Add LargeType to TypeTraits of list and list-view types

    @felipecrv
    felipecrv committed Oct 26, 2023
    Configuration menu
    Copy the full SHA
    eda45cc View commit details
    Browse the repository at this point in the history

  18. concatenate.cc: Extract a SumBufferSizes() function and make some tweaks

    @felipecrv
    felipecrv committed Oct 26, 2023
    Configuration menu
    Copy the full SHA
    91c1c00 View commit details
    Browse the repository at this point in the history

  19. concatenate.cc: Pass Buffer pointer already dereferenced to PutOffsets

    @felipecrv
    felipecrv committed Oct 26, 2023
    Configuration menu
    Copy the full SHA
    05570ad View commit details
    Browse the repository at this point in the history

  20. list_util.h: Add RangeOfValuesUsed() function

    @felipecrv
    felipecrv committed Oct 26, 2023
    Configuration menu
    Copy the full SHA
    56352d9 View commit details
    Browse the repository at this point in the history

  21. concatenate.cc: Rename variable from adjustment to displacement

    @felipecrv
    felipecrv committed Oct 26, 2023
    Configuration menu
    Copy the full SHA
    14b1a60 View commit details
    Browse the repository at this point in the history

  22. [Large]ListViewArray: Implement concatenation + tests

    @felipecrv
    felipecrv committed Oct 26, 2023
    Configuration menu
    Copy the full SHA
    f54ca23 View commit details
    Browse the repository at this point in the history

  23. Fix comment formatting

    @felipecrv
    felipecrv committed Oct 26, 2023
    Configuration menu
    Copy the full SHA
    a15bc8c View commit details
    Browse the repository at this point in the history

  24. BaseBuilder: Allow ListView scalars and slices to be added to List bu… 

    …ilders and vice-versa
    @felipecrv
    felipecrv committed Oct 26, 2023
    Configuration menu
    Copy the full SHA
    9398c98 View commit details
    Browse the repository at this point in the history

  25. if_else: Rename RunLoop to RunLoopOfNestedIfElseExec

    @felipecrv
    felipecrv committed Oct 26, 2023
    Configuration menu
    Copy the full SHA
    b5ea57f View commit details
    Browse the repository at this point in the history

  26. if_else_benchmarks: Add ListView benchmarks

    @felipecrv
    felipecrv committed Oct 26, 2023
    Configuration menu
    Copy the full SHA
    4b11630 View commit details
    Browse the repository at this point in the history

  27. if_else: Include LIST_VIEW and LARGE_LIST_VIEW

    @felipecrv
    felipecrv committed Oct 26, 2023
    Configuration menu
    Copy the full SHA
    1f749d5 View commit details
    Browse the repository at this point in the history

  28. list_util: Add ListView<->List conversion functions + tests

    @felipecrv
    felipecrv committed Oct 26, 2023
    Configuration menu
    Copy the full SHA
    24efd00 View commit details
    Browse the repository at this point in the history

  29. Declare dtor explicitly on base list[view] builders

    @felipecrv
    felipecrv committed Oct 26, 2023
    Configuration menu
    Copy the full SHA
    7da4b11 View commit details
    Browse the repository at this point in the history

  30. Revert "BaseBuilder: Allow ListView scalars and slices to be added to… 

    … List builders and vice-versa"

This reverts commit 9b13c8f.
    @felipecrv
    felipecrv committed Oct 26, 2023
    Configuration menu
    Copy the full SHA
    0b04fc6 View commit details
    Browse the repository at this point in the history

  31. concatenate.cc: Add a fast path to Concatenate when only 1 array is p… 

    …assed
    @felipecrv
    felipecrv committed Oct 26, 2023
    Configuration menu
    Copy the full SHA
    693477c View commit details
    Browse the repository at this point in the history

  32. [Large]ListViewArray: Re-write Flatten

    @felipecrv
    felipecrv committed Oct 26, 2023
    Configuration menu
    Copy the full SHA
    aa0846e View commit details
    Browse the repository at this point in the history

  33. scalar_test.cc: Instantiate basic scalar tests with list-view types

    @felipecrv
    felipecrv committed Oct 26, 2023
    Configuration menu
    Copy the full SHA
    f9e9305 View commit details
    Browse the repository at this point in the history

  34. concatenate_test.cc: De-duplicate code checking concatenation of list… 

    …s and list-views
    @felipecrv
    felipecrv committed Oct 26, 2023
    Configuration menu
    Copy the full SHA
    8cacd44 View commit details
    Browse the repository at this point in the history

  35. Revert "concatenate.cc: Add a fast path to Concatenate when only 1 ar… 

    …ray is passed"

This reverts commit 2f4471f.
    @felipecrv
    felipecrv committed Oct 26, 2023
    Configuration menu
    Copy the full SHA
    ac5b94e View commit details
    Browse the repository at this point in the history

  36. validate.cc: Be strict about nullability of offsets and sizes buffers… 

    … on list-views
    @felipecrv
    felipecrv committed Oct 26, 2023
    Configuration menu
    Copy the full SHA
    69345b4 View commit details
    Browse the repository at this point in the history

  37. list_util.h: Add SumOfLogicalListSizes() utility

    @felipecrv
    felipecrv committed Oct 26, 2023
    Configuration menu
    Copy the full SHA
    540086a View commit details
    Browse the repository at this point in the history

  38. list_util.cc: Use SumOfLogicalListValues when converting between list… 

    …s and list-views
    @felipecrv
    felipecrv committed Oct 26, 2023
    Configuration menu
    Copy the full SHA
    f3596b6 View commit details
    Browse the repository at this point in the history

  39. list_util.cc: Rewrite MinViewOffset and MaxViewEnd

    @felipecrv
    felipecrv committed Oct 26, 2023
    Configuration menu
    Copy the full SHA
    ef436d7 View commit details
    Browse the repository at this point in the history

  40. list_util.c: Zero initial padding area of sizes buffer

    @felipecrv
    felipecrv committed Oct 26, 2023
    Configuration menu
    Copy the full SHA
    d61d15d View commit details
    Browse the repository at this point in the history

  41. random.cc: Simplify and split the random generator into two algorithms

    @felipecrv
    felipecrv committed Oct 26, 2023
    Configuration menu
    Copy the full SHA
    0c18fdd View commit details
    Browse the repository at this point in the history

  42. concatenate.cc: Respect the new invariants imposed on the spec

    @felipecrv
    felipecrv committed Oct 26, 2023
    Configuration menu
    Copy the full SHA
    7e51ae7 View commit details
    Browse the repository at this point in the history

  43. validate.cc: Apply list-view invariants to null list-views as well

    @felipecrv
    felipecrv committed Oct 26, 2023
    Configuration menu
    Copy the full SHA
    32865ec View commit details
    Browse the repository at this point in the history

  44. util.cc: Implement Endianness swapping for list-views

    @felipecrv
    felipecrv committed Oct 26, 2023
    Configuration menu
    Copy the full SHA
    ffc8f16 View commit details
    Browse the repository at this point in the history

  45. IPC: Parameterize JSON simple tests arrow list type

    @felipecrv
    felipecrv committed Oct 26, 2023
    Configuration menu
    Copy the full SHA
    8e001f6 View commit details
    Browse the repository at this point in the history

  46. IPC: Include ListView types in the simple JSON list tests

    @felipecrv
    felipecrv committed Oct 26, 2023
    Configuration menu
    Copy the full SHA
    7ff434c View commit details
    Browse the repository at this point in the history

  47. IPC and JSON Integration tests for list-view types

    @felipecrv
    felipecrv committed Oct 26, 2023
    Configuration menu
    Copy the full SHA
    fb92b1e View commit details
    Browse the repository at this point in the history

  48. Add ListView support to the C data interface

    @felipecrv
    felipecrv committed Oct 26, 2023
    Configuration menu
    Copy the full SHA
    e40b1fa View commit details
    Browse the repository at this point in the history

  49. validate.cc: Re-do list-view modifications on top of concurrent changes

    @felipecrv
    felipecrv committed Oct 26, 2023
    Configuration menu
    Copy the full SHA
    17c0370 View commit details
    Browse the repository at this point in the history

  50. RangeOfValuesUsed: Don't call MaxViewEnd if MinViewOffset is a nullopt 

    Since that implies all non-null list-views are empty.
    @felipecrv
    felipecrv committed Oct 26, 2023
    Configuration menu
    Copy the full SHA
    fd927a8 View commit details
    Browse the repository at this point in the history

  51. concatenate.cc: Rename SumBufferSizes{->InBytes}

    @felipecrv
    felipecrv committed Oct 26, 2023
    Configuration menu
    Copy the full SHA
    627af2f View commit details
    Browse the repository at this point in the history

  52. concatenate.cc: Use mutable_data_as<T>

    @felipecrv
    felipecrv committed Oct 26, 2023
    Configuration menu
    Copy the full SHA
    e1cfd96 View commit details
    Browse the repository at this point in the history

  53. Clarify the validations in List[View]Array::FromArrays() docstrings

    @felipecrv
    felipecrv committed Oct 26, 2023
    Configuration menu
    Copy the full SHA
    3ecbcf1 View commit details
    Browse the repository at this point in the history

  54. Rewrite ListView Flatten

    @felipecrv
    felipecrv committed Oct 26, 2023
    Configuration menu
    Copy the full SHA
    0cf760c View commit details
    Browse the repository at this point in the history

  55. list_util: Use mutable_data_as<T>

    @felipecrv
    felipecrv committed Oct 26, 2023
    Configuration menu
    Copy the full SHA
    8f87038 View commit details
    Browse the repository at this point in the history

  56. More throroughly document builder member functions

    @felipecrv
    felipecrv committed Oct 26, 2023
    Configuration menu
    Copy the full SHA
    15b0904 View commit details
    Browse the repository at this point in the history

  57. fixup! concatenate.cc: Respect the new invariants imposed on the spec

    @felipecrv
    felipecrv committed Oct 26, 2023
    Configuration menu
    Copy the full SHA
    dde0f5a View commit details
    Browse the repository at this point in the history

Commits on Nov 9, 2023

  1. fixup! Clarify the validations in List[View]Array::FromArrays() docst… 

    …rings
    @felipecrv
    felipecrv committed Nov 9, 2023
    Configuration menu
    Copy the full SHA
    8d04a38 View commit details
    Browse the repository at this point in the history

  2. Make [Large]ListViewArray docstrings consistent with each other

    @felipecrv
    felipecrv committed Nov 9, 2023
    Configuration menu
    Copy the full SHA
    c490d84 View commit details
    Browse the repository at this point in the history

  3. fixup! [Large]ListViewScalar: Implement all operations

    @felipecrv
    felipecrv committed Nov 9, 2023
    Configuration menu
    Copy the full SHA
    36bb781 View commit details
    Browse the repository at this point in the history

  4. fixup! [Large]ListViewArray: Implement Validate

    @felipecrv
    felipecrv committed Nov 9, 2023
    Configuration menu
    Copy the full SHA
    4b77163 View commit details
    Browse the repository at this point in the history

  5. fixup! validate.cc: Be strict about nullability of offsets and sizes … 

    …buffers on list-views
    @felipecrv
    felipecrv committed Nov 9, 2023
    Configuration menu
    Copy the full SHA
    f02d5ce View commit details
    Browse the repository at this point in the history

Commits on Nov 10, 2023

  1. Make the C Bridge test set for list-views complete

    @felipecrv
    felipecrv committed Nov 10, 2023
    Configuration menu
    Copy the full SHA
    311603c View commit details
    Browse the repository at this point in the history

  2. fixup! IPC and JSON Integration tests for list-view types

    @felipecrv
    felipecrv committed Nov 10, 2023
    Configuration menu
    Copy the full SHA
    26e5288 View commit details
    Browse the repository at this point in the history

  3. fixup! random.cc: Simplify and split the random generator into two al… 

    …gorithms
    @felipecrv
    felipecrv committed Nov 10, 2023
    Configuration menu
    Copy the full SHA
    0da6695 View commit details
    Browse the repository at this point in the history

  4. fixup! [Large]ListViewArray: Implement Compare + most of the unit tests

    @felipecrv
    felipecrv committed Nov 10, 2023
    Configuration menu
    Copy the full SHA
    551466a View commit details
    Browse the repository at this point in the history

  5. fixup! list_util.h: Add RangeOfValuesUsed() function

    @felipecrv
    felipecrv committed Nov 10, 2023
    Configuration menu
    Copy the full SHA
    6922d27 View commit details
    Browse the repository at this point in the history

Commits on Nov 13, 2023

  1. fixup! list_util.cc: Rewrite MinViewOffset and MaxViewEnd

    @felipecrv
    felipecrv committed Nov 13, 2023
    Configuration menu
    Copy the full SHA
    8e6af3e View commit details
    Browse the repository at this point in the history

Commits on Nov 16, 2023

  1. fixup! More throroughly document builder member functions

    @felipecrv
    felipecrv committed Nov 16, 2023
    Configuration menu
    Copy the full SHA
    698b1df View commit details
    Browse the repository at this point in the history

  2. fixup! [Large]ListViewArrayBuilder: Add list-view builder classes

    @felipecrv
    felipecrv committed Nov 16, 2023
    Configuration menu
    Copy the full SHA
    e9fc005 View commit details
    Browse the repository at this point in the history

  3. fixup! [Large]ListViewArray: Implement Compare + most of the unit tests

    @felipecrv
    felipecrv committed Nov 16, 2023
    Configuration menu
    Copy the full SHA
    308781b View commit details
    Browse the repository at this point in the history

  4. fixup! Rewrite ListView Flatten

    @felipecrv
    felipecrv committed Nov 16, 2023
    Configuration menu
    Copy the full SHA
    ec0e10e View commit details
    Browse the repository at this point in the history

  5. fixup! [Large]ListViewArrayBuilder: Add list-view builder classes

    @felipecrv
    felipecrv committed Nov 16, 2023
    Configuration menu
    Copy the full SHA
    8b8c6f0 View commit details
    Browse the repository at this point in the history

  6. fixup! fixup! [Large]ListViewScalar: Implement all operations

    @felipecrv
    felipecrv committed Nov 16, 2023
    Configuration menu
    Copy the full SHA
    2d8e55f View commit details
    Browse the repository at this point in the history

  7. fixup! [Large]ListViewArray: Implement Validate

    @felipecrv
    felipecrv committed Nov 16, 2023
    Configuration menu
    Copy the full SHA
    0972d5c View commit details
    Browse the repository at this point in the history

  8. Remove sparsity parameter from random list-view generator

    @felipecrv
    felipecrv committed Nov 16, 2023
    Configuration menu
    Copy the full SHA
    5f57008 View commit details
    Browse the repository at this point in the history

  9. Move list-x-list-view converters from list_util.h/cc to array_nested.… 

    …h/cc
    @felipecrv
    felipecrv committed Nov 16, 2023
    Configuration menu
    Copy the full SHA
    d554a13 View commit details
    Browse the repository at this point in the history

  10. Move list-x-list-view converter tests to list_array_test.cc

    @felipecrv
    felipecrv committed Nov 16, 2023
    Configuration menu
    Copy the full SHA
    aef0cb6 View commit details
    Browse the repository at this point in the history

Commits on Nov 17, 2023

  1. array_nested.h: Document the IsValid(i) pre-cond on value_length/valu… 

    …e_slice
    @felipecrv
    felipecrv committed Nov 17, 2023
    Configuration menu
    Copy the full SHA
    7d44e76 View commit details
    Browse the repository at this point in the history

  2. fixup! list_util.h: Add RangeOfValuesUsed() function

    @felipecrv
    felipecrv committed Nov 17, 2023
    Configuration menu
    Copy the full SHA
    ab5711f View commit details
    Browse the repository at this point in the history

  3. fixup! [Large]ListViewArray: Implement Validate

    @felipecrv
    felipecrv committed Nov 17, 2023
    Configuration menu
    Copy the full SHA
    4fa8e74 View commit details
    Browse the repository at this point in the history

  4. concatenate_test.cc: Isolate all the random array generation code

    @felipecrv
    felipecrv committed Nov 17, 2023
    Configuration menu
    Copy the full SHA
    d42aae8 View commit details
    Browse the repository at this point in the history

  5. concatenate.cc: Preserve the properties of offsets and sizes required… 

    … by the final list-view spec
    @felipecrv
    felipecrv committed Nov 17, 2023
    Configuration menu
    Copy the full SHA
    252ff8b View commit details
    Browse the repository at this point in the history

  6. fixup! concatenate.cc: Preserve the properties of offsets and sizes r… 

    …equired by the final list-view spec
    @felipecrv
    felipecrv committed Nov 17, 2023
    Configuration menu
    Copy the full SHA
    b30d2e5 View commit details
    Browse the repository at this point in the history

  7. fixup! fixup! [Large]ListViewScalar: Implement all operations

    @felipecrv
    felipecrv committed Nov 17, 2023
    Configuration menu
    Copy the full SHA
    c2068bb View commit details
    Browse the repository at this point in the history

  8. fixup! concatenate_test.cc: Isolate all the random array generation code

    @felipecrv
    felipecrv committed Nov 17, 2023
    Configuration menu
    Copy the full SHA
    a41490e View commit details
    Browse the repository at this point in the history

  9. fixup! Make the C Bridge test set for list-views complete

    @felipecrv
    felipecrv committed Nov 17, 2023
    Configuration menu
    Copy the full SHA
    79c6acd View commit details
    Browse the repository at this point in the history

  10. fixup! list_util.h: Add RangeOfValuesUsed() function

    @felipecrv
    felipecrv committed Nov 17, 2023
    Configuration menu
    Copy the full SHA
    05f6f38 View commit details
    Browse the repository at this point in the history

  11. fixup! concatenate_test.cc: Isolate all the random array generation code

    @felipecrv
    felipecrv committed Nov 17, 2023
    Configuration menu
    Copy the full SHA
    773c800 View commit details
    Browse the repository at this point in the history

Commits on Nov 22, 2023

  1. fixup! list_util.h: Add SumOfLogicalListSizes() utility

    @felipecrv
    felipecrv committed Nov 22, 2023
    Configuration menu
    Copy the full SHA
    ab0dcaf View commit details
    Browse the repository at this point in the history

  2. fixup! concatenate.cc: Preserve the properties of offsets and sizes r… 

    …equired by the final list-view spec
    @felipecrv
    felipecrv committed Nov 22, 2023
    Configuration menu
    Copy the full SHA
    62ede21 View commit details
    Browse the repository at this point in the history