Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Add foreign table providers #921

Merged
merged 28 commits into from
Nov 15, 2024
Merged

Add foreign table providers #921

merged 28 commits into from
Nov 15, 2024

Commits on Nov 11, 2024

  1. testing FFI for table provider

    @timsaucer
    timsaucer committed Nov 11, 2024
    Configuration menu
    Copy the full SHA
    5cd598a View commit details
    Browse the repository at this point in the history

  2. Was able to get round trip schema from datafusion -> delta table -> d… 

    …atafusion
    @timsaucer
    timsaucer committed Nov 11, 2024
    Configuration menu
    Copy the full SHA
    7a476d2 View commit details
    Browse the repository at this point in the history

  3. Expand file structure

    @timsaucer
    timsaucer committed Nov 11, 2024
    Configuration menu
    Copy the full SHA
    f830058 View commit details
    Browse the repository at this point in the history

  4. WIP on execution plan

    @timsaucer
    timsaucer committed Nov 11, 2024
    Configuration menu
    Copy the full SHA
    eec590f View commit details
    Browse the repository at this point in the history

  5. Working through execution plan FFI

    @timsaucer
    timsaucer committed Nov 11, 2024
    Configuration menu
    Copy the full SHA
    f7e7aca View commit details
    Browse the repository at this point in the history

  6. Using datafusion-proto for execution plan properties

    @timsaucer
    timsaucer committed Nov 11, 2024
    Configuration menu
    Copy the full SHA
    803e6ff View commit details
    Browse the repository at this point in the history

  7. Adding plan properties parsing from ffi

    @timsaucer
    timsaucer committed Nov 11, 2024
    Configuration menu
    Copy the full SHA
    9df2f86 View commit details
    Browse the repository at this point in the history

  8. Standardize naming for FFI structs

    @timsaucer
    timsaucer committed Nov 11, 2024
    Configuration menu
    Copy the full SHA
    73a1899 View commit details
    Browse the repository at this point in the history

  9. Intermediate testing and troubleshooting

    @timsaucer
    timsaucer committed Nov 11, 2024
    Configuration menu
    Copy the full SHA
    0d23d55 View commit details
    Browse the repository at this point in the history

  10. Adding record batch stream ffi representation

    @timsaucer
    timsaucer committed Nov 11, 2024
    Configuration menu
    Copy the full SHA
    7d42f73 View commit details
    Browse the repository at this point in the history

  11. Mimimum viable product demonstrating foreign table provider

    @timsaucer
    timsaucer committed Nov 11, 2024
    Configuration menu
    Copy the full SHA
    405a89f View commit details
    Browse the repository at this point in the history

  12. Move ffi module to datafusion core

    @timsaucer
    timsaucer committed Nov 11, 2024
    Configuration menu
    Copy the full SHA
    d122393 View commit details
    Browse the repository at this point in the history

  13. Modifications need to compile against latest DF

    @timsaucer
    timsaucer committed Nov 11, 2024
    Configuration menu
    Copy the full SHA
    88afcc7 View commit details
    Browse the repository at this point in the history

  14. Set DF to 42.0.0

    @timsaucer
    timsaucer committed Nov 11, 2024
    Configuration menu
    Copy the full SHA
    5e386e6 View commit details
    Browse the repository at this point in the history

  15. Rebasing and pulling in a few changes for DF43.0

    @timsaucer
    timsaucer committed Nov 11, 2024
    Configuration menu
    Copy the full SHA
    2764bca View commit details
    Browse the repository at this point in the history

  16. Add wrapper for register table provider

    @timsaucer
    timsaucer committed Nov 11, 2024
    Configuration menu
    Copy the full SHA
    fbf54f1 View commit details
    Browse the repository at this point in the history

  17. Suppress deprecation warning

    @timsaucer
    timsaucer committed Nov 11, 2024
    Configuration menu
    Copy the full SHA
    01a15f1 View commit details
    Browse the repository at this point in the history

  18. Add example for FFI table provider

    @timsaucer
    timsaucer committed Nov 11, 2024
    Configuration menu
    Copy the full SHA
    7f58117 View commit details
    Browse the repository at this point in the history

  19. Add pytest for FFI module to CI

    @timsaucer
    timsaucer committed Nov 11, 2024
    Configuration menu
    Copy the full SHA
    cee25b8 View commit details
    Browse the repository at this point in the history

  20. Add license text

    @timsaucer
    timsaucer committed Nov 11, 2024
    Configuration menu
    Copy the full SHA
    a284554 View commit details
    Browse the repository at this point in the history

  21. Change the name of the FFI table provider test so it doesn't try to r… 

    …un during the first pass of pytest when the module hasn't been built
    @timsaucer
    timsaucer committed Nov 11, 2024
    Configuration menu
    Copy the full SHA
    5551ef9 View commit details
    Browse the repository at this point in the history

  22. Build example in build stage to be used during test stage

    @timsaucer
    timsaucer committed Nov 11, 2024
    Configuration menu
    Copy the full SHA
    50b4bf2 View commit details
    Browse the repository at this point in the history

  23. Combine pytests into one stage

    @timsaucer
    timsaucer committed Nov 11, 2024
    Configuration menu
    Copy the full SHA
    468ef5b View commit details
    Browse the repository at this point in the history

  24. Fix path for unit test

    @timsaucer
    timsaucer committed Nov 11, 2024
    Configuration menu
    Copy the full SHA
    a0a39fd View commit details
    Browse the repository at this point in the history

  25. Installing maturin for ffi test in test script

    @timsaucer
    timsaucer committed Nov 11, 2024
    Configuration menu
    Copy the full SHA
    9529bea View commit details
    Browse the repository at this point in the history

  26. Need to install the wheel for unit test

    @timsaucer
    timsaucer committed Nov 11, 2024
    Configuration menu
    Copy the full SHA
    15458c7 View commit details
    Browse the repository at this point in the history

Commits on Nov 12, 2024

  1. Add online documentation about using custom table providers

    @timsaucer
    timsaucer committed Nov 12, 2024
    Configuration menu
    Copy the full SHA
    64dcbd3 View commit details
    Browse the repository at this point in the history

  2. Raise an error if method is not implemented when it is expected

    @timsaucer
    timsaucer committed Nov 12, 2024
    Configuration menu
    Copy the full SHA
    e31e478 View commit details
    Browse the repository at this point in the history