Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Add an example to demonstrate multithreaded read_parquet pipelines #16828

Merged
merged 52 commits into from
Oct 11, 2024
Merged

Add an example to demonstrate multithreaded read_parquet pipelines #16828

merged 52 commits into from
Oct 11, 2024

Commits on Sep 18, 2024

  1. Add the new multithreaded parquet example

    @mhaseeb123
    mhaseeb123 committed Sep 18, 2024
    Configuration menu
    Copy the full SHA
    ff2480b View commit details
    Browse the repository at this point in the history

  2. Set the default output path to the current path

    @mhaseeb123
    mhaseeb123 committed Sep 18, 2024
    Configuration menu
    Copy the full SHA
    d06f7f2 View commit details
    Browse the repository at this point in the history

  3. Style fix

    @mhaseeb123
    mhaseeb123 committed Sep 18, 2024
    Configuration menu
    Copy the full SHA
    c13a408 View commit details
    Browse the repository at this point in the history

  4. Use stream pool for parquet write as well

    @mhaseeb123
    mhaseeb123 committed Sep 18, 2024
    Configuration menu
    Copy the full SHA
    12adeeb View commit details
    Browse the repository at this point in the history

  5. Add more details to the example

    @mhaseeb123
    mhaseeb123 committed Sep 18, 2024
    Configuration menu
    Copy the full SHA
    a8ae50a View commit details
    Browse the repository at this point in the history

  6. Minor improvements

    @mhaseeb123
    mhaseeb123 committed Sep 18, 2024
    Configuration menu
    Copy the full SHA
    6679f89 View commit details
    Browse the repository at this point in the history

  7. Minor improvement

    @mhaseeb123
    mhaseeb123 committed Sep 18, 2024
    Configuration menu
    Copy the full SHA
    e04602c View commit details
    Browse the repository at this point in the history

  8. Minor improvements

    @mhaseeb123
    mhaseeb123 committed Sep 18, 2024
    Configuration menu
    Copy the full SHA
    21ce7c7 View commit details
    Browse the repository at this point in the history

  9. Merge branch 'branch-24.10' into fea-parquet-multithreaded-example

    @mhaseeb123
    mhaseeb123 authored Sep 18, 2024
    Configuration menu
    Copy the full SHA
    b649530 View commit details
    Browse the repository at this point in the history

Commits on Sep 19, 2024

  1. Move the vector to concatenate tables

    @mhaseeb123
    mhaseeb123 committed Sep 19, 2024
    Configuration menu
    Copy the full SHA
    b8b8bb9 View commit details
    Browse the repository at this point in the history

Commits on Sep 23, 2024

  1. Minor improvement

    @mhaseeb123
    mhaseeb123 committed Sep 23, 2024
    Configuration menu
    Copy the full SHA
    188ce11 View commit details
    Browse the repository at this point in the history

  2. Merge branch 'branch-24.10' into fea-parquet-multithreaded-example

    @mhaseeb123
    mhaseeb123 authored Sep 23, 2024
    Configuration menu
    Copy the full SHA
    1827654 View commit details
    Browse the repository at this point in the history

  3. Merge branch 'branch-24.12' into fea-parquet-multithreaded-example

    @mhaseeb123
    mhaseeb123 authored Sep 23, 2024
    Configuration menu
    Copy the full SHA
    e14916f View commit details
    Browse the repository at this point in the history

Commits on Sep 26, 2024

  1. Merge branch 'branch-24.12' into fea-parquet-multithreaded-example

    @mhaseeb123
    mhaseeb123 authored Sep 26, 2024
    Configuration menu
    Copy the full SHA
    a528eb3 View commit details
    Browse the repository at this point in the history

Commits on Sep 27, 2024

  1. Make multithreaded parquet io example more sophisticated

    @mhaseeb123
    mhaseeb123 committed Sep 27, 2024
    Configuration menu
    Copy the full SHA
    990f2bb View commit details
    Browse the repository at this point in the history

  2. Minor updates

    @mhaseeb123
    mhaseeb123 committed Sep 27, 2024
    Configuration menu
    Copy the full SHA
    06817d0 View commit details
    Browse the repository at this point in the history

  3. Merge branch 'branch-24.12' into fea-parquet-multithreaded-example

    @mhaseeb123
    mhaseeb123 authored Sep 27, 2024
    Configuration menu
    Copy the full SHA
    9dc8a1e View commit details
    Browse the repository at this point in the history

  4. Minor improvements

    @mhaseeb123
    mhaseeb123 committed Sep 27, 2024
    Configuration menu
    Copy the full SHA
    af8ec6a View commit details
    Browse the repository at this point in the history

  5. Set default thread count = 1 instead of 2

    @mhaseeb123
    mhaseeb123 committed Sep 27, 2024
    Configuration menu
    Copy the full SHA
    d3778cc View commit details
    Browse the repository at this point in the history

  6. Minor improvement

    @mhaseeb123
    mhaseeb123 committed Sep 27, 2024
    Configuration menu
    Copy the full SHA
    c2b39cc View commit details
    Browse the repository at this point in the history

Commits on Oct 1, 2024

  1. Add io source types

    @mhaseeb123
    mhaseeb123 committed Oct 1, 2024
    Configuration menu
    Copy the full SHA
    8f39fb2 View commit details
    Browse the repository at this point in the history

  2. Minor comment updates

    @mhaseeb123
    mhaseeb123 committed Oct 1, 2024
    Configuration menu
    Copy the full SHA
    d0c2a62 View commit details
    Browse the repository at this point in the history

  3. Style fix and add to CI.

    @mhaseeb123
    mhaseeb123 committed Oct 1, 2024
    Configuration menu
    Copy the full SHA
    945c0c0 View commit details
    Browse the repository at this point in the history

  4. Minor improvement

    @mhaseeb123
    mhaseeb123 committed Oct 1, 2024
    Configuration menu
    Copy the full SHA
    f30c801 View commit details
    Browse the repository at this point in the history

  5. Merge branch 'branch-24.12' into fea-parquet-multithreaded-example

    @mhaseeb123
    mhaseeb123 authored Oct 1, 2024
    Configuration menu
    Copy the full SHA
    f701e34 View commit details
    Browse the repository at this point in the history

  6. Updates

    @mhaseeb123
    mhaseeb123 committed Oct 1, 2024
    Configuration menu
    Copy the full SHA
    719bfb6 View commit details
    Browse the repository at this point in the history

  7. Style fix.

    @mhaseeb123
    mhaseeb123 committed Oct 1, 2024
    Configuration menu
    Copy the full SHA
    2ade064 View commit details
    Browse the repository at this point in the history

Commits on Oct 2, 2024

  1. Print message when skipping a subdirectory

    @mhaseeb123
    mhaseeb123 committed Oct 2, 2024
    Configuration menu
    Copy the full SHA
    b559eaf View commit details
    Browse the repository at this point in the history

  2. Update cpp/examples/parquet_io/io_source.hpp

    @mhaseeb123
    mhaseeb123 authored Oct 2, 2024
    Configuration menu
    Copy the full SHA
    73de5bc View commit details
    Browse the repository at this point in the history

  3. Update cpp/examples/parquet_io/common_utils.cpp

    @mhaseeb123
    mhaseeb123 authored Oct 2, 2024
    Configuration menu
    Copy the full SHA
    52e6953 View commit details
    Browse the repository at this point in the history

  4. Do not use fmtlib

    @mhaseeb123
    mhaseeb123 committed Oct 2, 2024
    Configuration menu
    Copy the full SHA
    6194a50 View commit details
    Browse the repository at this point in the history

  5. Minor style fix

    @mhaseeb123
    mhaseeb123 committed Oct 2, 2024
    Configuration menu
    Copy the full SHA
    3420c3f View commit details
    Browse the repository at this point in the history

  6. Minor change

    @mhaseeb123
    mhaseeb123 authored Oct 2, 2024
    Configuration menu
    Copy the full SHA
    85d906d View commit details
    Browse the repository at this point in the history

Commits on Oct 3, 2024

  1. Address minor nits from reviews

    @mhaseeb123
    mhaseeb123 committed Oct 3, 2024
    Configuration menu
    Copy the full SHA
    70ec6fd View commit details
    Browse the repository at this point in the history

  2. Update cpp/examples/parquet_io/parquet_io_multithreaded.cpp 

    Co-authored-by: David Wendt <45795991+davidwendt@users.noreply.github.com>
    @mhaseeb123 @davidwendt
    mhaseeb123 and davidwendt authored Oct 3, 2024
    Configuration menu
    Copy the full SHA
    00390cd View commit details
    Browse the repository at this point in the history

  3. Move code to cpp files and minor refactoring

    @mhaseeb123
    mhaseeb123 committed Oct 3, 2024
    Configuration menu
    Copy the full SHA
    5ad8ecd View commit details
    Browse the repository at this point in the history

  4. Minor style fix

    @mhaseeb123
    mhaseeb123 committed Oct 3, 2024
    Configuration menu
    Copy the full SHA
    74763b0 View commit details
    Browse the repository at this point in the history

Commits on Oct 4, 2024

  1. Minor updates

    @mhaseeb123
    mhaseeb123 committed Oct 4, 2024
    Configuration menu
    Copy the full SHA
    06afb49 View commit details
    Browse the repository at this point in the history

  2. Merge branch 'branch-24.12' into fea-parquet-multithreaded-example

    @mhaseeb123
    mhaseeb123 authored Oct 4, 2024
    Configuration menu
    Copy the full SHA
    db9aace View commit details
    Browse the repository at this point in the history

  3. Nits from code reviews

    @mhaseeb123
    mhaseeb123 committed Oct 4, 2024
    Configuration menu
    Copy the full SHA
    1a04409 View commit details
    Browse the repository at this point in the history

  4. Merge branch 'fea-parquet-multithreaded-example' of https://github.co… 

    …m/mhaseeb123/cudf into fea-parquet-multithreaded-example
    @mhaseeb123
    mhaseeb123 committed Oct 4, 2024
    Configuration menu
    Copy the full SHA
    2fb523f View commit details
    Browse the repository at this point in the history

Commits on Oct 7, 2024

  1. Merge branch 'branch-24.12' into fea-parquet-multithreaded-example

    @mhaseeb123
    mhaseeb123 authored Oct 7, 2024
    Configuration menu
    Copy the full SHA
    8be6710 View commit details
    Browse the repository at this point in the history

  2. Merge branch 'branch-24.12' into fea-parquet-multithreaded-example

    @mhaseeb123
    mhaseeb123 authored Oct 7, 2024
    Configuration menu
    Copy the full SHA
    2a6db5d View commit details
    Browse the repository at this point in the history

  3. Merge branch 'branch-24.12' into fea-parquet-multithreaded-example

    @mhaseeb123
    mhaseeb123 authored Oct 7, 2024
    Configuration menu
    Copy the full SHA
    cc6242c View commit details
    Browse the repository at this point in the history

Commits on Oct 10, 2024

  1. Minor arg setting

    @mhaseeb123
    mhaseeb123 committed Oct 10, 2024
    Configuration menu
    Copy the full SHA
    3a59027 View commit details
    Browse the repository at this point in the history

  2. Merge branch 'branch-24.12' into fea-parquet-multithreaded-example

    @mhaseeb123
    mhaseeb123 authored Oct 10, 2024
    Configuration menu
    Copy the full SHA
    60e5d75 View commit details
    Browse the repository at this point in the history

Commits on Oct 11, 2024

  1. Adjust spacing

    @mhaseeb123
    mhaseeb123 committed Oct 11, 2024
    Configuration menu
    Copy the full SHA
    7cfd7ae View commit details
    Browse the repository at this point in the history

  2. Merge branch 'branch-24.12' into fea-parquet-multithreaded-example

    @mhaseeb123
    mhaseeb123 authored Oct 11, 2024
    Configuration menu
    Copy the full SHA
    d1fbad8 View commit details
    Browse the repository at this point in the history

  3. Apply suggestion

    @mhaseeb123
    mhaseeb123 committed Oct 11, 2024
    Configuration menu
    Copy the full SHA
    d9102f0 View commit details
    Browse the repository at this point in the history

  4. Merge branch 'fea-parquet-multithreaded-example' of https://github.co… 

    …m/mhaseeb123/cudf into fea-parquet-multithreaded-example
    @mhaseeb123
    mhaseeb123 committed Oct 11, 2024
    Configuration menu
    Copy the full SHA
    174e6c9 View commit details
    Browse the repository at this point in the history

  5. Minor

    @mhaseeb123
    mhaseeb123 committed Oct 11, 2024
    Configuration menu
    Copy the full SHA
    b61f18e View commit details
    Browse the repository at this point in the history

  6. Merge branch 'branch-24.12' into fea-parquet-multithreaded-example

    @mhaseeb123
    mhaseeb123 authored Oct 11, 2024
    Configuration menu
    Copy the full SHA
    803e8c9 View commit details
    Browse the repository at this point in the history