Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

[FEA] Implement all libcudf modules required by cuDF Python in pylibcudf #15162

Open
vyasr opened this issue Feb 27, 2024 · 0 comments
Open

[FEA] Implement all libcudf modules required by cuDF Python in pylibcudf #15162

vyasr opened this issue Feb 27, 2024 · 0 comments
Assignees
@mroeschke @Matt711
Labels
feature request New feature or request pylibcudf Issues specific to the pylibcudf package Python Affects Python cuDF API.

Comments

@vyasr
Copy link
Contributor

vyasr commented Feb 27, 2024 •
edited by Matt711
Loading

Is your feature request related to a problem? Please describe.
pylibcudf is intended to provide a low-level Python interface to the libcudf C++ API. cuDF's internals will ultimately be refactored to depend on pylibcudf. As a first step, we need to expose all libcudf algorithms used by cuDF Cython in pylibcudf.

Describe the solution you'd like
This is a tracking issue for APIs to expose in Cython. The APIs are grouped based on the pxd file exposing libcudf APIs in Cython, which roughly corresponds to namespaces in libcudf.

Module PRs (or assignees) Notes
aggregation.pxd #14945, #14970
binaryop.pxd #14821
column/column.pxd #13562 pylibcudf Columns share ownership
column/column_factories.pxd #15257
column/column_view.pxd #13562 pylibcudf Columns share ownership
concatenate.pxd #15011
contiguous_split.pxd #16953
copying.pxd #13562, #14508, #14640
datetime.pxd #15916, #16776, #17143
expressions.pxd #16056
filling.pxd #15225
groupby.pxd #14945
hash.pxd #15418
interop.pxd #17055
io/arrow_io_source.pxd #16050
io/avro.pxd #15899
io/csv.pxd #16011 reader only, writer needs porting
io/data_sink.pxd
io/datasource.pxd #16050
io/json.pxd #15952 #15966
io/orc.pxd #16042
io/orc_metadata.pxd
io/parquet.pxd #16078 #17245 writer needs porting
io/text.pxd #17232
io/timezone.pxd #16771
io/types.pxd
join.pxd #14972
labeling.pxd #16761
lists/combine.pxd #15928
lists/contains.pxd #15981
lists/count_elements.pxd #16072
lists/explode.pxd #15011
lists/extract.pxd #16071
lists/gather.pxd #16170
lists/lists_column_view.pxd #16175
lists/sorting.pxd #16179
lists/stream_compaction.pxd #16184
lists/reverse.pxd #16185
merge.pxd #15011
null_mask.pxd #15908
nvtext/byte_pair_encode.pxd #17101
nvtext/edit_distance.pxd #16957
nvtext/generate_ngrams.pxd #17006
nvtext/jaccard.pxd #17007
nvtext/minhash.pxd #17021
nvtext/ngrams_tokenize.pxd #17070
nvtext/normalize.pxd #17072
nvtext/replace.pxd #17084
nvtext/stemmer.pxd #17085
nvtext/subword_tokenize.pxd #17096
nvtext/tokenize.pxd #17100
partitioning.pxd #16781
quantiles.pxd #15874
reduce.pxd #14970
replace.pxd #15005
reshape.pxd #15827 sans byte_cast which is only used by cpp/java
rolling.pxd #14982
round.pxd #15863
scalar/scalar.pxd #14133
search.pxd #15166 #17271
sorting.pxd #15011
stream_compaction.pxd #15011
strings/convert/convert_booleans.pxd #16971
strings/convert/convert_datetime.pxd #16971
strings/convert/convert_durations.pxd #16982
strings/convert/convert_fixed_point.pxd #16984
strings/convert/convert_floats.pxd #16990
strings/convert/convert_integers.pxd #16991 #17270
strings/convert/convert_ipv4.pxd #16994
strings/convert/convert_lists.pxd #16997
strings/convert/convert_urls.pxd #17003
strings/split/partition.pxd #16940
strings/split/split.pxd #16940
strings/attributes.pxd #16785
strings/capitalize.pxd #15503
strings/case.pxd #15489
strings/char_types.pxd #16788
strings/combine.pxd #16790
strings/contains.pxd #16814
strings/extract.pxd #16823
strings/find.pxd #15604
strings/find_multiple.pxd #16920
strings/findall.pxd #16825
strings/json.pxd #17025
strings/padding.pxd #16833
strings/regex_flags.pxd #15880
strings/regex_program.pxd #15880
strings/repeat.pxd #16834
strings/replace.pxd #15839
strings/replace_re.pxd #17023
strings/side_type.pxd #16833
strings/strip.pxd #16833
strings/substring.pxd #15988
strings/translate.pxd #16934
strings/wrap.pxd #16935
strings_udf.pxd #17107 We decided not to include strings_udf in pylibcudf because (if we did) the C++ we'd be creating bindings for in https://github.com/rapidsai/cudf/tree/branch-24.12/python/cudf/udf_cpp is not a part of the libcudf API.
table/table.pxd #13562 Tables share column ownership in pylibcudf
table/table_view.pxd #13562 Tables share column ownership in pylibcudf
transform.pxd #16760
transpose.pxd #16749
types.pxd #13562 More types added as needed in other PRs
unary.pxd #14850
utilities/span.pxd #17021
wrappers/decimals.pxd #17048
wrappers/durations.pxd #17010
wrappers/timestamps.pxd #17010
@vyasr vyasr added the feature request New feature or request label Feb 27, 2024
@vyasr vyasr mentioned this issue Feb 28, 2024
Merged
3 tasks
rapids-bot bot pushed a commit that referenced this issue Mar 2, 2024
@vyasr
Implement search using pylibcudf (#15166)
e60aad1 
Contributes to #15162

Authors:
  - Vyas Ramasubramani (https://github.com/vyasr)

Approvers:
  - Bradley Dice (https://github.com/bdice)

URL: #15166
@vyasr vyasr mentioned this issue Mar 15, 2024
Merged
3 tasks
rapids-bot bot pushed a commit that referenced this issue Mar 18, 2024
@vyasr
Change exceptions thrown by copying APIs (#15319)
13cb4bc 
This PR also introduces `std::out_of_range` to cudf's code base in cases where it is appropriate.

Contributes to #12885 
Resolves #15315 
Contributes to #15162

Authors:
  - Vyas Ramasubramani (https://github.com/vyasr)

Approvers:
  - Bradley Dice (https://github.com/bdice)

URL: #15319
This was referenced Apr 9, 2024
Merged
Merged
rapids-bot bot pushed a commit that referenced this issue Apr 11, 2024
@brandon-b-miller
Migrate string case operations to pylibcudf (#15489)
8506ea6 
This PR creates `pylibcudf` `case` APIs and migrates the cuDF cython to leverage them. Part of #15162.

Authors:
  - https://github.com/brandon-b-miller
  - GALI PREM SAGAR (https://github.com/galipremsagar)

Approvers:
  - Vyas Ramasubramani (https://github.com/vyasr)

URL: #15489
@GregoryKimball GregoryKimball mentioned this issue Apr 18, 2024
Closed
@lithomas1 lithomas1 mentioned this issue May 22, 2024
Merged
3 tasks
rapids-bot bot pushed a commit that referenced this issue May 24, 2024
@lithomas1
Migrate reshape.pxd to pylibcudf (#15827)
8458306 
xref #15162

Authors:
  - Thomas Li (https://github.com/lithomas1)

Approvers:
  - Matthew Roeschke (https://github.com/mroeschke)
  - Vyas Ramasubramani (https://github.com/vyasr)

URL: #15827
This was referenced May 24, 2024
Merged
Merged
Merged
@vyasr vyasr added the pylibcudf Issues specific to the pylibcudf package label May 28, 2024
rapids-bot bot pushed a commit that referenced this issue May 29, 2024
@brandon-b-miller
Migrate string capitalize APIs to pylibcudf (#15503)
bdafa73 
This PR creates the `pylibcudf.strings.capitalize` namespace and migrates the cuDF cython to use it. Depends on #15489

Part of #15162

Authors:
  - https://github.com/brandon-b-miller

Approvers:
  - Kyle Edwards (https://github.com/KyleFromNVIDIA)
  - Vyas Ramasubramani (https://github.com/vyasr)

URL: #15503
@brandon-b-miller brandon-b-miller mentioned this issue May 29, 2024
Merged
rapids-bot bot pushed a commit that referenced this issue May 31, 2024
@lithomas1
Migrate round to pylibcudf (#15863)
e7be142 
xref #15162

Migrate round.pxd to use pylibcudf APIs.

Authors:
  - Thomas Li (https://github.com/lithomas1)

Approvers:
  - https://github.com/brandon-b-miller
  - Lawrence Mitchell (https://github.com/wence-)

URL: #15863
@lithomas1 lithomas1 mentioned this issue May 31, 2024
Merged
3 tasks
@charlesbluca charlesbluca mentioned this issue Jun 3, 2024
Merged
3 tasks
@Matt711 Matt711 mentioned this issue Jun 5, 2024
Merged
3 tasks
rapids-bot bot pushed a commit that referenced this issue Jun 5, 2024
@lithomas1
Migrate string replace.pxd to pylibcudf (#15839)
db1b365 
xref #15162

Change replace.pxd to use pylibcudf APIs.

Authors:
  - Thomas Li (https://github.com/lithomas1)

Approvers:
  - Vyas Ramasubramani (https://github.com/vyasr)

URL: #15839
rapids-bot bot pushed a commit that referenced this issue Jun 6, 2024
@brandon-b-miller
Migrate strings contains operations to pylibcudf (#15880)
7fd6918 
This PR creates pylibcudf strings `contains` APIs and migrates the cuDF cython to leverage them. Part of #15162.

Authors:
  - https://github.com/brandon-b-miller

Approvers:
  - Lawrence Mitchell (https://github.com/wence-)

URL: #15880
rapids-bot bot pushed a commit that referenced this issue Jun 6, 2024
@lithomas1
Start migrating I/O to pylibcudf (#15899)
3b734ec 
xref #15162 

Starts migrating cudf I/O cython to use pylibcudf APIs, starting with avro.

Authors:
  - Thomas Li (https://github.com/lithomas1)
  - GALI PREM SAGAR (https://github.com/galipremsagar)

Approvers:
  - Lawrence Mitchell (https://github.com/wence-)

URL: #15899
rapids-bot bot pushed a commit that referenced this issue Jun 6, 2024
@lithomas1
Migrate quantile.pxd to pylibcudf (#15874)
5f45803 
xref #15162 

Migrate quantile.pxd to use pylibcudf APIs.

Authors:
  - Thomas Li (https://github.com/lithomas1)

Approvers:
  - Lawrence Mitchell (https://github.com/wence-)
  - Vyas Ramasubramani (https://github.com/vyasr)

URL: #15874
This was referenced Jun 11, 2024
Merged
Merged
@Matt711 Matt711 mentioned this issue Jun 11, 2024
Merged
3 tasks
rapids-bot bot pushed a commit that referenced this issue Jun 12, 2024
@Matt711
Migrate lists/combine to pylibcudf (#15928)
d2cd1d4 
Part of #15162. concatenate_rows, concatenate_list_elements

Authors:
  - Matthew Murray (https://github.com/Matt711)

Approvers:
  - Vyas Ramasubramani (https://github.com/vyasr)
  - Thomas Li (https://github.com/lithomas1)

URL: #15928
@brandon-b-miller brandon-b-miller mentioned this issue Jun 12, 2024
Merged
rapids-bot bot pushed a commit that referenced this issue Oct 16, 2024
@Matt711
Migrate remaining nvtext NGrams APIs to pylibcudf (#17070)
3420c71 
Apart of #15162

Authors:
  - Matthew Murray (https://github.com/Matt711)

Approvers:
  - https://github.com/brandon-b-miller

URL: #17070
This was referenced Oct 16, 2024
Merged
Closed
rapids-bot bot pushed a commit that referenced this issue Oct 17, 2024
@mroeschke
Add strings.combine APIs to pylibcudf (#16790)
3683e46 
Contributes to #15162

Authors:
  - Matthew Roeschke (https://github.com/mroeschke)
  - Matthew Murray (https://github.com/Matt711)

Approvers:
  - Lawrence Mitchell (https://github.com/wence-)
  - Matthew Murray (https://github.com/Matt711)

URL: #16790
@Matt711 Matt711 mentioned this issue Oct 17, 2024
Merged
3 tasks
rapids-bot bot pushed a commit that referenced this issue Oct 17, 2024
@Matt711
Migrate NVText Normalizing APIs to Pylibcudf (#17072)
ce93c36 
Apart of #15162.

Authors:
  - Matthew Murray (https://github.com/Matt711)

Approvers:
  - Bradley Dice (https://github.com/bdice)

URL: #17072
rapids-bot bot pushed a commit that referenced this issue Oct 22, 2024
@mroeschke
Add string.replace_re APIs to pylibcudf (#17023)
4fe338c 
Contributes to #15162

Authors:
  - Matthew Roeschke (https://github.com/mroeschke)
  - Matthew Murray (https://github.com/Matt711)

Approvers:
  - Matthew Murray (https://github.com/Matt711)
  - GALI PREM SAGAR (https://github.com/galipremsagar)

URL: #17023
@Matt711 Matt711 mentioned this issue Oct 22, 2024
Merged
3 tasks
rapids-bot bot pushed a commit that referenced this issue Oct 22, 2024
@Matt711
Migrate NVText Replacing APIs to pylibcudf (#17084)
14cdf53 
Apart of #15162

Authors:
  - Matthew Murray (https://github.com/Matt711)
  - Vyas Ramasubramani (https://github.com/vyasr)

Approvers:
  - Vyas Ramasubramani (https://github.com/vyasr)
  - Mark Harris (https://github.com/harrism)

URL: #17084
rapids-bot bot pushed a commit that referenced this issue Oct 24, 2024
@Matt711
Migrate NVText Stemming APIs to pylibcudf (#17085)
d7cdf44 
Apart of #15162

Authors:
  - Matthew Murray (https://github.com/Matt711)

Approvers:
  - Bradley Dice (https://github.com/bdice)

URL: #17085
@Matt711 Matt711 mentioned this issue Oct 24, 2024
Open
3 tasks
rapids-bot bot pushed a commit that referenced this issue Oct 25, 2024
@mroeschke
Add to_dlpack/from_dlpack APIs to pylibcudf (#17055)
8bc9f19 
Contributes to #15162

Could use some advice how to type the input of `from_dlpack` and outut of `to_dlpack` which are PyCapsule objects.
EDIT: I notice Cython just types them as object https://github.com/cython/cython/blob/master/Cython/Includes/cpython/pycapsule.pxd. Stylistically do we want add `object var_name` or just leave untyped?

Authors:
  - Matthew Roeschke (https://github.com/mroeschke)
  - Matthew Murray (https://github.com/Matt711)

Approvers:
  - Matthew Murray (https://github.com/Matt711)
  - Vyas Ramasubramani (https://github.com/vyasr)

URL: #17055
rapids-bot bot pushed a commit that referenced this issue Oct 31, 2024
@Matt711
Add remaining datetime APIs to pylibcudf (#17143)
3cf186c 
Apart of #15162

Authors:
  - Matthew Murray (https://github.com/Matt711)

Approvers:
  - https://github.com/brandon-b-miller

URL: #17143
rapids-bot bot pushed a commit that referenced this issue Oct 31, 2024
@Matt711
Migrate NVText Tokenizing APIs to pylibcudf (#17100)
893d0fd 
Apart of #15162

Authors:
  - Matthew Murray (https://github.com/Matt711)

Approvers:
  - Yunsong Wang (https://github.com/PointKernel)
  - Muhammad Haseeb (https://github.com/mhaseeb123)
  - Vyas Ramasubramani (https://github.com/vyasr)

URL: #17100
rapids-bot bot pushed a commit that referenced this issue Oct 31, 2024
@Matt711
Migrate NVText Byte Pair Encoding APIs to pylibcudf (#17101)
0db2463 
Apart of #15162

Authors:
  - Matthew Murray (https://github.com/Matt711)

Approvers:
  - Vyas Ramasubramani (https://github.com/vyasr)

URL: #17101
rapids-bot bot pushed a commit that referenced this issue Oct 31, 2024
@Matt711
Migrate NVtext subword tokenizing APIs to pylibcudf (#17096)
a0711d0 
Apart of #15162

Authors:
  - Matthew Murray (https://github.com/Matt711)

Approvers:
  - Vyas Ramasubramani (https://github.com/vyasr)

URL: #17096
This was referenced Nov 1, 2024
Merged
Open
@Matt711 Matt711 mentioned this issue Nov 5, 2024
Draft
3 tasks
@mroeschke mroeschke mentioned this issue Nov 6, 2024
Draft
3 tasks
@Matt711 Matt711 mentioned this issue Nov 7, 2024
Merged
3 tasks
rapids-bot bot pushed a commit that referenced this issue Nov 7, 2024
@mroeschke
Add io.text APIs to pylibcudf (#17232)
c209dae 
Contributes to #15162

Authors:
  - Matthew Roeschke (https://github.com/mroeschke)
  - Vyas Ramasubramani (https://github.com/vyasr)

Approvers:
  - Lawrence Mitchell (https://github.com/wence-)

URL: #17232
@Matt711 Matt711 mentioned this issue Nov 7, 2024
Merged
3 tasks
rapids-bot bot pushed a commit that referenced this issue Nov 7, 2024
@Matt711
Use pylibcudf.strings.convert.convert_integers.is_integer in cudf p…
773aefc 
…ython (#17270)

Apart of #15162

Authors:
  - Matthew Murray (https://github.com/Matt711)

Approvers:
  - Matthew Roeschke (https://github.com/mroeschke)

URL: #17270
@Matt711 Matt711 mentioned this issue Nov 7, 2024
Merged
3 tasks
rapids-bot bot pushed a commit that referenced this issue Nov 7, 2024
@Matt711
Use pylibcudf.search APIs in cudf python (#17271)
c73defd 
Apart of #15162

Authors:
  - Matthew Murray (https://github.com/Matt711)

Approvers:
  - Matthew Roeschke (https://github.com/mroeschke)

URL: #17271
@Matt711 Matt711 mentioned this issue Nov 7, 2024
Open
3 tasks
rapids-bot bot pushed a commit that referenced this issue Nov 8, 2024
@Matt711
Plumb pylibcudf datetime APIs through cudf python (#17275)
18041b5 
Apart of #15162

Authors:
  - Matthew Murray (https://github.com/Matt711)

Approvers:
  - Matthew Roeschke (https://github.com/mroeschke)

URL: #17275
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees

@mroeschke mroeschke

@Matt711 Matt711

Labels
feature request New feature or request pylibcudf Issues specific to the pylibcudf package Python Affects Python cuDF API.
Projects
cuDF Python
Status: In Progress
cuDF/Dask/Numba/UCX
Status: In Progress
libcudf
Status: No status
Milestone
No milestone
Development

No branches or pull requests
3 participants
@vyasr @mroeschke @Matt711