EPIC: Add all SessionContext and DataFrame methods to Python API

[andygrove]

The Python bindings currently only expose a subset of functionality, and we want to expose as much as possible.

Here is a list of all available rust methods. Note that there may be reasons why we don't want to expose some of these.

SessionContext

• new
• with_config
• register_csv
• register_parquet
• sql
• catalog
• table_exist
• table
• tables
• register_table
• register_udf
• register_udaf
• Add Python bindings for SessionContext methods read_csv, read_json, read_parquet, read_avro #28
• register_variable
• read_table
• Add read_json to SessionContext #56
• [SessionContext] - Add read_csv/read_parquet/read_avro functions to SessionContext #57
• [DataFrame] - Add write_csv/write_parquet/write_json to DataFrame #58
• register_listing_table
• register_json
• register_avro
• register_catalog
• deregister_table
• session_id
• state

DataFrame

• new
• select
• select_columns
• with_column
• with_column_renamed
• explain
• collect
• show / show_limit
• schema
• filter
• aggregate
• sort
• join
• limit
• Add Python bindings for DataFrame methods intersect and except #27
• Add Python bindings for DataFrame methods union and union_distinct #25
• Add Python binding for DataFrame::distinct #29
• Add Python binding for DataFrame::repartition #30
• Add Python bindings for DataFrame methods write_csv, write_parquet, and write_json #26
• Implement DataFrame execute_stream and execute_stream_partitioned #12
• collect_partitioned
• to_logical_plan
• create_physical_plan
• registry

[francis-du]

@andygrove Hi Andy. Could you update this task list? If you have a time.

[nkarpov]

Noticing that join_on (supporting arbitrary expressions instead of keys) is not present in the list above.

I can't find a way to use the existing binding to join on an arbitrary expression

Should the Rust join_on have its own wrapper too? or should we have the existing join wrapper choose appropriately?

[ami-navon]

+1 register_json( )