Update .proto files to the same version that Arrow is using

[paleolimbot]

Currently fails spectacularly because:

• Installing duckdb with remotes and DUCKDB_R_EXTENSIONS=substrait fails with ModuleNotFoundError: No module named 'substrait_config'; the INSTALL 'substrait'; LOAD 'substrait'; SQL command also fails (IO Error: Failed to download extension "substrait" at URL "http://extensions.duckdb.org/deadbeeff/osx_arm64/substrait.duckdb_extension.gz")
• ARROW-16989: [C++] Substrait ProjectRel is interpreted incorrectly apache/arrow#13528 changed the ProjectRel to mean "add these columns" (i.e., mutate()) rather than "clear the column list and create new columns" (i.e., transmute()). I haven't looked into how to drop columns (something involving substrait::substrait$RelCommon$Emit$create())

[paleolimbot]

It looks like it's on its way, but we need apache/arrow#13914 to merge before we can use Arrow's substrait compiler to pass any mutate() tests.

DuckDB's substrait compiler now requires strong typing (previously it was perfectly happy to accept substrait$Type$create(), which we used to signify "we have no clue what the type is").

[paleolimbot]

Ok, the CMD check is clean! There are a lot of files in the diff because I updated the internal copy of substrait, but the main behaviour change is substrait_project(), which now appends (with the option to reorder or drop) instead of replaces. I kept the old version as substrait_select() (now implemented using substrait_project() because we used it in a lot of places although I could remove it entirely if that's confusing. There are also changes to the duckdb compiler because the substrait.Expression.ScalarFunction now uses the new list of substrait.FunctionArgument instead of a straight list of substrait.Expression. The function translations there are still a mess (I'm planning to clean that up in a PR for #194).

# needs install.packages("duckdb") to get the latest version (0.5.0)
library(substrait, warn.conflicts = FALSE)
library(dplyr, warn.conflicts = FALSE)

ggplot2::mpg |> 
  duckdb_substrait_compiler() |> 
  filter(hwy > 30) |> 
  arrange(cty) |> 
  collect()
#> # A tibble: 22 × 11
#>    manufacturer model      displ  year   cyl trans drv     cty   hwy fl    class
#>    <chr>        <chr>      <dbl> <int> <int> <chr> <chr> <int> <int> <chr> <chr>
#>  1 audi         a4           2    2008     4 manu… f        20    31 p     comp…
#>  2 hyundai      sonata       2.4  2008     4 manu… f        21    31 r     mids…
#>  3 toyota       camry        2.4  2008     4 manu… f        21    31 r     mids…
#>  4 toyota       camry        2.4  2008     4 auto… f        21    31 r     mids…
#>  5 toyota       camry sol…   2.4  2008     4 manu… f        21    31 r     comp…
#>  6 toyota       camry sol…   2.4  2008     4 auto… f        22    31 r     comp…
#>  7 nissan       altima       2.5  2008     4 auto… f        23    31 r     mids…
#>  8 nissan       altima       2.5  2008     4 manu… f        23    32 r     mids…
#>  9 honda        civic        1.6  1999     4 auto… f        24    32 r     subc…
#> 10 honda        civic        1.6  1999     4 auto… f        24    32 r     subc…
#> # … with 12 more rows

^{Created on 2022-09-08 by the reprex package (v2.0.1)}

[thisisnic]

I agree with your approach here of keeping both substrait_select() and substrait_project(). Just one small comment about variable naming.

[thisisnic]

I think having an argument named .drop_columns and then this variable names drop_columns is a little confusing - mind renaming this?

[paleolimbot]

That's an excellent point! I updated the name and the comments to reflect why they're different variables in the way that I wrote it.