[EPIC] Migrate to functions in `datafusion-spark` crate

[andygrove]

What is the problem the feature request solves?

There is a community effort to add Spark-compatible functions in the core DataFusion project in the datafusion-spark crate. See apache/datafusion#15914 for more information.

We want to donate existing Comet functions to this new crate. This epic issue is to track donating our functions and then using them in Comet. In some cases, there may be existing versions of the function already in datafusion-spark, so we will need to review these and see if they would benefit from any improvements based on the Comet versions.

Functions already in datafusion-spark:

• feat(spark): implement spark hash function crc32/sha1 datafusion#17032
• feat(spark): implement Spark string function like/ilike datafusion#16962
• Implement spark array function array datafusion#16936
• feat(spark): implement Spark math function rint datafusion#16924
• feat(spark): Implement Spark string function luhn_check datafusion#16848
• feat(spark): implement Spark datetime function last_day datafusion#16828
• feat: [datafusion-spark] Implement next_day function datafusion#16780
• [datafusion-spark] Implement factorical function datafusion#16125
• datafusion-spark: regexp_extract() datafusion#17037
• feat(spark): implement spark datetime function date_add/date_sub datafusion#17024
• feat(spark): implement Spark bitwise function shiftleft/shiftright/shiftrightunsighed datafusion#17013
• feat(spark): implement Spark conditional function if datafusion#16946
• feat(spark): implement Spark math function bit_get/bit_count datafusion#16942
• Implment Spark map function map datafusion#16940
• Implement Spark url function parse_url datafusion#16937
• feat(spark): implement Spark math function mod/pmod datafusion#16829

Functions to be donated from Comet:

• Aggregates
• avg / avg_decimal
• correlation
• covariance
• stddev
• sum_decimal
• variance
• Array
• array_insert
• array_repeat
• get_array_struct_fields
• list_extract
• Bitwise
• bitwise_count
• bitwise_get
• bitwise_not
• Bloom Filter
• bloom_filter_agg
• bloom_filter_might_contain
• Conditional
• if
• Conversion
• cast (not trivial)
• Date/Time
• date_add
• date_sub
• date_trunc
• extract_date_part
• timestamp_trunc
• JSON
• to_json
• Math
• ceil
• div
• floor
• modulo
• negative
• round
• Non-deterministic
• monotonicall_increasing_id
• rand
• randn
• Predicate
• is_nan
• rlike
• String
• string_space
• substring
• Struct
• create_named_struct
• get_struct_field

Describe the potential solution

No response

Additional context

No response