UDF

UDF API specification

A(BCD)EF:

• A (xor)
  • udf to just create a (vararg) UDF
  • udaf to just create a UDAF
  • udafUnnamed to create an unnamed UDAF (for when udaf fails to get a name from the Aggregator)
  • udf.register to immediately create and register a (vararg) UDF or UDAF. Requires B or requires C to be a NamedUserDefinedFunction, an Aggregator, or a (vararg) UDF / UDAF function reference.
• B
  • name. A String. Optional unless registering an unnamed (vararg) UDF or UDAF. Provides (new) name for the (vararg) UDF or UDAF.
• C (xor with E)
  • namedUdf. A NamedUserDefinedFunction(Vararg), requires A to be udaf.register (will provide name).
  • udf. A UserDefinedFunction(Vararg), requires A to be udaf.register and requires B (will provide name).
  • func. A UDF- or UDAF function reference (will provide name based on func).
  • varargFunc. A vararg UDF function reference. Requires A to be NOT udf (will provide name based on varargFunc).
  • agg. An Aggregator (can be created using aggregatorOf()) to create a UDAF. Requires A to be NOT udf (will provide name based on agg).
  • zero, reduce, merge, finish, Optional bufferEncoder, Optional outputEncoder. Creates UDAF in place instead of using agg (will provide name, "Aggregator" by default).
• D
  • nondeterministic. A Boolean. Optional, defaults to false. Updates (vararg) UDF or UDAF to non-deterministic. For instance, for randomization.
• E (xor with C)
  • func. An in-place UDF lambda (won't provide name, unless specified in B or F).
  • varargFunc. An in-place vararg UDF lambda (= takes a single array as parameter) (won't provide name, unless specified in B or F).
• F (Optional, or)
  • .register(). Will register the (vararg) UDF or UDAF. Requires B or requires C to be a NamedUserDefinedFunction, an Aggregator, or a (vararg) UDF / UDAF function reference.
  • .register(name: String). Will register the (vararg) UDF or UDAF with (new) name.
  • .withName(name: String). Will give the (named) udf the (new) name name.

Example

See the examples in code