Option to Mark Fields as Mandatory

[albp-rmt]

Firstly a quick thanks. After struggling with the SMTs currently on offer.. JSLT for Kafka Connectors is the silver bullet addressing most of our needs. All except for this one remaining issue; all fields produced by the transform are set as optional in the Schema Registry (we're using AVRO).

Ideally the optional state of fields would be configurable and would work something like Confluent's InsertField SMT which lets you mark fields as mandatory by suffixing the field name with a ! - see InsertField.

[fuchsst]

yes. in general maintaining a schema is an issue with this approach (of using the JSLT library)
the problem is, that the JSLT library internally works with Jackson, which basically means it only has the datatypes of JSON, which are very generic. we use this transformer to send data to HTTP REST endpoints, which only work with JSON anyway.

but whenever you need a proper schema (like Avro, written to Kafka) this becomes an issue. it will still work well in 90% of the cases (just picking a random number to indicate the relevance), but there will be practical cases where the loss of precision (e.g. float vs. double or int vs long or proper date/time types) will be an issue - the transformer tries to determine the correct type based on the output values of the fields. the only proper way to ensure field types would be a rewrite of the core of the JSLT library (as discussed here) and provide interfaces where e.g. the Kafka Connect Struct is a possible alternative input

non-optional fields in the schema go in the same direction. for that specific requirement i'd rather say it's better to have a second transformer that does exactly what you described: alter the schema and set the required-flag of defined fields. i'd not make it part of this transformer
i think this repo can serve as a basis. the accompanying blog post has some more links to relevant resources on how to write a custom transformer: https://tech.willhaben.at/kafka-connect-custom-single-message-transform-using-jslt-2fc57ae98395