Support AVRO Format for Write Queries

[devinjdangelo]

Is your feature request related to a problem or challenge?

Writing Avro files is currently not supported in DataFusion, such as via COPYor INSERT INTO queries. This would be a nice feature to have and recently came up in the discord channel.

Describe the solution you'd like

• Implement create_write_physical_plan for AvroFormat
• Implement FileSink on a AvroSink struct
• Implement file writing options / string parsing in FileTypeWriterOptions for avro

I expect some of the common functions in https://github.com/apache/arrow-datafusion/blob/main/datafusion/core/src/datasource/file_format/write.rs can be reused in the avro implementation.

Describe alternatives you've considered

Don't support writing avro files

Additional context

No response

[Veeupup]

@alamb Hi, I can help with this ticket : )

[alamb]

@alamb Hi, I can help with this ticket : )

Thanks @Veeupup -- for this one the first thing we need to do is get an avro writer and the second part is hooking it into DataFusion. There is some work upstream in arrow-rs to make an avro reader/writer in apache/arrow-rs#4886 but I think that project is stilled at the moment