[Feature Request]: enable setting max_writer_per_bundle for avroIO and other IO #29729
Description
What would you like to happen?
https://github.com/apache/beam/blob/master/sdks/java/core/src/main/java/org/apache/beam/sdk/io/WriteFiles.java#L166
Not able to customize max_writer_per_bundle for avroIO .
When we have multiple pcollection or partitioned pcollection , its not possible to open avro sink for every partition (specially when paritions are large , for example > 32 ) . It leads to OOM
Issue Priority
Priority: 3 (nice-to-have improvement)
Issue Components
- Component: Python SDK
- Component: Java SDK
- Component: Go SDK
- Component: Typescript SDK
- Component: IO connector
- Component: Beam YAML
- Component: Beam examples
- Component: Beam playground
- Component: Beam katas
- Component: Website
- Component: Spark Runner
- Component: Flink Runner
- Component: Samza Runner
- Component: Twister2 Runner
- Component: Hazelcast Jet Runner
- Component: Google Cloud Dataflow Runner