Dynamic EventPage Rechunking

[cjtitus]

Summary

Added try/except to Publisher to handle case where EventPages are too large, and dynamically re-chunk them.

Motivation

When flyscanning, we can produce quite large EventPages of data that cause the Kafka producer to choke. Fortunately, an EventPage can be re-chunked quite easily (once bluesky/event-model#297 is merged), so we should take advantage of this to produce documents that Kafka can handle.

Details

I have added a try/except to the Publisher to catch the KafkaException indicating that a message is too large. If the message is too large, and the document was an EventPage, we should try re-chunking the document to half its size, and re-sending.

I have made this process recursive, so that the page size will be halved until the messages are small enough. Hopefully this only needs to happen once or twice, but with fly-scans, quite large EventPages can be generated.

Testing

I have tested this by publishing quite large test pages, which successfully publish. All existing tests pass. Possibly more tests could be implemented.

[mrakitin]

Looks reasonable to me. Thanks, @cjtitus!

[cjtitus]

@tacaswell does everything look good now that I added a check for infinite recursion? I'd like to get this merged sooner rather than later, as we want to really begin serious fly-scanning next cycle.