[WIP] Issues with seqNo and Eventhub Retention

[Fokko]

It feels like we're having some issues with the retention of the Event Hub.
When starting a Spark job that reads from a consumer group that already has some pruned messages
because then run out of retention.

java.lang.IllegalStateException: In partition 1 of eventhubdev, with consumer group consumergroup-1,
request seqNo 1921738 is less than the received seqNo 1924294.
The earliest seqNo is 1924293 and the last seqNo is 1944993

First I need to fix another issue with the Simulator, which I'll do in another PR to keep everything nicely separated. Currently, the latest seqNo is implemented by taking the size() of the messages within a certain partition. This is not right, we want to take the max of the seqNo of the actual messages:

azure-event-hubs-spark/core/src/main/scala/org/apache/spark/eventhubs/utils/SimulatedEventHubs.scala

Line 252 in 61aba38

data.size

Thanks for contributing! We appreciate it :)

For a Pull Request to be accepted, you must:

• Run scalafmt on your code using the .scalafmt.conf present in this project
• All tests must pass when you run mvn clean test

Just in case, here are some tips that could prove useful when opening a pull request:

• Read the Contributor's Guide
• Make the title of pull request is clear and informative.
• There should be a small number of commits, all with informative messages.
• The pull request shouldn't introduce any breaking changes (unless will occur on the next release)
• Any public code should be properly documented
• Be sure to write tests for any changes in the pull request
• The code should build without any errors

[Fokko]

Please merge #409 first

[sabeegrewal]

@Fokko sorry for the hold up, #409 is merged. Can you rebase when you get the chance? Also, is this ready for review?

as always, thanks for taking the time.

[sabeegrewal]

Hey, I'm going to close this for now. If you continue to have issues after the upcoming release, then we should revisit this PR.