[Event Hubs] Review AMQP link lifecycle for Receiver

[ramya-rao-a]

As per the API proposal in #2718 (comment), we will now have a createReceiver() method on the Event Hub Client that returns a receiver. #3175 tracks the work to implement this change.

This issue is to review the lifecycle of the underlying AMQP link

In the previous design, links never got re-used. At the end of a streaming receiver or batching receiver operation (receive and receiveBatch), the underlying AMQP link was closed. In the new design

• We need to share the AMQP receiver link between receive, receiveBatch and getEventIterator such that I should be able to use say receiveBatch and then getEventIterator or receive and the second call should start receiving messages from where the first left off
• Ensure multiple receivers can be created and work as expected in parallel

[ramya-rao-a]

Below are the break up of tasks:

• When one of the 3 receive operations is in progress, making a call to another in the same receiver should throw error [Event Hubs] Multiple receive operations on the same receiver #3309
• Iterator should call receiveBatch() for the next batch while user iterates on the results of the current batch. And so, getEventIterator() and receiveBatch() will use the same AMQP link [Event Hubs] Iterator for receiver should receive events in the background #3308
  - The streaming receiver will use a different AMQP link than the batching ones. We will try to combine these 2 at a later date.
• If user calls any of the 3 receive operations after the previous one completes, the new receive operation should continue from where the previous left off. [Event Hubs] Multiple receive operations on the same receiver #3309
• Closing of the receiver should result in closing of the underlying AMQP links (batching and streaming). Once receiver is closed, further operations should throw error [Event Hubs] Ensure receiver is functional #3175
• Both batching and streaming receiver should retry establishing the AMQP receiver link in the below cases. The re-established link should start from the checkpoint i.e the last event provided to the user [Event Hubs] Ensure both batching and streaming receivers are able to re-establish links on retryable errors #3310
  • retryable receiver error
  • retryable session error
  • retryable connection error
• Retry options set at receiver level [Event Hubs] Usage of retry options on receiver  #3311
  • The work to re-establish the AMQP receiver link is already covered in [Event Hubs] Ensure both batching and streaming receivers are able to re-establish links on retryable errors #3310. This is an internal feature and the user provided retry options will not be touching this.
  • The receiveBatch() should respect the retry options. This can be done inside the receiveBatch() function by using the retry function directly on the receive() operation of the batching receiver
    - The iterator model doesnt need to use these options directly as all it does is call receiveBatch() which will respect the retry options
    - The streaming receiver is meant to be a long running operation and [Event Hubs] Ensure both batching and streaming receivers are able to re-establish links on retryable errors #3310 will ensure the link is always alive. So user provided retry options dont apply here
• timeout set at receiver level is usable by all 3 receive operations [Event Hubs] Usage of timeout option on the receiver #3313
  • For the iterator, the timeout is for each iteration, so it simply passes the value to receiveBatch() call
  • For streaming receiver, there is no default timeout. Use the one from receiver options only if explicitly provided by the user.
• Stretch Goal: When the streaming receive() is called after either of the other 2 receive operations, re-use the AMQP receiver link used by the batching receiver. Doing the other way around is not possible because the streaming receiver would have moved much farther up in the stream than the last event processed by the user. [EventHubs] Revisit the design of streaming and batching receivers #3320

[ramya-rao-a]

Closing this issue as separate github issues have been created to track the individual tasks