Improve Test Reliability by Replacing ZIO.sleep with Synchronization Primitives in ConsumerSpec

[AdrielC]

Follow up on #1389

Description:

This pull request improves the reliability of tests in ConsumerSpec.scala by replacing the use of ZIO.sleep with synchronization primitives such as Promise and Ref. The previous implementation relied on arbitrary sleep durations to wait for certain conditions, which could lead to non-deterministic behavior and flakiness in the tests.

1. Test: “handle rebalancing by completing topic-partition streams”

• Issue: The test used ZIO.sleep to wait before starting the second consumer, which could cause timing-related flakiness.
• Solution: Introduced a Ref[Set[Int]] to track the partitions assigned to the first consumer and a Promise[Nothing, Unit] to signal when the consumer has been assigned at least half of the partitions, replacing the ZIO.sleep.
• Result: The second consumer now starts only after the first consumer has been assigned the required number of partitions.

2. Test: “produce diagnostic events when rebalancing”

• Issue: Similar to the first test, this one used ZIO.sleep to delay the start of the second consumer, leading to potential flakiness.
• Solution: Applied the same approach as in the previous test by using a Ref[Set[Int]] and a Promise[Nothing, Unit] to synchronize the start of the second consumer.
• Result: The second consumer starts deterministically after the first consumer is ready.

3. Test: “it’s possible to start a new consumption session from a Consumer that had a consumption session stopped previously”

• Issue: The test used ZIO.sleep to wait before stopping the consumer.
• Solution: Replaced the sleep with a Ref[Int] to count the number of messages consumed and a Promise[Nothing, Unit] to signal when to stop consumption.
• Result: The consumer is now stopped after consuming a specified number of messages.

[erikvanoosten]

Awesome stuff. Thanks @AdrielC .
Let's wait until #1400 is merged, and then merge this PR.