fix ResponseStream when used from other threads

[bkueng]

I'm running into the same issue as #1006: I'm trying to flush a stream from another thread.

This PR adds 3 changes:

do not add NotifyOn::Write when flushing buffers

• if called from the worker thread (doing the epoll) this has no effect
  as it's cleared before going back to epoll
• if called from another thread, it causes an unnecessary wakeup of the
  worker thread. The worker thread then throws an exception
  ("Assertion Error: could not find write data"), as the toWrite data is
  already removed.

In case the flush call (asyncWriteImpl) cannot finish completely due to
EWOULDBLOCK, it still sets NotifyOn::Write.

remove "Assertion Error: could not find write data" assertion

Even with the previous change there is still a case where this can happen:

• the worker thread returns from epoll with a write notification
  (e.g. due to a previous EWOULDBLOCK)
• if at the same time another thread calls flush, which writes all data
• then the worker thread finds an empty toWrite

prevent busy-loop in case of EWOULDBLOCK in asyncWriteImpl

By making sure the loop is exited. As NotifyOn::Write is set, the worker
thread will then be notified when data can be written.

Tested with code instrumentation by manually setting errno = EAGAIN,
and checking that the worker thread then gets woken up and completes the
write.

Fixes: #1006

[Tachi107]

Thanks for the patch! Neatly separated in small, useful commits. Looks good to me :)

[kiplingw]

Thanks @bkueng. Please remember to bump the patch version, per our documentation. Otherwise it looks fine.

[codecov]

Codecov Report

Attention: Patch coverage is 33.33333% with 2 lines in your changes missing coverage. Please review.

Project coverage is 76.28%. Comparing base (0be5c9c) to head (8e60fd4).
Report is 2 commits behind head on master.

Files with missing lines	Patch %	Lines
src/common/transport.cc	33.33%	2 Missing ⚠️

Additional details and impacted files

@@            Coverage Diff             @@
##           master    #1280      +/-   ##
==========================================
- Coverage   77.90%   76.28%   -1.62%     
==========================================
  Files          57       60       +3     
  Lines        7912    10021    +2109     
==========================================
+ Hits         6164     7645    +1481     
- Misses       1748     2376     +628     

☔ View full report in Codecov by Sentry.
📢 Have feedback on the report? Share it here.