Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
The
self._state_lock
is designed to protect settingself._status
: this change undoes that protection in the case that an error occurs, which introduces a race condition AFAICT.There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Hi Tres, we have covered that in the PR description:
Maybe you can elaborate on the race condition? Thanks
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
TL; DR - Not holding the lock after changing the state to IN_PROGRESS indeed seems safe. However, I would still like to see the effect of this change covered by a test before merging. I can help with this, if needed.
Long reply:
I checked the Batch code, and from what I can tell, that the following threads are relevant:
MonitorBatchPublisher
(if autcommit == True): Triggers_commit()
aftermax_latency
time elapses.CommitBatchPublisher
: If somebody callscommit()
and the batch is in ACCEPTING_MESSAGES state,_commit()
gets called in this thread.publisher.publish()
,batch.publish()
gets called. If that causes an overflow,commit()
is called in theCommitBatchPublisher
thread.The first thing that
_commit()
does after acquiring the lock is changing the batch status to IN_PROGRESS.In that state, any further calls to
publish()
become a no-op, because thewill_accept()
check returnsFalse
for any state that is not ACCEPTING_MESSAGES.Additionally, any other
_commit()
calls also become a no-op, because the batch status is now IN_PROGRESS.Only the thread that changed the state to IN_PROGRESS can proceed, thus shielding the rest of the
_commit()
method with_state_lock
indeed seems unnecessary, which includes the call toself._client.api.publish()
.Without the
_state_lock
held, further calls tobatch.publish()
will not block longer than needed. The batch will simply not accept a new message, returnFalse
, andpublisher.publish()
will create an entirely new batch as a result.