If the last reply was from the author, the ball is in the reviewers(') court, and vice versa.
Optional: make it per-person. There is always only one author, but there can be several reviewers. If one reviewer makes a comment, the "attention list" becomes the previous attention list minus that one reviewer plus the author.