Skip to content

Commit

Permalink
fix(ServeTask): Return error if server is not ready (#6019) (#6020)
Browse files Browse the repository at this point in the history
The grpcWorker.ServeTask function is called when a request is received
over the network (sent by a peer). This function doesn't consider the
health of the node before starting the processing of the request. If we
process a request before the node is ready, we might end up seeing
crashes similar to https://dgraph.atlassian.net/browse/DGRAPH-1934

Fixes DGRAPH-1934

This bug was seen when running jepsen bank test with
kill-alpha and kill-zero nemesis.

(cherry picked from commit 9293c97)
  • Loading branch information
Ibrahim Jarif authored Jul 16, 2020
1 parent 10325de commit ab413d1
Showing 1 changed file with 6 additions and 0 deletions.
6 changes: 6 additions & 0 deletions worker/task.go
Original file line number Diff line number Diff line change
Expand Up @@ -1854,6 +1854,12 @@ func (w *grpcWorker) ServeTask(ctx context.Context, q *pb.Query) (*pb.Result, er
return nil, ctx.Err()
}

// It could be possible that the server isn't ready but a peer sends a
// request. In that case we should check for the health here.
if err := x.HealthCheck(); err != nil {
return nil, err
}

gid, err := groups().BelongsToReadOnly(q.Attr, q.ReadTs)
switch {
case err != nil:
Expand Down

0 comments on commit ab413d1

Please sign in to comment.