Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Server stopped working when Error in SetExecutionDone #817

Closed
prakashdivyy opened this issue Sep 7, 2020 · 0 comments · Fixed by #851
Closed

Server stopped working when Error in SetExecutionDone #817

prakashdivyy opened this issue Sep 7, 2020 · 0 comments · Fixed by #851
Labels

Comments

@prakashdivyy
Copy link
Contributor

Describe the bug
The leader of the raft will crash when there is error in SetExecutionDone when the job was deleted

To Reproduce
Steps to reproduce the behavior:

  1. Create jobs that concurrent handling 3 jobs at the same time
  2. One of the created job must delete one job from other job created. (example job_a will delete job_b and job_c)

Expected behavior
There is no error after set execution done

Screenshots

time="2020-09-07T13:48:38Z" level=info msg="agent: Calling AgentRun" job_name=some_job_name node="10.0.6.51:6868"
time="2020-09-07T13:48:38Z" level=warning msg="grpc: Received execution done for a deleted job" node=dkron-server-5d9d4d9555-b6vzk
time="2020-09-07T13:48:38Z" level=error msg="store: Error in SetExecutionDone" error="grpc: Received execution done for a deleted job" node=dkron-server-5d9d4d9555-b6vzk
panic: runtime error: invalid memory address or nil pointer dereference
[signal SIGSEGV: segmentation violation code=0x1 addr=0x8 pc=0x2090b2c]
goroutine 76191 [running]:
github.com/distribworks/dkron/v3/dkron.(*GRPCServer).ExecutionDone(0xc001017470, 0x35c8d20, 0xc0019b9890, 0xc0019b98c0, 0x0, 0x0, 0x0)
	/home/runner/work/dkron/dkron/dkron/grpc.go:199 +0xcec
github.com/distribworks/dkron/v3/plugin/types._Dkron_ExecutionDone_Handler(0x29f1780, 0xc001017470, 0x35c8d20, 0xc0019b9890, 0xc001bb4780, 0x0, 0x35c8d20, 0xc0019b9890, 0xc0018d6500, 0x26e)
	/home/runner/work/dkron/dkron/plugin/types/dkron.pb.go:1566 +0x217
google.golang.org/grpc.(*Server).processUnaryRPC(0xc00094b1e0, 0x35ea7e0, 0xc002ba3500, 0xc000d86a00, 0xc001017590, 0x4b50818, 0x0, 0x0, 0x0)
	/home/runner/go/pkg/mod/google.golang.org/grpc@v1.29.1/server.go:1082 +0x50a
google.golang.org/grpc.(*Server).handleStream(0xc00094b1e0, 0x35ea7e0, 0xc002ba3500, 0xc000d86a00, 0x0)
	/home/runner/go/pkg/mod/google.golang.org/grpc@v1.29.1/server.go:1405 +0xccb
google.golang.org/grpc.(*Server).serveStreams.func1.1(0xc0004fcd50, 0xc00094b1e0, 0x35ea7e0, 0xc002ba3500, 0xc000d86a00)
	/home/runner/go/pkg/mod/google.golang.org/grpc@v1.29.1/server.go:746 +0xa1
created by google.golang.org/grpc.(*Server).serveStreams.func1
	/home/runner/go/pkg/mod/google.golang.org/grpc@v1.29.1/server.go:744 +0xa1

Specifications:

  • Docker image v3.0.5 running on kubernetes with 3 server and 3 agent
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
Projects
None yet
Development

Successfully merging a pull request may close this issue.

2 participants