REST distribution get nil pointer exception

[sabau]

Summary of Bug

We get:
{"error":"ABCIQuery: Response error: RPC error -32603 - Internal error: runtime error: invalid memory address or nil pointer dereference"}

https://stargate.cosmos.network/distribution/delegators/cosmos167w96tdvmazakdwkw2u57227eduula2cy572lf/rewards/cosmosvaloper1qwl879nx9t6kef4supyazayf7vjhennyh568ys

https://stargate.cosmos.network/distribution/validators/cosmosvaloper1rcr8c8enfl5rdfv8jcxj8uwy64dmr7md2mnl08

I'm not sure we want to return an exception here and error code should be 404 instead of 500.
Probably the address combination does not match any reward in the first case, while in the second case that address is not a validator, seems both are not present resources

Seems related to:
#3704
#2466

on the other hand on CLI it seems to be handled:

[ec2-user@ip-xxx ~]$ ./gaiacli query distr commission cosmosvaloper1rcr8c8enfl5rdfv8jcxj8uwy64dmr7md2mnl08 --home . --trust-node

[ec2-user@ip-xxx ~]$ 

---

For Admin Use

• Not duplicate issue
• Appropriate labels applied
• Appropriate contributors tagged
• Contributor assigned/self-assigned

[alexanderbez]

Great find @sabau!

[alexanderbez]

@sabau any chance you can share the exact sdk version and possibly a stack trace?

[sabau]

It's the one deployed by @mircea-c, should be 0.33.0 right?
To check the CLI I tried to replicate the same issue with the full node at the same version

[alexanderbez]

It would be great to get a stracktrace from the node though. I took a look at the implementation and I don't really see where a nil pointer exception could be thrown (I may have missed it) from the client to the querier.

[mircea-c]

This is what I could find on the node. There's one more instance of this same issue today.

Apr 03 09:32:16 ip-10-50-1-229.ec2.internal gaiad[24632]: E[2019-04-03|09:32:16.086] Panic in RPC HTTP handler   module=rpc-server err="runtime error: invalid memory address or nil pointer dereference" 
stack="goroutine 239936140 [running]:
runtime/debug.Stack(0xc006afff58, 0xf48460, 0x1dd4750)
    /usr/local/go/src/runtime/debug/stack.go:24 +0x9d\ngithub.com/cosmos/cosmos-sdk/vendor/github.com/tendermint/tendermint/rpc/lib/server.RecoverAndLogHandler.func1.1(0xc007fe4540, 0x13700c0, 0xc002d66fa0, 0xbf213d48051b2e65, 0x2946b1dbe83a2, 0x1e314e0, 0xc003af2700)
    /home/centos/go/src/github.com/cosmos/cosmos-sdk/vendor/github.com/tendermint/tendermint/rpc/lib/server/http_server.go:133 +0x552\npanic(0xf48460, 0x1dd4750)
    /usr/local/go/src/runtime/panic.go:522 +0x1b5\ngithub.com/cosmos/cosmos-sdk/x/distribution/keeper.Keeper.calculateDelegationRewards(0x1361640, 0xc0008834f0, 0xc00017c070, 0xc00017c070, 0x1361640, 0xc000883540, 0x1361680, 0xc000883550, 0xc0000cd4e0, 0x5, ...)
    /home/centos/go/src/github.com/cosmos/cosmos-sdk/x/distribution/keeper/delegation.go:57 +0x51\ngithub.com/cosmos/cosmos-sdk/x/distribution/keeper.queryDelegationRewards(0x136f5c0, 0xc00bc5b7a0, 0xc00bfb3080, 0xd, 0xc00bc5ae90, 0x0, 0x0, 0xc0053b4ea0, 0x90, 0x120, ...)
    /home/centos/go/src/github.com/cosmos/cosmos-sdk/x/distribution/keeper/querier.go:208 +0x35d\ngithub.com/cosmos/cosmos-sdk/x/distribution/keeper.NewQuerier.func1(0x136f5c0, 0xc00bc5b7a0, 0xc00bfb3080, 0xd, 0xc00bc5ae90, 0x1, 0x1, 0xc0053b4ea0, 0x90, 0x120, ...)
    /home/centos/go/src/github.com/cosmos/cosmos-sdk/x/distribution/keeper/querier.go:46 +0x4f4\ngithub.com/cosmos/cosmos-sdk/baseapp.handleQueryCustom(0xc000ab3100, 0xc00bc5ae70, 0x3, 0x3, 0xc0053b4ea0, 0x90, 0x120, 0xc00247c0e0, 0x1f, 0x0, .../home/centos/go/src/github.com/cosmos/cosmos-sdk/baseapp/baseapp.go:498 +0x46a\ngithub.com/cosmos/cosmos-sdk/baseapp.(*BaseApp).Query(0xc000ab3100, 0xc0053b4ea0, 0x90, 0x120, 0xc00247c0e0, 0x1f, 0x0, 0x0, 0x0, 0x0, ...)
    /home/centos/go/src/github.com/cosmos/cosmos-sdk/baseapp/baseapp.go:396 +0x2e9\ngithub.com/cosmos/cosmos-sdk/vendor/github.com/tendermint/tendermint/abci/client.(*localClient).QuerySync(0xc000032de0, 0xc0053b4ea0, 0x90, 0x120, 0xc00247c0e0, 0x1f, 0x0, 0x0, 0x0, 0x0, ...)
    /home/centos/go/src/github.com/cosmos/cosmos-sdk/vendor/github.com/tendermint/tendermint/abci/client/local_client.go:207 +0xf3\ngithub.com/cosmos/cosmos-sdk/vendor/github.com/tendermint/tendermint/proxy.(*appConnQuery).QuerySync(0xc000974290, 0xc0053b4ea0, 0x90, 0x120, 0xc00247c0e0, 0x1f, 0x0, 0x0, 0x0, 0x0, ...)
    /home/centos/go/src/github.com/cosmos/cosmos-sdk/vendor/github.com/tendermint/tendermint/proxy/app_conn.go:143 +0x6d\ngithub.com/cosmos/cosmos-sdk/vendor/github.com/tendermint/tendermint/rpc/core.ABCIQuery(0xc00247c0e0, 0x1f, 0xc0053b4ea0, 0x90, 0x120, 0x0, 0x0, 0x0, 0x0, 0x0)
    /home/centos/go/src/github.com/cosmos/cosmos-sdk/vendor/github.com/tendermint/tendermint/rpc/core/abci.go:56 +0x114\nreflect.Value.call(0xf55920, 0x11f5788, 0x13, 0x107eabb, 0x4, 0xc007d01ce0, 0x4, 0x4, 0xc000483368, 0x0, ...)
    /usr/local/go/src/reflect/value.go:447 +0x461\nreflect.Value.Call(0xf55920, 0x11f5788, 0x13, 0xc007d01ce0, 0x4, 0x4, 0xc007d01ce0, 0x4, 0x4)
    /usr/local/go/src/reflect/value.go:308 +0xa4\ngithub.com/cosmos/cosmos-sdk/vendor/github.com/tendermint/tendermint/rpc/lib/server.makeJSONRPCHandler.func1(0x136a740, 0xc007fe4540, 0xc003af2700)
    /home/centos/go/src/github.com/cosmos/cosmos-sdk/vendor/github.com/tendermint/tendermint/rpc/lib/server/handlers.go:145 +0x407\ngithub.com/cosmos/cosmos-sdk/vendor/github.com/tendermint/tendermint/rpc/lib/server.handleInvalidJSONRPCPaths.func1(0x136a740, 0xc007fe4540, 0xc003af2700)
    /home/centos/go/src/github.com/cosmos/cosmos-sdk/vendor/github.com/tendermint/tendermint/rpc/lib/server/handlers.go:165 +0x9c\nnet/http.HandlerFunc.ServeHTTP(0xc0027f9810, 0x136a740, 0xc007fe4540, 0xc003af2700)
    /usr/local/go/src/net/http/server.go:1995 +0x44\nnet/http.(*ServeMux).ServeHTTP(0xc002d7c480, 0x136a740, 0xc007fe4540, 0xc003af2700)
    /usr/local/go/src/net/http/server.go:2375 +0x1d6\ngithub.com/cosmos/cosmos-sdk/vendor/github.com/tendermint/tendermint/rpc/lib/server.maxBytesHandler.ServeHTTP(0x1353de0, 0xc002d7c480, 0xf4240, 0x136a740, 0xc007fe4540, 0xc003af2700)
    /home/centos/go/src/github.com/cosmos/cosmos-sdk/vendor/github.com/tendermint/tendermint/rpc/lib/server/http_server.go:178 +0xd0\ngithub.com/cosmos/cosmos-sdk/vendor/github.com/tendermint/tendermint/rpc/lib/server.RecoverAndLogHandler.func1(0x136aec0, 0xc0097882a0, 0xc003af2700)
    /home/centos/go/src/github.com/cosmos/cosmos-sdk/vendor/github.com/tendermint/tendermint/rpc/lib/server/http_server.go:151 +0x317\nnet/http.HandlerFunc.ServeHTTP(0xc002dac780, 0x136aec0, 0xc0097882a0, 0xc003af2700)
    /usr/local/go/src/net/http/server.go:1995 +0x44\nnet/http.serverHandler.ServeHTTP(0xc000b08a90, 0x136aec0, 0xc0097882a0, 0xc003af2700)
    /usr/local/go/src/net/http/server.go:2774 +0xa8\nnet/http.(*conn).serve(0xc00387b400, 0x136f500, 0xc0075492c0)
    /usr/local/go/src/net/http/server.go:1878 +0x851\ncreated by net/http.(*Server).Serve
    /usr/local/go/src/net/http/server.go:2884 +0x2f4\n"

[alexanderbez]

Very helpful indeed. Ok, I see the problem -- pretty trivial fix:

In https://github.com/cosmos/cosmos-sdk/blob/v0.33.2/x/distribution/keeper/querier.go#L195-L216, the validation and/or delegation may not exist. In such a case, we should simply return an error.

[alessio]

Is this done?

[alexanderbez]

Yes. Not sure why it wasn't closed automatically?