Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

JSON-RPC Errors with multiple reporting commands since activating M1. #6869

Closed
4 tasks done
TippyFlitsUK opened this issue Jul 25, 2021 · 11 comments
Closed
4 tasks done

JSON-RPC Errors with multiple reporting commands since activating M1. #6869

TippyFlitsUK opened this issue Jul 25, 2021 · 11 comments
Assignees
@nonsense
Labels
area/markets/provider area/markets/storage kind/bug Kind: Bug P1 P1: Must be resolved
Milestone
1.11.1

Comments

@TippyFlitsUK
Copy link
Contributor

TippyFlitsUK commented Jul 25, 2021 •
edited
Loading

Checklist

  • This is not a question or a support request. If you have any lotus related questions, please ask in the lotus forum.
  • I am reporting a bug w.r.t one of the M1 tags. If not, choose another issue option here.
  • I am not reporting a bug around deal making. If yes, create a M1 Bug Report For Deal Making.
  • I have searched on the issue tracker and the lotus forum, and there is no existing related issue or discussion.

Lotus component

lotus JSON-RPC API

Lotus Tag and Version

Daemon:  1.11.1-dev+mainnet+git.48e8f54c6+api1.2.0
Local: lotus-miner version 1.11.1-dev+mainnet+git.48e8f54c6

Describe the Bug

I have both separate instances of miner and markets instances running but am getting errors with all the storage deal reporting tools.

There are multiple commands that no longer work since the separation although the miner does appear to be running without issue. I have completed window posts and sealed sectors successfully on this release. I am unable to see any information with regards to deals and data transfers as all commands are broken.

Logging Information

lotus-miner storage-deals list

2021-07-25T18:10:21.959+0100	ERROR	rpc	go-jsonrpc@v0.1.4-0.20210217175800-45ea43ac2bec/handler.go:158	panic in rpc method 'Filecoin.MarketListIncompleteDeals': runtime error: invalid memory address or nil pointer dereference
github.com/filecoin-project/go-jsonrpc.doCall.func1
	/home/tippyflits/go/pkg/mod/github.com/filecoin-project/go-jsonrpc@v0.1.4-0.20210217175800-45ea43ac2bec/handler.go:158
runtime.gopanic
	/usr/local/go/src/runtime/panic.go:965
runtime.panicmem
	/usr/local/go/src/runtime/panic.go:212
runtime.sigpanic
	/usr/local/go/src/runtime/signal_unix.go:734
github.com/filecoin-project/lotus/node/impl.(*StorageMinerAPI).MarketListIncompleteDeals
	/home/tippyflits/lotus/node/impl/storminer.go:460
reflect.Value.call
	/usr/local/go/src/reflect/value.go:476
reflect.Value.Call
	/usr/local/go/src/reflect/value.go:337
github.com/filecoin-project/lotus/metrics.proxy.func1
	/home/tippyflits/lotus/metrics/proxy.go:60
github.com/filecoin-project/lotus/api.(*StorageMinerStruct).MarketListIncompleteDeals
	/home/tippyflits/lotus/api/proxy_gen.go:2963
reflect.Value.call
	/usr/local/go/src/reflect/value.go:476
reflect.Value.Call
	/usr/local/go/src/reflect/value.go:337
github.com/filecoin-project/go-jsonrpc/auth.PermissionedProxy.func1
	/home/tippyflits/go/pkg/mod/github.com/filecoin-project/go-jsonrpc@v0.1.4-0.20210217175800-45ea43ac2bec/auth/auth.go:62
github.com/filecoin-project/lotus/api.(*StorageMinerStruct).MarketListIncompleteDeals
	/home/tippyflits/lotus/api/proxy_gen.go:2963
reflect.Value.call
	/usr/local/go/src/reflect/value.go:476
reflect.Value.Call
	/usr/local/go/src/reflect/value.go:337
github.com/filecoin-project/go-jsonrpc.doCall
	/home/tippyflits/go/pkg/mod/github.com/filecoin-project/go-jsonrpc@v0.1.4-0.20210217175800-45ea43ac2bec/handler.go:162
github.com/filecoin-project/go-jsonrpc.(*RPCServer).handle
	/home/tippyflits/go/pkg/mod/github.com/filecoin-project/go-jsonrpc@v0.1.4-0.20210217175800-45ea43ac2bec/handler.go:259
2021-07-25T18:10:21.962+0100	ERROR	rpc	go-jsonrpc@v0.1.4-0.20210217175800-45ea43ac2bec/server.go:95	RPC Error: fatal error calling 'Filecoin.MarketListIncompleteDeals': panic in rpc method 'Filecoin.MarketListIncompleteDeals': runtime error: invalid memory address or nil pointer dereference
2021-07-25T18:10:21.962+0100	WARN	rpc	go-jsonrpc@v0.1.4-0.20210217175800-45ea43ac2bec/server.go:101	rpc error: fatal error calling 'Filecoin.MarketListIncompleteDeals': panic in rpc method 'Filecoin.MarketListIncompleteDeals': runtime error: invalid memory address or nil pointer dereference

LOTUS_MINER_PATH=/mnt/nvme0n1/.lotusmarket lotus-miner storage-deals list

2021-07-25T18:12:23.187+0100	ERROR	rpc	go-jsonrpc@v0.1.4-0.20210217175800-45ea43ac2bec/handler.go:158	panic in rpc method 'Filecoin.MarketListIncompleteDeals': runtime error: invalid memory address or nil pointer dereference
github.com/filecoin-project/go-jsonrpc.doCall.func1
	/home/tippyflits/go/pkg/mod/github.com/filecoin-project/go-jsonrpc@v0.1.4-0.20210217175800-45ea43ac2bec/handler.go:158
runtime.gopanic
	/usr/local/go/src/runtime/panic.go:965
runtime.panicmem
	/usr/local/go/src/runtime/panic.go:212
runtime.sigpanic
	/usr/local/go/src/runtime/signal_unix.go:734
github.com/filecoin-project/lotus/node/impl.(*StorageMinerAPI).MarketListIncompleteDeals
	/home/tippyflits/lotus/node/impl/storminer.go:460
reflect.Value.call
	/usr/local/go/src/reflect/value.go:476
reflect.Value.Call
	/usr/local/go/src/reflect/value.go:337
github.com/filecoin-project/lotus/metrics.proxy.func1
	/home/tippyflits/lotus/metrics/proxy.go:60
github.com/filecoin-project/lotus/api.(*StorageMinerStruct).MarketListIncompleteDeals
	/home/tippyflits/lotus/api/proxy_gen.go:2963
reflect.Value.call
	/usr/local/go/src/reflect/value.go:476
reflect.Value.Call
	/usr/local/go/src/reflect/value.go:337
github.com/filecoin-project/go-jsonrpc/auth.PermissionedProxy.func1
	/home/tippyflits/go/pkg/mod/github.com/filecoin-project/go-jsonrpc@v0.1.4-0.20210217175800-45ea43ac2bec/auth/auth.go:62
github.com/filecoin-project/lotus/api.(*StorageMinerStruct).MarketListIncompleteDeals
	/home/tippyflits/lotus/api/proxy_gen.go:2963
reflect.Value.call
	/usr/local/go/src/reflect/value.go:476
reflect.Value.Call
	/usr/local/go/src/reflect/value.go:337
github.com/filecoin-project/go-jsonrpc.doCall
	/home/tippyflits/go/pkg/mod/github.com/filecoin-project/go-jsonrpc@v0.1.4-0.20210217175800-45ea43ac2bec/handler.go:162
github.com/filecoin-project/go-jsonrpc.(*RPCServer).handle
	/home/tippyflits/go/pkg/mod/github.com/filecoin-project/go-jsonrpc@v0.1.4-0.20210217175800-45ea43ac2bec/handler.go:259
2021-07-25T18:12:23.188+0100	ERROR	rpc	go-jsonrpc@v0.1.4-0.20210217175800-45ea43ac2bec/server.go:95	RPC Error: fatal error calling 'Filecoin.MarketListIncompleteDeals': panic in rpc method 'Filecoin.MarketListIncompleteDeals': runtime error: invalid memory address or nil pointer dereference
2021-07-25T18:12:23.188+0100	WARN	rpc	go-jsonrpc@v0.1.4-0.20210217175800-45ea43ac2bec/server.go:101	rpc error: fatal error calling 'Filecoin.MarketListIncompleteDeals': panic in rpc method 'Filecoin.MarketListIncompleteDeals': runtime error: invalid memory address or nil pointer dereference

lotus-miner info

2021-07-25T18:14:38.706+0100	ERROR	rpc	go-jsonrpc@v0.1.4-0.20210217175800-45ea43ac2bec/handler.go:158	panic in rpc method 'Filecoin.MarketListIncompleteDeals': runtime error: invalid memory address or nil pointer dereference
github.com/filecoin-project/go-jsonrpc.doCall.func1
	/home/tippyflits/go/pkg/mod/github.com/filecoin-project/go-jsonrpc@v0.1.4-0.20210217175800-45ea43ac2bec/handler.go:158
runtime.gopanic
	/usr/local/go/src/runtime/panic.go:965
runtime.panicmem
	/usr/local/go/src/runtime/panic.go:212
runtime.sigpanic
	/usr/local/go/src/runtime/signal_unix.go:734
github.com/filecoin-project/lotus/node/impl.(*StorageMinerAPI).MarketListIncompleteDeals
	/home/tippyflits/lotus/node/impl/storminer.go:460
reflect.Value.call
	/usr/local/go/src/reflect/value.go:476
reflect.Value.Call
	/usr/local/go/src/reflect/value.go:337
github.com/filecoin-project/lotus/metrics.proxy.func1
	/home/tippyflits/lotus/metrics/proxy.go:60
github.com/filecoin-project/lotus/api.(*StorageMinerStruct).MarketListIncompleteDeals
	/home/tippyflits/lotus/api/proxy_gen.go:2963
reflect.Value.call
	/usr/local/go/src/reflect/value.go:476
reflect.Value.Call
	/usr/local/go/src/reflect/value.go:337
github.com/filecoin-project/go-jsonrpc/auth.PermissionedProxy.func1
	/home/tippyflits/go/pkg/mod/github.com/filecoin-project/go-jsonrpc@v0.1.4-0.20210217175800-45ea43ac2bec/auth/auth.go:62
github.com/filecoin-project/lotus/api.(*StorageMinerStruct).MarketListIncompleteDeals
	/home/tippyflits/lotus/api/proxy_gen.go:2963
reflect.Value.call
	/usr/local/go/src/reflect/value.go:476
reflect.Value.Call
	/usr/local/go/src/reflect/value.go:337
github.com/filecoin-project/go-jsonrpc.doCall
	/home/tippyflits/go/pkg/mod/github.com/filecoin-project/go-jsonrpc@v0.1.4-0.20210217175800-45ea43ac2bec/handler.go:162
github.com/filecoin-project/go-jsonrpc.(*RPCServer).handle
	/home/tippyflits/go/pkg/mod/github.com/filecoin-project/go-jsonrpc@v0.1.4-0.20210217175800-45ea43ac2bec/handler.go:259
2021-07-25T18:14:38.706+0100	ERROR	rpc	go-jsonrpc@v0.1.4-0.20210217175800-45ea43ac2bec/server.go:95	RPC Error: fatal error calling 'Filecoin.MarketListIncompleteDeals': panic in rpc method 'Filecoin.MarketListIncompleteDeals': runtime error: invalid memory address or nil pointer dereference
2021-07-25T18:14:38.706+0100	WARN	rpc	go-jsonrpc@v0.1.4-0.20210217175800-45ea43ac2bec/server.go:101	rpc error: fatal error calling 'Filecoin.MarketListIncompleteDeals': panic in rpc method 'Filecoin.MarketListIncompleteDeals': runtime error: invalid memory address or nil pointer dereference

LOTUS_MINER_PATH=/mnt/nvme0n1/.lotusmarket lotus-miner info

2021-07-25T18:15:48.040+0100	ERROR	rpc	go-jsonrpc@v0.1.4-0.20210217175800-45ea43ac2bec/handler.go:158	panic in rpc method 'Filecoin.MarketListIncompleteDeals': runtime error: invalid memory address or nil pointer dereference
github.com/filecoin-project/go-jsonrpc.doCall.func1
	/home/tippyflits/go/pkg/mod/github.com/filecoin-project/go-jsonrpc@v0.1.4-0.20210217175800-45ea43ac2bec/handler.go:158
runtime.gopanic
	/usr/local/go/src/runtime/panic.go:965
runtime.panicmem
	/usr/local/go/src/runtime/panic.go:212
runtime.sigpanic
	/usr/local/go/src/runtime/signal_unix.go:734
github.com/filecoin-project/lotus/node/impl.(*StorageMinerAPI).MarketListIncompleteDeals
	/home/tippyflits/lotus/node/impl/storminer.go:460
reflect.Value.call
	/usr/local/go/src/reflect/value.go:476
reflect.Value.Call
	/usr/local/go/src/reflect/value.go:337
github.com/filecoin-project/lotus/metrics.proxy.func1
	/home/tippyflits/lotus/metrics/proxy.go:60
github.com/filecoin-project/lotus/api.(*StorageMinerStruct).MarketListIncompleteDeals
	/home/tippyflits/lotus/api/proxy_gen.go:2963
reflect.Value.call
	/usr/local/go/src/reflect/value.go:476
reflect.Value.Call
	/usr/local/go/src/reflect/value.go:337
github.com/filecoin-project/go-jsonrpc/auth.PermissionedProxy.func1
	/home/tippyflits/go/pkg/mod/github.com/filecoin-project/go-jsonrpc@v0.1.4-0.20210217175800-45ea43ac2bec/auth/auth.go:62
github.com/filecoin-project/lotus/api.(*StorageMinerStruct).MarketListIncompleteDeals
	/home/tippyflits/lotus/api/proxy_gen.go:2963
reflect.Value.call
	/usr/local/go/src/reflect/value.go:476
reflect.Value.Call
	/usr/local/go/src/reflect/value.go:337
github.com/filecoin-project/go-jsonrpc.doCall
	/home/tippyflits/go/pkg/mod/github.com/filecoin-project/go-jsonrpc@v0.1.4-0.20210217175800-45ea43ac2bec/handler.go:162
github.com/filecoin-project/go-jsonrpc.(*RPCServer).handle
	/home/tippyflits/go/pkg/mod/github.com/filecoin-project/go-jsonrpc@v0.1.4-0.20210217175800-45ea43ac2bec/handler.go:259
2021-07-25T18:15:48.041+0100	ERROR	rpc	go-jsonrpc@v0.1.4-0.20210217175800-45ea43ac2bec/server.go:95	RPC Error: fatal error calling 'Filecoin.MarketListIncompleteDeals': panic in rpc method 'Filecoin.MarketListIncompleteDeals': runtime error: invalid memory address or nil pointer dereference
2021-07-25T18:15:48.041+0100	WARN	rpc	go-jsonrpc@v0.1.4-0.20210217175800-45ea43ac2bec/server.go:101	rpc error: fatal error calling 'Filecoin.MarketListIncompleteDeals': panic in rpc method 'Filecoin.MarketListIncompleteDeals': runtime error: invalid memory address or nil pointer dereference

Repo Steps

1. Run any command that outputs data related to deals.
2. View output as above.
@jennijuju jennijuju added this to the 1.11.1 milestone Jul 25, 2021
@nonsense
Copy link
Member

@TippyFlitsUK would you please share your repos' config.toml files? I see you are running a markets instance, and a proving/sealing instance, so I am wondering if the configs for these two are correct.

On the markets instance, you're supposed to have:

[Subsystems]
  EnableMining = false
  EnableSealing = false
  EnableSectorStorage = false
  EnableStorageMarket = true
  SealerApiInfo = "....."
  SectorIndexApiInfo = "....."

On the proving/sealing instance, you're supposed to have:

[Subsystems]
  EnableMining = true
  EnableSealing = true
  EnableSectorStorage = true
  EnableStorageMarket = false
  SealerApiInfo = ""
  SectorIndexApiInfo = ""

I believe that if you have MINER_API_INFO environment variable set, even though you call LOTUS_MINER_PATH=/mnt/nvme0n1/.lotusmarket lotus-miner storage-deals list, you are actually targeting the node specified in the MINER_API_INFO. So you might need to unset it.

Note that currently panics from the JSON RPC server are expected if you call a method that belongs to a subsystem that is not enabled. We are working on adding nicer errors and to gracefully handle this, but it has not yet landed on master.

@jennijuju
Copy link
Member

@stuberman I believe you had a same issue? Could you provide the config too if so?

@stuberman
Copy link

I have not upgraded my miner, still running 1.11.11-dev+mainnet+git.e6eb51b2d and so I have not experienced this yet.

@TippyFlitsUK
Copy link
Contributor Author

I can confirm that the 2 config.toml values are set correctly.

Miner Instance:

[Subsystems]
  EnableMining = true
  EnableSealing = true
  EnableSectorStorage = true
  EnableMarkets = false
  SealerApiInfo = ""
  SectorIndexApiInfo = ""

Markets Instance:

[Subsystems]
  EnableMining = false
  EnableSealing = false
  EnableSectorStorage = false
  EnableMarkets = true
  SealerApiInfo = "---"
  SectorIndexApiInfo = "---"

The MINER_API_INFO environment variable WAS set. I removed it and restarted my miner but am still getting errors.
(FYI both my miner and markets instances are running on the same machine so the MINER_API_INFO environment variable was pointing to the same node.)

lotus-miner storage-deals list

ERROR: fatal error calling 'Filecoin.MarketListIncompleteDeals': panic in rpc method 'Filecoin.MarketListIncompleteDeals': runtime error: invalid memory address or nil pointer dereference

2021-07-26T12:52:32.954+0100	ERROR	rpc	go-jsonrpc@v0.1.4-0.20210217175800-45ea43ac2bec/handler.go:158	panic in rpc method 'Filecoin.MarketListIncompleteDeals': runtime error: invalid memory address or nil pointer dereference
github.com/filecoin-project/go-jsonrpc.doCall.func1
	/home/tippyflits/go/pkg/mod/github.com/filecoin-project/go-jsonrpc@v0.1.4-0.20210217175800-45ea43ac2bec/handler.go:158
runtime.gopanic
	/usr/local/go/src/runtime/panic.go:965
runtime.panicmem
	/usr/local/go/src/runtime/panic.go:212
runtime.sigpanic
	/usr/local/go/src/runtime/signal_unix.go:734
github.com/filecoin-project/lotus/node/impl.(*StorageMinerAPI).MarketListIncompleteDeals
	/home/tippyflits/lotus/node/impl/storminer.go:460
reflect.Value.call
	/usr/local/go/src/reflect/value.go:476
reflect.Value.Call
	/usr/local/go/src/reflect/value.go:337
github.com/filecoin-project/lotus/metrics.proxy.func1
	/home/tippyflits/lotus/metrics/proxy.go:60
github.com/filecoin-project/lotus/api.(*StorageMinerStruct).MarketListIncompleteDeals
	/home/tippyflits/lotus/api/proxy_gen.go:2963
reflect.Value.call
	/usr/local/go/src/reflect/value.go:476
reflect.Value.Call
	/usr/local/go/src/reflect/value.go:337
github.com/filecoin-project/go-jsonrpc/auth.PermissionedProxy.func1
	/home/tippyflits/go/pkg/mod/github.com/filecoin-project/go-jsonrpc@v0.1.4-0.20210217175800-45ea43ac2bec/auth/auth.go:62
github.com/filecoin-project/lotus/api.(*StorageMinerStruct).MarketListIncompleteDeals
	/home/tippyflits/lotus/api/proxy_gen.go:2963
reflect.Value.call
	/usr/local/go/src/reflect/value.go:476
reflect.Value.Call
	/usr/local/go/src/reflect/value.go:337
github.com/filecoin-project/go-jsonrpc.doCall
	/home/tippyflits/go/pkg/mod/github.com/filecoin-project/go-jsonrpc@v0.1.4-0.20210217175800-45ea43ac2bec/handler.go:162
github.com/filecoin-project/go-jsonrpc.(*RPCServer).handle
	/home/tippyflits/go/pkg/mod/github.com/filecoin-project/go-jsonrpc@v0.1.4-0.20210217175800-45ea43ac2bec/handler.go:259
2021-07-26T12:52:32.954+0100	ERROR	rpc	go-jsonrpc@v0.1.4-0.20210217175800-45ea43ac2bec/server.go:95	RPC Error: fatal error calling 'Filecoin.MarketListIncompleteDeals': panic in rpc method 'Filecoin.MarketListIncompleteDeals': runtime error: invalid memory address or nil pointer dereference
2021-07-26T12:52:32.954+0100	WARN	rpc	go-jsonrpc@v0.1.4-0.20210217175800-45ea43ac2bec/server.go:101	rpc error: fatal error calling 'Filecoin.MarketListIncompleteDeals': panic in rpc method 'Filecoin.MarketListIncompleteDeals': runtime error: invalid memory address or nil pointer dereference

LOTUS_MINER_PATH=/mnt/nvme0n1/.lotusmarket lotus-miner storage-deals list

ERROR: could not get API info: could not get api endpoint: API not running (no endpoint)

@jennijuju
Copy link
Member

With #6857 - I’m expecting it not to panic at least

@jennijuju
Copy link
Member

From @TippyFlitsUK
. I've checked the Estuary error logs and this is the message they have been receiving from my miner since the separation...

failed to open stream to peer: protocol not supported

@nonsense
Copy link
Member

With #6857 - I’m expecting it not to panic at least

Actually we need a slightly different codegen for this not to panic - the method is there, but the subsystem is missing, so we need the codegen to maybe check for nil ptr. So unfortunately this is not solving for these cases.

@jennijuju
Copy link
Member

This was also reported by @Meatball13 in #6852 (comment)

@jennijuju jennijuju added P1 P1: Must be resolved and removed P1 P1: Must be resolved labels Jul 27, 2021
@jennijuju
Copy link
Member

Running on the same instance is also reportinf this error #6861 (reply in thread)

@jennijuju
Copy link
Member

@raulk could you please confirm if this can be closed by #6936?

@raulk
Copy link
Member

raulk commented Aug 9, 2021

Yes, this can be closed and the environment variables have been documented in filecoin-project/filecoin-docs#996.

@raulk raulk closed this as completed Aug 9, 2021
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees

@nonsense nonsense

Labels
area/markets/provider area/markets/storage kind/bug Kind: Bug P1 P1: Must be resolved
Projects
None yet
Milestone
1.11.1
Development

No branches or pull requests
5 participants
@nonsense @raulk @TippyFlitsUK @jennijuju @stuberman