Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

panic: runtime error: invalid memory address or nil pointer dereference #18388

Closed
FilipRy opened this issue Jan 3, 2019 · 14 comments
Closed

Comments

@FilipRy
Copy link

FilipRy commented Jan 3, 2019

Hi there,

I have a topology consisting of 48 docker containers. Each container runs its own geth node. (The docker containers are build from my custom docker image -> see below). Each geth node is a miner. After I create many transactions and ca 1500 blocks are generated. One geth node reports the runtime error mentioned below.

My dockerfile:

FROM ethereum/client-go:v1.8.20
COPY eth_common /eth_common

RUN geth --rpc --rpcaddr 0.0.0.0 --rpcport 8545 --datadir /data --syncmode "fast" --cache "1024" --rpcapi="db,eth,net,web3,personal,web3" --txpool.pricelimit "0" --miner.gasprice "0" --maxpeers "80" --nodiscover init eth_common/genesis.json

ENTRYPOINT ["geth"]

Genesis.json:

{
    "config": {
        "chainId": 33,
        "homesteadBlock": 0,
        "eip155Block": 0,
        "eip158Block": 0
    },
    "nonce": "0x0000000000000033",
    "timestamp": "0x0",
    "parentHash": "0x0000000000000000000000000000000000000000000000000000000000000000",
    "gasLimit": "0x8000000",
    "difficulty": "0x100",
    "mixhash": "0x0000000000000000000000000000000000000000000000000000000000000000",
    "coinbase": "0x3333333333333333333333333333333333333333",
    "alloc": { }
}

Steps to reproduce the behaviour

  1. Build the docker image
  2. Run geth in 48 dockers: docker run -it -d -p ${enodePortNr}:30303 --network="eth" --name ${client.name}_bc --net-alias=${client.name}_bc mygeth --rpc --rpcaddr 0.0.0.0 --rpcport 8545 --datadir /data --syncmode "fast" --cache "1024" --rpcapi="db,eth,net,web3,personal,web3" --txpool.pricelimit "0" --miner.gasprice "0" --maxpeers "80" --nodiscover --verbosity 4'
  3. Create personal accounts
  4. Start miners
  5. One geth node reports error.

Backtrace

INFO [01-03|12:30:47.040] Commit new mining work                   number=1502 sealhash=4cad4d…5af5ef uncles=0 txs=11  gas=298540  fees=0           elapsed=19.790ms
DEBUG[01-03|12:30:47.236] Executing EVM call finished              runtime=1.629426ms
DEBUG[01-03|12:30:47.534] Fetching batch of headers                id=bf3d4dc0f09e0c8d conn=inbound count=192 fromnum=1502 skip=0   reverse=false
DEBUG[01-03|12:30:48.645] Queued propagated block                  peer=bf3d4dc0f09e0c8d number=1505 hash=1d499f…43a93b queued=1
DEBUG[01-03|12:30:48.646] Fetching single header                   id=bf3d4dc0f09e0c8d conn=inbound hash=110c16…a7ac37
DEBUG[01-03|12:30:48.647] Fetching batch of headers                id=bf3d4dc0f09e0c8d conn=inbound count=192 fromnum=1506 skip=0   reverse=false
panic: runtime error: invalid memory address or nil pointer dereference
[signal SIGSEGV: segmentation violation code=0x1 addr=0x0 pc=0x51736e]

goroutine 61284 [running]:
math/big.(*Int).Cmp(0xc018ba8fa0, 0x0, 0xb93457648fe46786)
	/usr/local/go/src/math/big/int.go:319 +0x2e
github.com/ethereum/go-ethereum/core.(*BlockChain).insertSidechain(0xc0053ae000, 0xc0149e3530, 0xc0000c75c0, 0xc0000c7501, 0x1, 0x1, 0xc018b9dcd0, 0x1, 0x1, 0xc00592ed80, ...)
	/go-ethereum/build/_workspace/src/github.com/ethereum/go-ethereum/core/blockchain.go:1333 +0x10b2
github.com/ethereum/go-ethereum/core.(*BlockChain).insertChain(0xc0053ae000, 0xc006ff46b8, 0x1, 0x1, 0x1, 0x0, 0x0, 0x0, 0x0, 0x0, ...)
	/go-ethereum/build/_workspace/src/github.com/ethereum/go-ethereum/core/blockchain.go:1139 +0x2e15
github.com/ethereum/go-ethereum/core.(*BlockChain).InsertChain(0xc0053ae000, 0xc006ff46b8, 0x1, 0x1, 0x8, 0x8, 0xe03500)
	/go-ethereum/build/_workspace/src/github.com/ethereum/go-ethereum/core/blockchain.go:1088 +0xb89
github.com/ethereum/go-ethereum/eth.NewProtocolManager.func6(0xc006ff46b8, 0x1, 0x1, 0x1e92401, 0xc015369830, 0xf072b3d137b959f8)
	/go-ethereum/build/_workspace/src/github.com/ethereum/go-ethereum/eth/handler.go:183 +0x30b
github.com/ethereum/go-ethereum/eth/fetcher.(*Fetcher).insert.func1(0xc00028c000, 0x30d0f2620337c97e, 0xe9ad3862cafae1bd, 0xf072b3d137b959f8, 0xfa219c028ac451b, 0xc01f574510, 0xc023e581a0, 0x10)
	/go-ethereum/build/_workspace/src/github.com/ethereum/go-ethereum/eth/fetcher/fetcher.go:669 +0x53d
created by github.com/ethereum/go-ethereum/eth/fetcher.(*Fetcher).insert
	/go-ethereum/build/_workspace/src/github.com/ethereum/go-ethereum/eth/fetcher/fetcher.go:643 +0x2ad
@karalabe
Copy link
Member

Could you try out #18436 if it fixes your issue?

@Lvovich2018
Copy link

Hi. I also get this error. Here's my launch line.
geth --rpc --rpcaddr "0.0.0.0" --rpcport 8545 --rpccorsdomain "*" --rpcapi "db,eth,net,web3,personal" --cache 8192 --syncmode "full"
Here are the messages on the screen.

`INFO [02-04|19:08:30.845] Maximum peer count ETH=25 LES=0 total=25
INFO [02-04|19:08:31.000] Starting peer-to-peer node instance=Geth/v1.8.22-stable-7fa3509e/linux-amd64/go1.11.5
INFO [02-04|19:08:31.001] Allocated cache and file handles database=/home/t/tihonov9/.ethereum/geth/chaindata cache=4096 handles=519999
INFO [02-04|19:08:35.030] Initialised chain configuration config="{ChainID: 1 Homestead: 1150000 DAO: 1920000 DAOSupport: true EIP150: 2463000 EIP155: 2675000 EIP158: 2675000 Byzantium: 4370000 Constantinople: 7280000 ConstantinopleFix: 7280000 Engine: ethash}"
INFO [02-04|19:08:35.030] Disk storage enabled for ethash caches dir=/home/t/tihonov9/.ethereum/geth/ethash count=3
INFO [02-04|19:08:35.030] Disk storage enabled for ethash DAGs dir=/home/t/tihonov9/.ethash count=2
INFO [02-04|19:08:35.030] Initialising Ethereum protocol versions="[63 62]" network=1
WARN [02-04|19:08:35.205] Head state missing, repairing chain number=7155110 hash=7e38bc…30b877
INFO [02-04|19:08:35.908] Rewound blockchain to past state number=7154010 hash=00b374…5490bf
INFO [02-04|19:08:35.909] Loaded most recent local header number=7157401 hash=f797d4…32c8cd td=8963951431232627924909 age=3d9h56m
INFO [02-04|19:08:35.909] Loaded most recent local full block number=7154010 hash=00b374…5490bf td=8955533262045259057180 age=4d2h21m
INFO [02-04|19:08:35.909] Loaded most recent local fast block number=7157401 hash=f797d4…32c8cd td=8963951431232627924909 age=3d9h56m
INFO [02-04|19:08:35.909] Loaded local transaction journal transactions=0 dropped=0
INFO [02-04|19:08:35.909] Regenerated local transaction journal transactions=0 accounts=0
INFO [02-04|19:08:35.921] New local node record seq=3 id=d206b65a2f6abdc3 ip=127.0.0.1 udp=30303 tcp=30303
INFO [02-04|19:08:35.921] Started P2P networking self=enode://2d25bbceafeb32652614f6e1f68a4f06aeed16db793db304fb0d0490e965d6c46822940b8411a23ae48da8b87c10702481b7d2151e26866fa153e61fe52f12d1@127.0.0.1:30303
INFO [02-04|19:08:35.922] IPC endpoint opened url=/home/t/tihonov9/.ethereum/geth.ipc
INFO [02-04|19:08:35.922] HTTP endpoint opened url=http://0.0.0.0:8545 cors=* vhosts=localhost
WARN [02-04|19:08:37.032] Failed account unlock attempt address=0x93324FC20077F03B0112720020E300a10da33D51 err="could not decrypt key with given passphrase"
WARN [02-04|19:08:38.120] Failed account unlock attempt address=0x93324FC20077F03B0112720020E300a10da33D51 err="could not decrypt key with given passphrase"
WARN [02-04|19:08:39.139] Failed account unlock attempt address=0x93324FC20077F03B0112720020E300a10da33D51 err="could not decrypt key with given passphrase"
WARN [02-04|19:08:40.226] Failed account unlock attempt address=0x93324FC20077F03B0112720020E300a10da33D51 err="could not decrypt key with given passphrase"
WARN [02-04|19:08:41.250] Failed account unlock attempt address=0x93324FC20077F03B0112720020E300a10da33D51 err="could not decrypt key with given passphrase"
WARN [02-04|19:08:42.281] Failed account unlock attempt address=0x93324FC20077F03B0112720020E300a10da33D51 err="could not decrypt key with given passphrase"
WARN [02-04|19:08:43.312] Failed account unlock attempt address=0x93324FC20077F03B0112720020E300a10da33D51 err="could not decrypt key with given passphrase"
WARN [02-04|19:08:44.452] Failed account unlock attempt address=0x93324FC20077F03B0112720020E300a10da33D51 err="could not decrypt key with given passphrase"
WARN [02-04|19:08:45.486] Failed account unlock attempt address=0x93324FC20077F03B0112720020E300a10da33D51 err="could not decrypt key with given passphrase"
WARN [02-04|19:08:46.514] Failed account unlock attempt address=0x93324FC20077F03B0112720020E300a10da33D51 err="could not decrypt key with given passphrase"
WARN [02-04|19:08:47.537] Failed account unlock attempt address=0x93324FC20077F03B0112720020E300a10da33D51 err="could not decrypt key with given passphrase"
WARN [02-04|19:08:48.562] Failed account unlock attempt address=0x93324FC20077F03B0112720020E300a10da33D51 err="could not decrypt key with given passphrase"
WARN [02-04|19:08:49.607] Failed account unlock attempt address=0x93324FC20077F03B0112720020E300a10da33D51 err="could not decrypt key with given passphrase"
WARN [02-04|19:08:50.630] Failed account unlock attempt address=0x93324FC20077F03B0112720020E300a10da33D51 err="could not decrypt key with given passphrase"
WARN [02-04|19:08:51.660] Failed account unlock attempt address=0x93324FC20077F03B0112720020E300a10da33D51 err="could not decrypt key with given passphrase"
WARN [02-04|19:08:52.695] Failed account unlock attempt address=0x93324FC20077F03B0112720020E300a10da33D51 err="could not decrypt key with given passphrase"
INFO [02-04|19:08:53.453] New local node record seq=4 id=d206b65a2f6abdc3 ip=5.101.156.182 udp=30303 tcp=30303
WARN [02-04|19:08:53.724] Failed account unlock attempt address=0x93324FC20077F03B0112720020E300a10da33D51 err="could not decrypt key with given passphrase"
WARN [02-04|19:08:54.753] Failed account unlock attempt address=0x93324FC20077F03B0112720020E300a10da33D51 err="could not decrypt key with given passphrase"
WARN [02-04|19:08:55.779] Failed account unlock attempt address=0x93324FC20077F03B0112720020E300a10da33D51 err="could not decrypt key with given passphrase"
WARN [02-04|19:08:56.817] Failed account unlock attempt address=0x93324FC20077F03B0112720020E300a10da33D51 err="could not decrypt key with given passphrase"
WARN [02-04|19:08:57.868] Failed account unlock attempt address=0x93324FC20077F03B0112720020E300a10da33D51 err="could not decrypt key with given passphrase"
WARN [02-04|19:08:58.894] Failed account unlock attempt address=0x93324FC20077F03B0112720020E300a10da33D51 err="could not decrypt key with given passphrase"
WARN [02-04|19:08:59.943] Failed account unlock attempt address=0x93324FC20077F03B0112720020E300a10da33D51 err="could not decrypt key with given passphrase"
WARN [02-04|19:09:00.840] Failed account unlock attempt address=0xcCc1Bb3A253DC91b9585787f70f80bEEA078CFA5 err="could not decrypt key with given passphrase"
WARN [02-04|19:09:00.979] Failed account unlock attempt address=0x93324FC20077F03B0112720020E300a10da33D51 err="could not decrypt key with given passphrase"
WARN [02-04|19:09:02.048] Failed account unlock attempt address=0x93324FC20077F03B0112720020E300a10da33D51 err="could not decrypt key with given passphrase"
WARN [02-04|19:09:03.195] Failed account unlock attempt address=0x93324FC20077F03B0112720020E300a10da33D51 err="could not decrypt key with given passphrase"
WARN [02-04|19:09:04.253] Failed account unlock attempt address=0x93324FC20077F03B0112720020E300a10da33D51 err="could not decrypt key with given passphrase"
WARN [02-04|19:09:04.983] Failed account unlock attempt address=0xF736d28db97648F1C5aB329E44b4b71f5395F74c err="could not decrypt key with given passphrase"
WARN [02-04|19:09:05.284] Failed account unlock attempt address=0x93324FC20077F03B0112720020E300a10da33D51 err="could not decrypt key with given passphrase"
INFO [02-04|19:09:05.921] Block synchronisation started
WARN [02-04|19:09:06.309] Failed account unlock attempt address=0x93324FC20077F03B0112720020E300a10da33D51 err="could not decrypt key with given passphrase"
panic: runtime error: invalid memory address or nil pointer dereference
[signal SIGSEGV: segmentation violation code=0x1 addr=0x1c0 pc=0xa67753]

goroutine 11159 [running]:
github.com/ethereum/go-ethereum/eth/downloader.(*Downloader).findAncestor(0xc094f66340, 0xc08ceda4b0, 0xc094452b40, 0x0, 0x0, 0xc000046f70)
/home/travis/gopath/src/github.com/ethereum/go-ethereum/eth/downloader/downloader.go:825 +0x15d3
github.com/ethereum/go-ethereum/eth/downloader.(*Downloader).syncWithPeer(0xc094f66340, 0xc08ceda4b0, 0xf09afdf02b0baecb, 0x6c05906dccece85f, 0xc54f87c504a14fb0, 0xdb999058d1e11d3b, 0xc0481b9e00, 0x0, 0x0)
/home/travis/gopath/src/github.com/ethereum/go-ethereum/eth/downloader/downloader.go:437 +0x37d
github.com/ethereum/go-ethereum/eth/downloader.(*Downloader).synchronise(0xc094f66340, 0xc095793100, 0x10, 0xf09afdf02b0baecb, 0x6c05906dccece85f, 0xc54f87c504a14fb0, 0xdb999058d1e11d3b, 0xc0481b9e00, 0x0, 0x0, ...)
/home/travis/gopath/src/github.com/ethereum/go-ethereum/eth/downloader/downloader.go:406 +0x362
github.com/ethereum/go-ethereum/eth/downloader.(*Downloader).Synchronise(0xc094f66340, 0xc095793100, 0x10, 0xf09afdf02b0baecb, 0x6c05906dccece85f, 0xc54f87c504a14fb0, 0xdb999058d1e11d3b, 0xc0481b9e00, 0x0, 0x46f3b0, ...)
/home/travis/gopath/src/github.com/ethereum/go-ethereum/eth/downloader/downloader.go:324 +0x91
github.com/ethereum/go-ethereum/eth.(*ProtocolManager).synchronise(0xc045bc3790, 0xc047b70000)
/home/travis/gopath/src/github.com/ethereum/go-ethereum/eth/sync.go:200 +0x313
created by github.com/ethereum/go-ethereum/eth.(*ProtocolManager).syncer
/home/travis/gopath/src/github.com/ethereum/go-ethereum/eth/sync.go:155 +0x1df
`
(Sorry, I can't insert the log beautifully.)

This problem started from 2019-01-31

@karalabe
Copy link
Member

karalabe commented Feb 4, 2019

@Lvovich2018 Before even diving into this crash, please immediately close your node's RPC port from the internet. Your logs are full of failed unlock attempts, pointing to someone remotely trying to brute force your password:

WARN [02-04|19:08:37.032] Failed account unlock attempt address=0x93324FC20077F03B0112720020E300a10da33D51 err="could not decrypt key with given passphrase"
WARN [02-04|19:08:38.120] Failed account unlock attempt address=0x93324FC20077F03B0112720020E300a10da33D51 err="could not decrypt key with given passphrase"
WARN [02-04|19:08:39.139] Failed account unlock attempt address=0x93324FC20077F03B0112720020E300a10da33D51 err="could not decrypt key with given passphrase"
WARN [02-04|19:08:40.226] Failed account unlock attempt address=0x93324FC20077F03B0112720020E300a10da33D51 err="could not decrypt key with given passphrase"
WARN [02-04|19:08:41.250] Failed account unlock attempt address=0x93324FC20077F03B0112720020E300a10da33D51 err="could not decrypt key with given passphrase"
WARN [02-04|19:08:42.281] Failed account unlock attempt address=0x93324FC20077F03B0112720020E300a10da33D51 err="could not decrypt key with given passphrase"
WARN [02-04|19:08:43.312] Failed account unlock attempt address=0x93324FC20077F03B0112720020E300a10da33D51 err="could not decrypt key with given passphrase"
WARN [02-04|19:08:44.452] Failed account unlock attempt address=0x93324FC20077F03B0112720020E300a10da33D51 err="could not decrypt key with given passphrase"
WARN [02-04|19:08:45.486] Failed account unlock attempt address=0x93324FC20077F03B0112720020E300a10da33D51 err="could not decrypt key with given passphrase"
WARN [02-04|19:08:46.514] Failed account unlock attempt address=0x93324FC20077F03B0112720020E300a10da33D51 err="could not decrypt key with given passphrase"
WARN [02-04|19:08:47.537] Failed account unlock attempt address=0x93324FC20077F03B0112720020E300a10da33D51 err="could not decrypt key with given passphrase"
WARN [02-04|19:08:48.562] Failed account unlock attempt address=0x93324FC20077F03B0112720020E300a10da33D51 err="could not decrypt key with given passphrase"
WARN [02-04|19:08:49.607] Failed account unlock attempt address=0x93324FC20077F03B0112720020E300a10da33D51 err="could not decrypt key with given passphrase"
WARN [02-04|19:08:50.630] Failed account unlock attempt address=0x93324FC20077F03B0112720020E300a10da33D51 err="could not decrypt key with given passphrase"
WARN [02-04|19:08:51.660] Failed account unlock attempt address=0x93324FC20077F03B0112720020E300a10da33D51 err="could not decrypt key with given passphrase"
WARN [02-04|19:08:52.695] Failed account unlock attempt address=0x93324FC20077F03B0112720020E300a10da33D51 err="could not decrypt key with given passphrase"

@Lvovich2018
Copy link

OK, closed. But the error remained.

`INFO [02-04|19:44:34.464] Maximum peer count ETH=25 LES=0 total=25
INFO [02-04|19:44:34.618] Starting peer-to-peer node instance=Geth/v1.8.22-stable-7fa3509e/linux-amd64/go1.11.5
INFO [02-04|19:44:34.618] Allocated cache and file handles database=/home/t/tihonov9/.ethereum/geth/chaindata cache=4096 handles=519999
INFO [02-04|19:44:37.575] Initialised chain configuration config="{ChainID: 1 Homestead: 1150000 DAO: 1920000 DAOSupport: true EIP150: 2463000 EIP155: 2675000 EIP158: 2675000 Byzantium: 4370000 Constantinople: 7280000 ConstantinopleFix: 7280000 Engine: ethash}"
INFO [02-04|19:44:37.575] Disk storage enabled for ethash caches dir=/home/t/tihonov9/.ethereum/geth/ethash count=3
INFO [02-04|19:44:37.575] Disk storage enabled for ethash DAGs dir=/home/t/tihonov9/.ethash count=2
INFO [02-04|19:44:37.575] Initialising Ethereum protocol versions="[63 62]" network=1
WARN [02-04|19:44:38.011] Head state missing, repairing chain number=7155110 hash=7e38bc…30b877
INFO [02-04|19:44:38.745] Rewound blockchain to past state number=7154010 hash=00b374…5490bf
INFO [02-04|19:44:38.746] Loaded most recent local header number=7157401 hash=f797d4…32c8cd td=8963951431232627924909 age=3d10h32m
INFO [02-04|19:44:38.746] Loaded most recent local full block number=7154010 hash=00b374…5490bf td=8955533262045259057180 age=4d2h57m
INFO [02-04|19:44:38.746] Loaded most recent local fast block number=7157401 hash=f797d4…32c8cd td=8963951431232627924909 age=3d10h32m
INFO [02-04|19:44:38.746] Loaded local transaction journal transactions=0 dropped=0
INFO [02-04|19:44:38.746] Regenerated local transaction journal transactions=0 accounts=0
INFO [02-04|19:44:38.758] New local node record seq=5 id=d206b65a2f6abdc3 ip=127.0.0.1 udp=30303 tcp=30303
INFO [02-04|19:44:38.758] Started P2P networking self=enode://2d25bbceafeb32652614f6e1f68a4f06aeed16db793db304fb0d0490e965d6c46822940b8411a23ae48da8b87c10702481b7d2151e26866fa153e61fe52f12d1@127.0.0.1:30303
INFO [02-04|19:44:38.760] IPC endpoint opened url=/home/t/tihonov9/.ethereum/geth.ipc
INFO [02-04|19:44:38.760] HTTP endpoint opened url=http://127.0.0.1:8545 cors=* vhosts=localhost
INFO [02-04|19:44:53.703] New local node record seq=6 id=d206b65a2f6abdc3 ip=5.101.156.182 udp=30303 tcp=30303
INFO [02-04|19:44:58.758] Block synchronisation started
panic: runtime error: invalid memory address or nil pointer dereference
[signal SIGSEGV: segmentation violation code=0x1 addr=0x1c0 pc=0xa67753]

goroutine 305 [running]:
github.com/ethereum/go-ethereum/eth/downloader.(*Downloader).findAncestor(0xc000225520, 0xc08d499b30, 0xc08dd9e480, 0x0, 0x0, 0xc00003e070)
/home/travis/gopath/src/github.com/ethereum/go-ethereum/eth/downloader/downloader.go:825 +0x15d3
github.com/ethereum/go-ethereum/eth/downloader.(*Downloader).syncWithPeer(0xc000225520, 0xc08d499b30, 0x9febcc36adda7512, 0x7be5f5b0e9821a04, 0xc9d350efa308c3cf, 0xe7a145033620a59, 0xc08f174ea0, 0x0, 0x0)
/home/travis/gopath/src/github.com/ethereum/go-ethereum/eth/downloader/downloader.go:437 +0x37d
github.com/ethereum/go-ethereum/eth/downloader.(*Downloader).synchronise(0xc000225520, 0xc08e6e8e30, 0x10, 0x9febcc36adda7512, 0x7be5f5b0e9821a04, 0xc9d350efa308c3cf, 0xe7a145033620a59, 0xc08f174ea0, 0x0, 0x0, ...)
/home/travis/gopath/src/github.com/ethereum/go-ethereum/eth/downloader/downloader.go:406 +0x362
github.com/ethereum/go-ethereum/eth/downloader.(*Downloader).Synchronise(0xc000225520, 0xc08e6e8e30, 0x10, 0x9febcc36adda7512, 0x7be5f5b0e9821a04, 0xc9d350efa308c3cf, 0xe7a145033620a59, 0xc08f174ea0, 0x0, 0xc0007c6f30, ...)
/home/travis/gopath/src/github.com/ethereum/go-ethereum/eth/downloader/downloader.go:324 +0x91
github.com/ethereum/go-ethereum/eth.(*ProtocolManager).synchronise(0xc0483981a0, 0xc0901660c0)
/home/travis/gopath/src/github.com/ethereum/go-ethereum/eth/sync.go:200 +0x313
created by github.com/ethereum/go-ethereum/eth.(*ProtocolManager).syncer
/home/travis/gopath/src/github.com/ethereum/go-ethereum/eth/sync.go:155 +0x1df
`

@fedecaccia
Copy link

I have the same error trying to re-synchronize a node. Only works deleting database and downloading it all again.

@Lvovich2018
Copy link

I have the same error trying to re-synchronize a node. Only works deleting database and downloading it all again.

Thank you for your advice. Unfortunately, I can't do that. And there is no guarantee that it will help.

@Lvovich2018
Copy link

Any recommendations on my problem?

@Lvovich2018
Copy link

geth --rpc --rpcport 8545 --rpccorsdomain "*" --rpcapi "db,eth,net,web3,personal" --cache 8192 --syncmode "full"
and I get the same result:

`INFO [02-05|09:19:50.099] Maximum peer count ETH=25 LES=0 total=25
INFO [02-05|09:19:50.255] Starting peer-to-peer node instance=Geth/v1.8.22-stable-7fa3509e/linux-amd64/go1.11.5
INFO [02-05|09:19:50.255] Allocated cache and file handles database=/home/t/tihonov9/.ethereum/geth/chaindata cache=4096 handles=519999
INFO [02-05|09:19:54.226] Initialised chain configuration config="{ChainID: 1 Homestead: 1150000 DAO: 1920000 DAOSupport: true EIP150: 2463000 EIP155: 2675000 EIP158: 2675000 Byzantium: 4370000 Constantinople: 7280000 ConstantinopleFix: 7280000 Engine: ethash}"
INFO [02-05|09:19:54.226] Disk storage enabled for ethash caches dir=/home/t/tihonov9/.ethereum/geth/ethash count=3
INFO [02-05|09:19:54.226] Disk storage enabled for ethash DAGs dir=/home/t/tihonov9/.ethash count=2
INFO [02-05|09:19:54.226] Initialising Ethereum protocol versions="[63 62]" network=1
WARN [02-05|09:19:54.393] Head state missing, repairing chain number=7155110 hash=7e38bc…30b877
INFO [02-05|09:19:55.106] Rewound blockchain to past state number=7154010 hash=00b374…5490bf
INFO [02-05|09:19:55.107] Loaded most recent local header number=7157401 hash=f797d4…32c8cd td=8963951431232627924909 age=4d7m53s
INFO [02-05|09:19:55.107] Loaded most recent local full block number=7154010 hash=00b374…5490bf td=8955533262045259057180 age=4d16h32m
INFO [02-05|09:19:55.107] Loaded most recent local fast block number=7157401 hash=f797d4…32c8cd td=8963951431232627924909 age=4d7m53s
INFO [02-05|09:19:55.107] Loaded local transaction journal transactions=0 dropped=0
INFO [02-05|09:19:55.107] Regenerated local transaction journal transactions=0 accounts=0
INFO [02-05|09:19:55.130] New local node record seq=7 id=d206b65a2f6abdc3 ip=127.0.0.1 udp=30303 tcp=30303
INFO [02-05|09:19:55.130] Started P2P networking self=enode://2d25bbceafeb32652614f6e1f68a4f06aeed16db793db304fb0d0490e965d6c46822940b8411a23ae48da8b87c10702481b7d2151e26866fa153e61fe52f12d1@127.0.0.1:30303
INFO [02-05|09:19:55.131] IPC endpoint opened url=/home/t/tihonov9/.ethereum/geth.ipc
INFO [02-05|09:19:55.131] HTTP endpoint opened url=http://127.0.0.1:8545 cors=* vhosts=localhost
INFO [02-05|09:20:14.233] New local node record seq=8 id=d206b65a2f6abdc3 ip=5.101.156.182 udp=30303 tcp=30303
INFO [02-05|09:20:25.130] Block synchronisation started
panic: runtime error: invalid memory address or nil pointer dereference
[signal SIGSEGV: segmentation violation code=0x1 addr=0x1c0 pc=0xa67753]

goroutine 508 [running]:
github.com/ethereum/go-ethereum/eth/downloader.(*Downloader).findAncestor(0xc08ff7f520, 0xc08f7b2a50, 0xc0944b4240, 0x0, 0x0, 0xc00004e570)
/home/travis/gopath/src/github.com/ethereum/go-ethereum/eth/downloader/downloader.go:825 +0x15d3
github.com/ethereum/go-ethereum/eth/downloader.(*Downloader).syncWithPeer(0xc08ff7f520, 0xc08f7b2a50, 0x2217595fecd5ce2b, 0x663214e84ed49888, 0xe654fec1f09cbe0f, 0xff77193375451257, 0xc0451e3380, 0x0, 0x0)
/home/travis/gopath/src/github.com/ethereum/go-ethereum/eth/downloader/downloader.go:437 +0x37d
github.com/ethereum/go-ethereum/eth/downloader.(*Downloader).synchronise(0xc08ff7f520, 0xc08b502720, 0x10, 0x2217595fecd5ce2b, 0x663214e84ed49888, 0xe654fec1f09cbe0f, 0xff77193375451257, 0xc0451e3380, 0x0, 0x0, ...)
/home/travis/gopath/src/github.com/ethereum/go-ethereum/eth/downloader/downloader.go:406 +0x362
github.com/ethereum/go-ethereum/eth/downloader.(*Downloader).Synchronise(0xc08ff7f520, 0xc08b502720, 0x10, 0x2217595fecd5ce2b, 0x663214e84ed49888, 0xe654fec1f09cbe0f, 0xff77193375451257, 0xc0451e3380, 0x0, 0x41aac5, ...)
/home/travis/gopath/src/github.com/ethereum/go-ethereum/eth/downloader/downloader.go:324 +0x91
github.com/ethereum/go-ethereum/eth.(*ProtocolManager).synchronise(0xc097ad3520, 0xc0458ce6c0)
/home/travis/gopath/src/github.com/ethereum/go-ethereum/eth/sync.go:200 +0x313
created by github.com/ethereum/go-ethereum/eth.(*ProtocolManager).syncer
/home/travis/gopath/src/github.com/ethereum/go-ethereum/eth/sync.go:155 +0x1df
`

What am I to do?

@holiman
Copy link
Contributor

holiman commented Feb 5, 2019

The original report has been fixed. The report by @Lvovich2018 is not fixed, and is a bit odd.

https://github.com/ethereum/go-ethereum/blob/release/1.8/eth/downloader/downloader.go#L824

				var known bool
				switch d.mode {
				case FullSync:
					known = d.blockchain.HasBlock(h, n)
				case FastSync:
					known = d.blockchain.HasFastBlock(h, n)
				default:
					known = d.lightchain.HasHeader(h, n)
				}
				if !known {
					end = check
					break
				}
		header := d.lightchain.GetHeaderByHash(h) // Independent of sync mode, header surely exists

It fails at the last line. @karalabe what's the reasoning behind that comment -- the chain is started in fullsync mode, it seems that the assumption that the lightchain has the header hash is not true. Could it be that the lightchain only has the canon blocks?

@karalabe
Copy link
Member

karalabe commented Feb 6, 2019

The odd thing in this code segment is that d.blockchain.HasFastBlock(h, n) returns true, which means that the header + block body + receipt exists and is retrievable. If d.lightchain.GetHeaderByHash(h) returns nil, the only fault that could cause it is if the hash->number mapping is missing from the database (HasFastBlock is provided the number too, so it doesn't need to access the hash->number mapping, rather can pull the data directly).

We need to figure out what could have caused the fast sync head block to be present in the database, but it's number mapping to be missing. Not sure yet.

@karalabe
Copy link
Member

karalabe commented Feb 6, 2019

Looking at our data access methods, it's again a bit weird:

// WriteHeader stores a block header into the database and also stores the hash-
// to-number mapping.
func WriteHeader(db DatabaseWriter, header *types.Header) {
	// Write the hash -> number mapping
	var (
		hash    = header.Hash()
		number  = header.Number.Uint64()
		encoded = encodeBlockNumber(number)
	)
	key := headerNumberKey(hash)
	if err := db.Put(key, encoded); err != nil {
		log.Crit("Failed to store hash to number mapping", "err", err)
	}
	// Write the encoded header
	data, err := rlp.EncodeToBytes(header)
	if err != nil {
		log.Crit("Failed to RLP encode header", "err", err)
	}
	key = headerKey(number, hash)
	if err := db.Put(key, data); err != nil {
		log.Crit("Failed to store header", "err", err)
	}
}

// DeleteHeader removes all block header data associated with a hash.
func DeleteHeader(db DatabaseDeleter, hash common.Hash, number uint64) {
	if err := db.Delete(headerKey(number, hash)); err != nil {
		log.Crit("Failed to delete header", "err", err)
	}
	if err := db.Delete(headerNumberKey(hash)); err != nil {
		log.Crit("Failed to delete hash to number mapping", "err", err)
	}
}

I.e. We first write the hash->number mapping, and only then the header. Upon deletion, we first delete the header, and only then the hash->number mapping. Based on this, it should be impossible to have the header present, but the hash->number mapping missing.

@karalabe
Copy link
Member

karalabe commented Feb 6, 2019

@Lvovich2018 Could you please run with --vmodule=downloader=5 and paste the logs here? Would be nice to see exactly which blocks the downloader is attempting to process when it blows.

@Lvovich2018
Copy link

@Lvovich2018 Could you please run with --vmodule=downloader=5 and paste the logs here? Would be nice to see exactly which blocks the downloader is attempting to process when it blows.

@karalabe, sorry, I can't do that right now. I only found a solution to my problem an hour ago. Helped downgrade geth on 1.8.19. Now the blocks are synchronizing, I can't stop the process. If on new versions this problem appears, I will surely insert logging. Thanks for your help.

@holiman
Copy link
Contributor

holiman commented Feb 7, 2019

Since original report is fixed, let's move over to #18994

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

No branches or pull requests

6 participants
@karalabe @holiman @fedecaccia @FilipRy @Lvovich2018 and others