segfault when running geth on a server for a couple of hours.

[kfir-drivenets]

System information

Geth version:
Geth
Version: 1.6.7-stable
Git Commit: ab5646c
Architecture: amd64
Protocol Versions: [63 62]
Network Id: 1
Go Version: go1.8.1
Operating System: linux
GOPATH=
GOROOT=/usr/lib/go-1.8
OS & Version: Linux kfir-Z10PE-D8-WS 4.4.0-31-generic #50~14.04.1-Ubuntu SMP Wed Jul 13 01:07:32 UTC 2016 x86_64 x86_64 x86_64 GNU/Linux

Steps to reproduce the behaviour

Run geth and a large server for a couple of hours.

Backtrace

Note that this is only a partial backtrace due to the large number of co-routines running.

....
goroutine 653885 [runnable]:
github.com/ethereum/go-ethereum/consensus/ethash.hashimoto(0xc422b11d78, 0x20, 0x20, 0x48a47316527793f9, 0x3ffff880, 0xc4933a7b00, 0xc422b11c48, 0x40b914, 0xd3a7c0, 0xc4283063c0, ...)
        /build/ethereum-YPcqTf/ethereum-1.6.7+build10273+trusty/build/_workspace/src/github.com/ethereum/go-ethereum/consensus/ethash/algorithm.go:312 +0x36e
github.com/ethereum/go-ethereum/consensus/ethash.hashimotoFull(0x7f78bc000008, 0xffffe20, 0xffffe20, 0xc422b11d78, 0x20, 0x20, 0x48a47316527793f9, 0xc4933a7aa0, 0x20, 0x20, ...)
        /build/ethereum-YPcqTf/ethereum-1.6.7+build10273+trusty/build/_workspace/src/github.com/ethereum/go-ethereum/consensus/ethash/algorithm.go:355 +0xbe
github.com/ethereum/go-ethereum/consensus/ethash.(*Ethash).mine(0xc422b5d380, 0xc4574ea3f0, 0xa, 0x48a473164a119429, 0xc4283063c0, 0xc428306420)
        /build/ethereum-YPcqTf/ethereum-1.6.7+build10273+trusty/build/_workspace/src/github.com/ethereum/go-ethereum/consensus/ethash/sealer.go:130 +0x451
github.com/ethereum/go-ethereum/consensus/ethash.(*Ethash).Seal.func1(0xc434ac8c30, 0xc422b5d380, 0xc4574ea3f0, 0xc4283063c0, 0xc428306420, 0xa, 0x48a473164a119429)
        /build/ethereum-YPcqTf/ethereum-1.6.7+build10273+trusty/build/_workspace/src/github.com/ethereum/go-ethereum/consensus/ethash/sealer.go:72 +0x87
created by github.com/ethereum/go-ethereum/consensus/ethash.(*Ethash).Seal
        /build/ethereum-YPcqTf/ethereum-1.6.7+build10273+trusty/build/_workspace/src/github.com/ethereum/go-ethereum/consensus/ethash/sealer.go:73 +0x1d7

goroutine 987711 [select]:
github.com/ethereum/go-ethereum/event.(*TypeMuxSubscription).deliver(0xc4246df380, 0xc42b363350)
        /build/ethereum-YPcqTf/ethereum-1.6.7+build10273+trusty/build/_workspace/src/github.com/ethereum/go-ethereum/event/event.go:207 +0x17a
github.com/ethereum/go-ethereum/event.(*TypeMux).Post(0xc42009f890, 0xdb4240, 0xc4a0120000, 0x89e9cb, 0x13084b8)
        /build/ethereum-YPcqTf/ethereum-1.6.7+build10273+trusty/build/_workspace/src/github.com/ethereum/go-ethereum/event/event.go:97 +0x1a5
created by github.com/ethereum/go-ethereum/core.(*TxPool).promoteTx
        /build/ethereum-YPcqTf/ethereum-1.6.7+build10273+trusty/build/_workspace/src/github.com/ethereum/go-ethereum/core/tx_pool.go:552 +0x2ac

goroutine 987985 [runnable]:
github.com/ethereum/go-ethereum/consensus/ethash.(*Ethash).Seal.func1(0xc442aae1a0, 0xc422b5d380, 0xc453f4b5f0, 0xc45c854420, 0xc45c854480, 0x23, 0x5c1bafd9f9167586)
        /build/ethereum-YPcqTf/ethereum-1.6.7+build10273+trusty/build/_workspace/src/github.com/ethereum/go-ethereum/consensus/ethash/sealer.go:70
created by github.com/ethereum/go-ethereum/consensus/ethash.(*Ethash).Seal
        /build/ethereum-YPcqTf/ethereum-1.6.7+build10273+trusty/build/_workspace/src/github.com/ethereum/go-ethereum/consensus/ethash/sealer.go:73 +0x1d7
goroutine 913500 [IO wait]:
net.runtime_pollWait(0x7f796c139078, 0x72, 0x33)
        /usr/lib/go-1.8/src/runtime/netpoll.go:164 +0x59
net.(*pollDesc).wait(0xc44e833b18, 0x72, 0x16d78e0, 0x16d0a98)
        /usr/lib/go-1.8/src/net/fd_poll_runtime.go:75 +0x38
net.(*pollDesc).waitRead(0xc44e833b18, 0xc429637200, 0x20)
        /usr/lib/go-1.8/src/net/fd_poll_runtime.go:80 +0x34
net.(*netFD).Read(0xc44e833ab0, 0xc429637200, 0x20, 0x20, 0x0, 0x16d78e0, 0x16d0a98)
        /usr/lib/go-1.8/src/net/fd_unix.go:250 +0x1b7
net.(*conn).Read(0xc4506ba048, 0xc429637200, 0x20, 0x20, 0x0, 0x0, 0x0)
        /usr/lib/go-1.8/src/net/net.go:181 +0x70
io.ReadAtLeast(0x7f796443a0e8, 0xc4506ba048, 0xc429637200, 0x20, 0x20, 0x20, 0xe87ae0, 0x5c2100, 0x7f796443a0e8)
        /usr/lib/go-1.8/src/io/io.go:307 +0xa9
io.ReadFull(0x7f796443a0e8, 0xc4506ba048, 0xc429637200, 0x20, 0x20, 0x20, 0xed0f8c3ed, 0xc410c999b7)
        /usr/lib/go-1.8/src/io/io.go:325 +0x58
github.com/ethereum/go-ethereum/p2p.(*rlpxFrameRW).ReadMsg(0xc42263c420, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x200, 0xf1)
        /build/ethereum-YPcqTf/ethereum-1.6.7+build10273+trusty/build/_workspace/src/github.com/ethereum/go-ethereum/p2p/rlpx.go:628 +0x12b
github.com/ethereum/go-ethereum/p2p.(*rlpx).ReadMsg(0xc42496b4a0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0)
        /build/ethereum-YPcqTf/ethereum-1.6.7+build10273+trusty/build/_workspace/src/github.com/ethereum/go-ethereum/p2p/rlpx.go:89 +0x165
github.com/ethereum/go-ethereum/p2p.(*Peer).readLoop(0xc4558dfa90, 0xc4cf9bdc20)
        /build/ethereum-YPcqTf/ethereum-1.6.7+build10273+trusty/build/_workspace/src/github.com/ethereum/go-ethereum/p2p/peer.go:212 +0x7f
created by github.com/ethereum/go-ethereum/p2p.(*Peer).run
        /build/ethereum-YPcqTf/ethereum-1.6.7+build10273+trusty/build/_workspace/src/github.com/ethereum/go-ethereum/p2p/peer.go:151 +0xf2
[signal SIGSEGV: segmentation violation code=0x1 addr=0x7f78e3a2a088 pc=0x461077]

goroutine 653899 [running]:
runtime.throw(0xeae113, 0x5)
        /usr/lib/go-1.8/src/runtime/panic.go:596 +0x95 fp=0xc4240cb9f0 sp=0xc4240cb9d0
runtime.sigpanic()
        /usr/lib/go-1.8/src/runtime/signal_unix.go:297 +0x28c fp=0xc4240cba40 sp=0xc4240cb9f0
runtime.memmove(0xc44e92c200, 0x7f78e3a2a088, 0x40)
        /usr/lib/go-1.8/src/runtime/memmove_amd64.s:184 +0x657 fp=0xc4240cba48 sp=0xc4240cba40
github.com/ethereum/go-ethereum/consensus/ethash.hashimoto(0xc4240cbd78, 0x20, 0x20, 0x6ebea2b1ceda2bb4, 0x3ffff880, 0xc422c37940, 0xc4240cbc48, 0x40b914, 0xd3a7c0, 0xc4283063c0, ...)
        /build/ethereum-YPcqTf/ethereum-1.6.7+build10273+trusty/build/_workspace/src/github.com/ethereum/go-ethereum/consensus/ethash/algorithm.go:312 +0x36e fp=0xc4240cbbe0 sp=0xc4240cba48
github.com/ethereum/go-ethereum/consensus/ethash.hashimotoFull(0x7f78bc000008, 0xffffe20, 0xffffe20, 0xc4240cbd78, 0x20, 0x20, 0x6ebea2b1ceda2bb4, 0xc422c378e0, 0x20, 0x20, ...)
        /build/ethereum-YPcqTf/ethereum-1.6.7+build10273+trusty/build/_workspace/src/github.com/ethereum/go-ethereum/consensus/ethash/algorithm.go:355 +0xbe fp=0xc4240cbc58 sp=0xc4240cbbe0
github.com/ethereum/go-ethereum/consensus/ethash.(*Ethash).mine(0xc422b5d380, 0xc4574ea3f0, 0x18, 0x6ebea2b1c6734e54, 0xc4283063c0, 0xc428306420)
        /build/ethereum-YPcqTf/ethereum-1.6.7+build10273+trusty/build/_workspace/src/github.com/ethereum/go-ethereum/consensus/ethash/sealer.go:130 +0x451 fp=0xc4240cbf68 sp=0xc4240cbc58
github.com/ethereum/go-ethereum/consensus/ethash.(*Ethash).Seal.func1(0xc434ac8c30, 0xc422b5d380, 0xc4574ea3f0, 0xc4283063c0, 0xc428306420, 0x18, 0x6ebea2b1c6734e54)
        /build/ethereum-YPcqTf/ethereum-1.6.7+build10273+trusty/build/_workspace/src/github.com/ethereum/go-ethereum/consensus/ethash/sealer.go:72 +0x87 fp=0xc4240cbfa8 sp=0xc4240cbf68
runtime.goexit()
        /usr/lib/go-1.8/src/runtime/asm_amd64.s:2197 +0x1 fp=0xc4240cbfb0 sp=0xc4240cbfa8
created by github.com/ethereum/go-ethereum/consensus/ethash.(*Ethash).Seal
        /build/ethereum-YPcqTf/ethereum-1.6.7+build10273+trusty/build/_workspace/src/github.com/ethereum/go-ethereum/consensus/ethash/sealer.go:73 +0x1d7

[karalabe]

We kind of need the beginning of the logs especially, that contains the crashing goroutine. I've been running Geth for quite extensively (weeks without restarts) and didn't see an issue until now. If it happens again, could you get us the first few hundred lines of the crash log? That's the most interesting part.

[linas]

This appears to be identical to #14552

[sstelfox]

I've had this trip on me a couple times over the past few days. I've dumped the log in its entirety from the process in this gist. I've been running the docker alpine version posted publicly, and I've been starting it up with this command:

docker run -d --name ethereum-node -v $HOME/ethereum:/root \
  -p 127.0.0.1:8545:8545 -p 127.0.0.1:30303:30303 ethereum/client-go:alpine \
  --fast --cache=2048 --rpc

Edit: I realized a gist isn't large enough to hold the entire log, so it's been split continuously over three files, all still in the same gist.

[linas]

@sstelfox it might be more useful to update #14552 which is the same bug, so that conversation is tracked in one place, instead of several.

[sstelfox]

@linas Sorry I looked at all of them and this was the most recently updated one. Assumptions make fools of us all :) I appreciate you bumping the other thread with this information.

[adamschmideg]

Closing this issue being the same as #14552