eth/downloader: interrupted state download causes sync to restart from block 0

[fjl]

Geth version: 1.3.6
OS & Version: OS X 10.11

I tried out the Wallet App with a friend and the progress bar reached the
latest block. Then the sync suddenly restarted from block 0.

Log

I0417 19:44:35.201690   55847 downloader.go:288] Block synchronisation started
I0417 19:44:38.860173   55847 blockchain.go:889] imported 192 header(s) (0 ignored) in 765.481174ms. #192 [ad474131… / cc25a4ae…]
I0417 19:44:38.877690   55847 blockchain.go:1044] imported 192 receipt(s) (0 ignored) in 10.316723ms. #192 [ad474131… / cc25a4ae…]
I0417 19:44:41.505509   55847 blockchain.go:889] imported 192 header(s) (0 ignored) in 22.273854ms. #384 [313c47b7… / 78068943…]
I0417 19:44:41.520639   55847 blockchain.go:1044] imported 131 receipt(s) (0 ignored) in 9.599827ms. #323 [313c47b7… / 6de28537…]
...
I0417 20:18:59.018752   55847 blockchain.go:889] imported 192 header(s) (0 ignored) in 184.933508ms. #772733 [9139dccc… / 61683092…]
I0417 20:18:59.025051   55847 blockchain.go:1044] imported 7 receipt(s) (0 ignored) in 974.11µs. #772548 [9139dccc… / 3fef7288…]
I0417 20:18:59.148519   55847 blockchain.go:1044] imported 185 receipt(s) (0 ignored) in 29.431894ms. #772733 [5cb522d0… / 61683092…]
I0417 20:18:59.327091   55847 blockchain.go:889] imported 192 header(s) (0 ignored) in 166.953683ms. #772925 [d339b635… / 0657e10a…]
I0417 20:18:59.330315   55847 blockchain.go:1044] imported 1 receipt(s) (0 ignored) in 227.4µs. #772734 [d339b635… / d339b635…]
I0417 20:18:59.419339   55847 blockchain.go:1044] imported 8 receipt(s) (0 ignored) in 6.987842ms. #772742 [22367599… / a2efbb71…]
I0417 20:18:59.645338   55847 blockchain.go:889] imported 192 header(s) (0 ignored) in 179.484956ms. #773117 [6756c030… / 1b2d400d…]
I0417 20:18:59.867927   55847 downloader.go:1340] imported 1 state entries in 570.544µs: processed 1 in total
I0417 20:19:00.089453   55847 downloader.go:1340] imported 1 state entries in 366.67µs: processed 2 in total
I0417 20:19:00.264475   55847 downloader.go:1340] imported 1 state entries in 656.467µs: processed 3 in total
I0417 20:19:00.601459   55847 downloader.go:1340] imported 1 state entries in 375.182µs: processed 4 in total
I0417 20:19:00.982615   55847 downloader.go:1340] imported 1 state entries in 403.569µs: processed 5 in total
I0417 20:19:01.353471   55847 downloader.go:1340] imported 1 state entries in 179.582µs: processed 6 in total
I0417 20:19:01.693055   55847 downloader.go:1340] imported 1 state entries in 456.856µs: processed 7 in total
I0417 20:19:02.797873   55847 downloader.go:1102] Rolled back 2048 headers (LH: 773117->771069, FB: 772742->771069, LB: 0->0)
I0417 20:19:04.749200   55847 downloader.go:267] Synchronisation failed: no peers to keep download active
I0417 20:19:17.329294   55847 blockchain.go:1251] imported 192 block(s) (0 queued 0 ignored) including 0 txs in 923.657122ms. #192 [ad474131 / cc25a4ae]
I0417 20:19:17.535908   55847 blockchain.go:1251] imported 206 block(s) (0 queued 0 ignored) including 3 txs in 205.114869ms. #398 [313c47b7 / b6d9eb53]
I0417 20:19:17.796995   55847 blockchain.go:1251] imported 256 block(s) (0 queued 0 ignored) including 1 txs in 256.878836ms. #654 [0705b7da / db169d9a]
I0417 20:19:18.057634   55847 blockchain.go:1251] imported 256 block(s) (0 queued 0 ignored) including 0 txs in 260.067242ms. #910 [88052af7 / 256c19fb]

[karalabe]

This is a security feature, not a bug. From #1889 :

---

Blockchain protocols in general (i.e. Bitcoin, Ethereum, and the others) are susceptible to Sybil attacks, where an attacker tries to completely isolate a node from the rest of the network, making it believe a false truth as to what the state of the real network is. This permits the attacker to spend certain funds in both the real network and this "fake bubble". However, the attacker can only maintain this state as long as it's feeding new valid blocks it itself is forging; and to successfully shadow the real network, it needs to do this with a chain height and difficulty close to the real network. In short, to pull off a successful Sybil attack, the attacker needs to match the network's hash rate, so it's a very expensive attack.

Compared to the classical Sybil attack, fast sync provides such an attacker with an extra ability, that of feeding a node a view of the network that's not only different from the real network, but also that might go around the EVM mechanics. The Ethereum protocol only validates state root hashes by processing all the transactions against the previous state root. But by skipping the transaction processing, we cannot prove that the state root contained within the fast sync pivot point is valid or not, so as long as an attacker can maintain a fake blockchain that's on par with the real network, it could create an invalid view of the network's state.

To avoid opening up nodes to this extra attacker ability, fast sync (beside being solely opt-in) will only ever run during an initial sync (i.e. when the node's own blockchain is empty). After a node managed to successfully sync with the network, fast sync is forever disabled. This way anybody can quickly catch up with the network, but after the node caught up, the extra attack vector is plugged in. This feature permits users to safely use the fast sync flag (--fast), without having to worry about potential state root attacks happening to them in the future. As an additional safety feature, if a fast sync fails close to or after the random pivot point, fast sync is disabled as a safety precaution and the node reverts to full, block-processing based synchronization.

[karalabe]

We could add a few retries so that if state download fails due to no more peers, we give it a few (1-2-3) more trials before marking the entire thing as failed and assuming an attack. We could do this for 1.5 though, as it could be quite an involved change.

[karalabe]

Per the new 1.4.6 release, states are downloaded concurrently with everything else during fast sync so there's a huge window to get all the states before the critical section. Further the critical section has a retry allowance counter of 10 attempts. Hopefully these will solve all our issues in the foreseeable future. We might consider trie sync progress tracking to be even more resilient with the critical section, but that requires updates to the trie/state db code too. We can consider it for 1.5 or above.