Panic after manny days of running

[tatowicz]

thread 'IO Worker #0' panicked at 'Potential DB corruption encountered: Database missing expected key: 9852…245a', ethcore/src/state.rs:352
stack backtrace:
1: 0x55613fddef4f -
2: 0x55613fde518b -
3: 0x55613fde4e13 -
4: 0x55613fdca57d -
5: 0x55613fde53d1 -
6: 0x55613fdcb6ba -
7: 0x55613f7cf502 -
8: 0x55613f7b53ca -
9: 0x55613f7b54c9 -
10: 0x55613f910954 -
11: 0x55613f90fc52 -
12: 0x55613f78f3f6 -
13: 0x55613f7c18d0 -
14: 0x55613f80b55f -
15: 0x55613fdef45b -
16: 0x55613fdef3fe -
17: 0x55613f80cbd7 -
18: 0x55613fde3884 -
19: 0x7ff202177183 - start_thread
20: 0x7ff202b9337c - clone
21: 0x0 -

Not sure how much you can do with this info, but I though I would make you aware.

[slothbag]

Similar issue https://github.com/ethcore/parity/issues/1900

[keorn]

@Anthony-Tatowicz Which version of Parity are you running? Did you use the new parity restore <snapshot> feature to get synced?

[elaineo]

Also encountered this issue with version Parity/v1.3.3-beta/x86_64-linux-gnu/rustc1.12.0

not using restore

[pyskell]

Me and another user are also encountering this issue. Going to attempt the restore feature and will report back on that.

However this seems to be a breaking bug as fully syncing directly from the Ethereum Classic chain does not seem to work at the moment.

Let me know if there's any other logs or information which would be helpful.

[pyskell]

Nevermind, unable to fully sync, stuck at block 2,421,696.

Each time I start parity I get a slightly different version of the error below, panicking at different expected keys:

thread 'IO Worker #2' panicked at 'Potential DB corruption encountered: Database missing expected key: 1348…1230', ethcore/src/state.rs:629
stack backtrace:
   1:     0x7f3f8de417b9 - <unknown>
   2:     0x7f3f8de4948c - <unknown>
   3:     0x7f3f8de48359 - <unknown>
   4:     0x7f3f8de48a48 - <unknown>
   5:     0x7f3f8de488a2 - <unknown>
   6:     0x7f3f8de48810 - <unknown>
   7:     0x7f3f8da7f5da - <unknown>
   8:     0x7f3f8da01a4f - <unknown>
   9:     0x7f3f8d9c3e50 - <unknown>
  10:     0x7f3f8da37461 - <unknown>
  11:     0x7f3f8da39837 - <unknown>
  12:     0x7f3f8d9ef69a - <unknown>
  13:     0x7f3f8d8ecab5 - <unknown>
  14:     0x7f3f8de50f76 - <unknown>
  15:     0x7f3f8d94da3e - <unknown>
  16:     0x7f3f8de46ff2 - <unknown>
  17:     0x7f3f8c5830a3 - start_thread
  18:     0x7f3f8cf9387c - clone
  19:                0x0 - <unknown>
2016-10-14 13:07:16  Finishing work, please wait...

[aordine]

I experience the same problem. The system is Ubuntu 16.04.1 LTS (GNU/Linux 4.4.0-38-generic x86_64) with 2G RAM and 4G swap.

root@aordine-nyc1:~# parity -j --db-path /mnt/volume-nyc1-02/.parity/ --cache-size-db 1024
Option '--jsonrpc' does nothing. It's on by default
2016-10-18 09:47:50 Starting Parity/v1.3.8-beta/x86_64-linux-gnu/rustc1.12.0
2016-10-18 09:47:50 Using state DB journalling strategy fast
2016-10-18 09:47:50 Configured for Frontier/Homestead using Ethash engine
2016-10-18 09:47:56 Public node URL: enode://2f950592f5556e0f26ce63d1e146866411cdfefe823e22f2166340456981f4421628304085e2d2d6cc7e0cf8285
1852f747a69bfae951ab3059544759e12bd7e@198.199.78.245:30303
thread 'IO Worker #3' panicked at 'Potential DB corruption encountered: Database missing expected key: 997d…f777', ethcore/src/state.rs:6
29
stack backtrace:
1: 0x55735cef5449 -
2: 0x55735cefd11c -
3: 0x55735cefbfe9 -
4: 0x55735cefc6d8 -
5: 0x55735cefc532 -
6: 0x55735cefc4a0 -
7: 0x55735cb332ca -
8: 0x55735cb46770 -
9: 0x55735cb37562 -
10: 0x55735ca8d90f -
11: 0x55735ca91afb -
12: 0x55735ca9217e -
13: 0x55735ca9d8d0 -
14: 0x55735ca9ed72 -
15: 0x55735c998555 -
16: 0x55735cf04c06 -
17: 0x55735ca03efe -
18: 0x55735cefac82 -
19: 0x7f88a70cd6f9 - start_thread
20: 0x7f88a7b73b5c - clone
21: 0x0 -
2016-10-18 09:47:58 Stage 3 block verification failed for #2422460 (ea76…d42b)
Error: Block(UnknownParent(44d048d5e4ac7bff897a0f529e065f93cc08de001118358f4975a046ebf90f76))
2016-10-18 09:47:58 Finishing work, please wait...

[keorn]

@aordine @pyskell This issue is about instability of long running Parity, the bug you are referring to is covered by issue: https://github.com/ethcore/parity/issues/2603
Since @Anthony-Tatowicz has not responded for a while with details and the bug is similar to https://github.com/ethcore/parity/issues/2603, lets close this for now.

[pyskell]

@keorn Can we mark it as a duplicate then? There's quite a bit of worthwhile output in this thread that can help solve whatever the issue may be.

This doesn't seem to have to do specifically with many days of runtime as even after restarting parity, or attempting to sync a new copy of the chain from the network, the same issue is encountered. So even a brand new machine, running the latest version of parity, will be unable to sync to either network. Even using the newer parity restore <snapshot> does not work (my earlier comment was in error). The only thing that has worked is fully downloading another user's blockchain.

I have a working copy of the blockchain here (courtesy of another user) if it can be of any use debugging: full parity copy

It's also worth noting that the blocks parity seems to have issues with (around 2,420,000 on ETC) come around the time that the recent DoS-style exploit in the ethereum protocol occurred.

[keorn]

Since I referred to the other issue, this issue shows up there.
Thanks for more info! Please put any additional in #2603.

As for the problematic blocks, unfortunately they will always be heavy on the syncing nodes. They just contain transactions which are expensive to process and there is not much way around it. The issue #2603 also appeared during the attacks, which could be linked to the necessary heavy i/o.

[aordine]

At this time, parity 1.3.8 went through the EIP 150 hard fork block
successfully. And then massively slowed down, processing one block in 30
sec to 1 minute. I stopped it at block 2472474 to export a backup. Not sure
what contributed to the change.

On Tue, Oct 18, 2016 at 5:40 PM, keorn notifications@github.com wrote:

Since I referred to the other issue, this issue shows up there.
Thanks for more info! Please put any additional in #2603
https://github.com/ethcore/parity/issues/2603.

As for the problematic blocks, unfortunately they will always be heavy on
the syncing nodes. They just contain transactions which are expensive to
process and there is not much way around it. The issue #2603
https://github.com/ethcore/parity/issues/2603 also appeared during the
attacks, which could be linked to the necessary heavy i/o.

—
You are receiving this because you were mentioned.
Reply to this email directly, view it on GitHub
https://github.com/ethcore/parity/issues/1961#issuecomment-254647631,
or mute the thread
https://github.com/notifications/unsubscribe-auth/ARu8LgYF5mzDJhUWmfjQCQZ6I2PynzLiks5q1TzQgaJpZM4JoFgT
.