Syncing Very Slow Around Block 4.700.000

[agericke]

Before filing a new issue, please provide the following information.

I'm running: Parity

• Which Parity version?:

Parity
  version Parity/v1.11.3-beta-a66e36b-20180605/x86_64-linux-gnu/rustc1.26.1
Copyright 2015, 2016, 2017, 2018 Parity Technologies (UK) Ltd
License GPLv3+: GNU GPL version 3 or later <http://gnu.org/licenses/gpl.html>.
This is free software: you are free to change and redistribute it.
There is NO WARRANTY, to the extent permitted by law.

By Wood/Paronyan/Kotewicz/Drwięga/Volf
   Habermeier/Czaban/Greeff/Gotchac/Redmann

• Which operating system?: Linux
• How installed?: binaries
• Are you fully synchronized?: no
• Which network are you connected to?: ethereum
• Did you try to restart the node?: yes

My parity node has been syncing for more than two weeks more or less. Initially it was syncing at high speeds, but during the last week speed has been very slow. In the 31 of june i was synchronized until block 4480000, and my node was processing the data at speeds around 4-8 Mgas/s.

Before, i was processing at speeds around 13-14 Mgas/s, but since 31 of december this speed began to decrease and now i am having speeds of 1 Mgas/s or even 0 Mgas/s. Today, 6 of June, my node is synchronized until block 4.760.000. So during the last week, i have been able to synchronize only 300K blocks.

I have initilize parity with the following options:

parity --pruning archive --tracing on

I need all the data, so i cannot synchronize in a fast or light way.
I have tried to change the --cache-size option to 1024, 2048, and 4096 but had no improvement. Normally, I have more than 25 connections to others peers, although i usually do not have any active connection.

I have an SSD drive, and connection speeds are 89 Mbps for download and 78 Mbps for Upload.

I have tried doing all actions in https://wiki.parity.io/FAQ-Basic-Operations,-Configuration,-and-Synchronization.html#what-can-i-do-when-parity-has-trouble-getting-in-sync, but could not solve the problem.

Time is correct with a difference of -0.011 seconds.

I attach here some images of the problem. In parityNode.log you can find the output with sync=trace option.

parityNode.log

[Tbaut]

Thanks for the detailed report. Can you monitor the resources of the computer to see where the bottleneck is? (how much RAM, CPU, IOPS used).
The latest blocks (from 5M) are heavier so this situation will probably not get better on your side :/

[santtu]

@agericke I've done a full archival sync, and while it took several weeks to complete, yours seems quite excessive. I understand you have been running the node since at least from start of the year, right?

As @Tbaut asked, you should check your machine metrics. Are you sure the system is not spending time in iowait? E.g. check top/vmstat and see how much io% is (should not be an issue with SSD, but you never know). Another possibility that I know is that recent parity versions appear to be leaking memory (#8618) or sockets (#8774), so your system might be trashing --- are you running out of memory, or is swap being heavily used, and have you restarted parity recently?

[5chdn]

@agericke do you see the same behavior on 1.10.6?

[MatthiasEgli-chainsecurity]

We are seeing similar issues using a very powerful machine (8 cores, 32GB memory, 4x NVMe SSDs with performance-optimized ZFS setup). CPU usage is around 50% on one core, mostly waiting for IO even though IO is very fast. We see 7-8 MGas and ~200 Tx/s with this, which is still much slower compared to previous blocks where we had >200 MGas and corresponding higher number of transactions. All caches configured aren't used fully by Parity, so it doesn't look like this would be the issue.

Looking at vmstat, we are seeing ~250'000 bi operations, so heavy disk reads. We are going to take a closer look using perf tools to see what kind of operations are causing this slowdown.

[briansoule]

+1 also experiencing similar issues in a datacenter
Syncing #4727543 0x6ee3…5a52 0 blk/s 70 tx/s 3 Mgas/s 0+ 389 Qed #4727934 26/50 peers 6 MiB chain 18 MiB db 46 MiB queue 15 MiB sync RPC: 0 conn, 0 req/s, 0 µs

[5chdn]

Could anyone confirm this is not happening with 1.10.7?

[5chdn]

That's very likely related to CryptoKitties congestion. https://qz.com/1145833/cryptokitties-is-causing-ethereum-network-congestion/

This is a very huge contract and would explain why it requires so much read operations and slows down the synchronization, especially on archive nodes significantly. Question remains: Is this worse compared to older versions or Parity?

[Tbaut]

closing as stale.