Skip to content
This repository has been archived by the owner on Nov 6, 2020. It is now read-only.

parity just hangs in the background after running for some time #2523

Closed
gituser opened this issue Oct 7, 2016 · 18 comments
Closed

parity just hangs in the background after running for some time #2523

gituser opened this issue Oct 7, 2016 · 18 comments
Assignees
@tomusdrw
Labels
M4-core ⛓ Core client code / Rust. Z0-unconfirmed 🤔 Issue might be valid, but it’s not yet known.

Comments

@gituser
Copy link

gituser commented Oct 7, 2016 •
edited
Loading

Hi.

Been investigating this issue with @tomusdrw but without any luck, maybe someone else encountered this as well.

Parity hangs after some time. Sometimes it can run for 5 hours without issues, sometimes 10 hours, sometimes 24 hours. But every time running time is different. On the restart it starts syncing again without problems.

I run parity version version Parity/v1.3.4-beta-a8b2cf9-20161006/x86_64-linux-gnu/rustc1.12.0 (beta + ported fresh rocksDB commit - ethcore/parity@e380955.

Before that I've tried plain beta version Parity/v1.3.4-beta-50021c7-20161005/x86_64-linux-gnu/rustc1.12.0 same situation.

I also ran parity without any load in fresh VM and it hanged there as well with same symptoms.

On the contrary v1.3.1 is running just fine in other VM.

All VMs are identical and running Debian Jessie 8.0 x64.

The worst of the situation parity stops responding to IPC / RPC but still hangs in the background. There is no crash or anything it just stops altogether, but still running and you can only kill it with kill -9.

ps output:

ethereum 18993 13.8 19.2 3257072 2022640 ?     Sl   11:33  51:29 /home/ethereum/parity daemon /var/run/parity_product.pid --identity parity_ethdb --chain=frontier --mode=passive --mode-timeout 30 --mode-alarm 300 --cache-size 2048 -l sync=trace --log-file /home/ethereum/.parity/product.log --datadir /home/ethereum/.parity --port 50310 --jsonrpc-interface 127.0.0.1 --jsonrpc-port 9110 --jsonrpc-apis web3,eth,net,ethcore,personal,traces,rpc --ipc-path /home/ethereum/.parity/parity.ipc --no-dapps --no-signer --tx-gas-limit 1500000 --db-path /home/ethereum/.parity --keys-path /home/ethereum/.parity/keys --signer-path /home/ethereum/.parity/signer --dapps-path /home/ethereum/.parity/dapps

strace output: https://paste.sh/0XCbZnHK#8WLrI9BqLQvFvQi9UudoYalu

Could be something related to time, but I have ntpd set and why it doesn't reproduce on v1.3.1?
On all my server I'm running ntpd to adjust time from my ISP time servers.

gdb threads:

Thread 20 (Thread 0x7f858e1ff700 (LWP 18994)):
#0  0x00007f8593b3ac03 in epoll_wait () from /lib/x86_64-linux-gnu/libc.so.6
#1  0x00007f859478e377 in mio::poll::Poll::poll::he70ab65962172840 ()
#2  0x00007f859450b677 in _$LT$ethcore_io..service..IoManager$LT$Message$GT$$GT$::start::he8bdc11768ead116 ()
#3  0x00007f859449a51c in std::panicking::try::do_call::hb6d45f6290beac63 ()
#4  0x00007f8594a0dce7 in __rust_maybe_catch_panic ()
#5  0x00007f85944f9162 in _$LT$F$u20$as$u20$alloc..boxed..FnBox$LT$A$GT$$GT$::call_box::hf1605b4a580d217c ()
#6  0x00007f8594a03d63 in std::sys::thread::Thread::new::thread_start::h4c0ad33b336bc6ea ()
#7  0x00007f859312a0a4 in start_thread () from /lib/x86_64-linux-gnu/libpthread.so.0
#8  0x00007f8593b3a62d in clone () from /lib/x86_64-linux-gnu/libc.so.6

Thread 19 (Thread 0x7f858dbff700 (LWP 18995)):
#0  0x00007f859312e08f in pthread_cond_wait@@GLIBC_2.3.2 () from /lib/x86_64-linux-gnu/libpthread.so.0
#1  0x00007f859481e979 in parking_lot::parking_lot::park::h3f0f2856450906df ()
#2  0x00007f859481f51d in parking_lot::raw_rwlock::RawRwLock::lock_shared_slow::h1a93a7ea5b207a23 ()
#3  0x00007f8594804e1b in ethcore_util::kvdb::Database::get::h46e319c8c073b457 ()
#4  0x00007f85947fa8af in _$LT$ethcore_util..journaldb..overlayrecentdb..OverlayRecentDB$u20$as$u20$ethcore_util..hashdb..HashDB$GT$::get::h2757dd02b55e6ba5 ()
#5  0x00007f859480aae2 in ethcore_util::trie::triedb::TrieDB::get_raw_or_lookup::hb4ef0ffbe4a5b40d ()
#6  0x00007f859480a9c8 in ethcore_util::trie::triedb::TrieDB::get_from_node::h59e071a80b61e9e3 ()
#7  0x00007f859480aa27 in ethcore_util::trie::triedb::TrieDB::get_from_node::h59e071a80b61e9e3 ()
#8  0x00007f859480aa27 in ethcore_util::trie::triedb::TrieDB::get_from_node::h59e071a80b61e9e3 ()
#9  0x00007f8594817072 in _$LT$ethcore_util..trie..sectriedb..SecTrieDB$LT$$u27$db$GT$$u20$as$u20$ethcore_util..trie..Trie$GT$::get::h160be0eed6cbb7a3 ()
#10 0x00007f8594624690 in ethcore::state::State::nonce::h8035124f80d305ad ()
#11 0x00007f85945a7ed0 in ethcore::client::traits::BlockChainClient::latest_nonce::h02936665a77130fb ()
#12 0x00007f859456a9ac in _$LT$core..iter..Map$LT$I$C$$u20$F$GT$$u20$as$u20$core..iter..iterator..Iterator$GT$::next::h773c10e865cdc33b ()
#13 0x00007f85945dd1a0 in ethcore::miner::miner::Miner::add_transactions_to_queue::hc3e05bcc3ebfbac9 ()
#14 0x00007f85945df328 in _$LT$ethcore..miner..miner..Miner$u20$as$u20$ethcore..miner..MinerService$GT$::import_external_transactions::hd1f991eebc00aa3d ()
#15 0x00007f85945961ab in ethcore::client::client::Client::import_queued_transactions::hc197a686fdea5ed1 ()
#16 0x00007f8594494d16 in std::panicking::try::do_call::h2b36c1a2f5abb3dd ()
#17 0x00007f8594a0dce7 in __rust_maybe_catch_panic ()
#18 0x00007f85944f80bf in _$LT$F$u20$as$u20$alloc..boxed..FnBox$LT$A$GT$$GT$::call_box::h4241de757127c6a1 ()
#19 0x00007f8594a03d63 in std::sys::thread::Thread::new::thread_start::h4c0ad33b336bc6ea ()
#20 0x00007f859312a0a4 in start_thread () from /lib/x86_64-linux-gnu/libpthread.so.0
#21 0x00007f8593b3a62d in clone () from /lib/x86_64-linux-gnu/libc.so.6

Thread 18 (Thread 0x7f858d9fe700 (LWP 18996)):
#0  0x00007f859312e08f in pthread_cond_wait@@GLIBC_2.3.2 () from /lib/x86_64-linux-gnu/libpthread.so.0
#1  0x00007f859481e979 in parking_lot::parking_lot::park::h3f0f2856450906df ()
#2  0x00007f859481f154 in parking_lot::raw_mutex::RawMutex::lock_slow::h3e50386fffba6e3a ()
#3  0x00007f85945df2ee in _$LT$ethcore..miner..miner..Miner$u20$as$u20$ethcore..miner..MinerService$GT$::import_external_transactions::hd1f991eebc00aa3d ()
#4  0x00007f85945961ab in ethcore::client::client::Client::import_queued_transactions::hc197a686fdea5ed1 ()
#5  0x00007f8594494d16 in std::panicking::try::do_call::h2b36c1a2f5abb3dd ()
#6  0x00007f8594a0dce7 in __rust_maybe_catch_panic ()
#7  0x00007f85944f80bf in _$LT$F$u20$as$u20$alloc..boxed..FnBox$LT$A$GT$$GT$::call_box::h4241de757127c6a1 ()
#8  0x00007f8594a03d63 in std::sys::thread::Thread::new::thread_start::h4c0ad33b336bc6ea ()
#9  0x00007f859312a0a4 in start_thread () from /lib/x86_64-linux-gnu/libpthread.so.0
#10 0x00007f8593b3a62d in clone () from /lib/x86_64-linux-gnu/libc.so.6

Thread 17 (Thread 0x7f858d7fd700 (LWP 18997)):
#0  0x00007f859312e08f in pthread_cond_wait@@GLIBC_2.3.2 () from /lib/x86_64-linux-gnu/libpthread.so.0
#1  0x00007f8594994e5d in rocksdb::port::CondVar::Wait() ()
#2  0x00007f8594972988 in rocksdb::InstrumentedCondVar::Wait() ()
#3  0x00007f8594898fe6 in rocksdb::DBImpl::DelayWrite(unsigned long) ()
#4  0x00007f85948c8039 in rocksdb::DBImpl::WriteImpl(rocksdb::WriteOptions const&, rocksdb::WriteBatch*, rocksdb::WriteCallback*, unsigned long*, unsigned long, bool) ()
#5  0x00007f85948c92f3 in rocksdb::DBImpl::Write(rocksdb::WriteOptions const&, rocksdb::WriteBatch*) [clone .localalias.1574] ()
#6  0x00007f85948909de in rocksdb_write ()
#7  0x00007f859488c91c in rocksdb::rocksdb::DB::write_opt::hfd7884461ac90e84 ()
#8  0x00007f85948033e8 in ethcore_util::kvdb::Database::flush::h73a8cdd4c3eb5626 ()
#9  0x00007f8594594f59 in ethcore::client::client::Client::import_verified_blocks::had5931574a418431 ()
#10 0x00007f8594494d16 in std::panicking::try::do_call::h2b36c1a2f5abb3dd ()
#11 0x00007f8594a0dce7 in __rust_maybe_catch_panic ()
#12 0x00007f85944f80bf in _$LT$F$u20$as$u20$alloc..boxed..FnBox$LT$A$GT$$GT$::call_box::h4241de757127c6a1 ()
#13 0x00007f8594a03d63 in std::sys::thread::Thread::new::thread_start::h4c0ad33b336bc6ea ()
#14 0x00007f859312a0a4 in start_thread () from /lib/x86_64-linux-gnu/libpthread.so.0
#15 0x00007f8593b3a62d in clone () from /lib/x86_64-linux-gnu/libc.so.6

Thread 16 (Thread 0x7f858d3fc700 (LWP 18998)):
#0  0x00007f859312e08f in pthread_cond_wait@@GLIBC_2.3.2 () from /lib/x86_64-linux-gnu/libpthread.so.0
#1  0x00007f859481e979 in parking_lot::parking_lot::park::h3f0f2856450906df ()
#2  0x00007f859481f154 in parking_lot::raw_mutex::RawMutex::lock_slow::h3e50386fffba6e3a ()
#3  0x00007f85945df2ee in _$LT$ethcore..miner..miner..Miner$u20$as$u20$ethcore..miner..MinerService$GT$::import_external_transactions::hd1f991eebc00aa3d ()
#4  0x00007f85945961ab in ethcore::client::client::Client::import_queued_transactions::hc197a686fdea5ed1 ()
#5  0x00007f8594494d16 in std::panicking::try::do_call::h2b36c1a2f5abb3dd ()
#6  0x00007f8594a0dce7 in __rust_maybe_catch_panic ()
#7  0x00007f85944f80bf in _$LT$F$u20$as$u20$alloc..boxed..FnBox$LT$A$GT$$GT$::call_box::h4241de757127c6a1 ()
#8  0x00007f8594a03d63 in std::sys::thread::Thread::new::thread_start::h4c0ad33b336bc6ea ()
#9  0x00007f859312a0a4 in start_thread () from /lib/x86_64-linux-gnu/libpthread.so.0
#10 0x00007f8593b3a62d in clone () from /lib/x86_64-linux-gnu/libc.so.6

Thread 15 (Thread 0x7f8586dff700 (LWP 19000)):
#0  0x00007f859312e08f in pthread_cond_wait@@GLIBC_2.3.2 () from /lib/x86_64-linux-gnu/libpthread.so.0
#1  0x00007f8594497c52 in std::panicking::try::do_call::hb44672f9a9a36a28 ()
#2  0x00007f8594a0dce7 in __rust_maybe_catch_panic ()
#3  0x00007f85944f8a7d in _$LT$F$u20$as$u20$alloc..boxed..FnBox$LT$A$GT$$GT$::call_box::h867b6b87cb1e0efe ()
#4  0x00007f8594a03d63 in std::sys::thread::Thread::new::thread_start::h4c0ad33b336bc6ea ()
#5  0x00007f859312a0a4 in start_thread () from /lib/x86_64-linux-gnu/libpthread.so.0
#6  0x00007f8593b3a62d in clone () from /lib/x86_64-linux-gnu/libc.so.6

Thread 14 (Thread 0x7f8586bfe700 (LWP 19001)):
#0  0x00007f859312e08f in pthread_cond_wait@@GLIBC_2.3.2 () from /lib/x86_64-linux-gnu/libpthread.so.0
#1  0x00007f8594497c52 in std::panicking::try::do_call::hb44672f9a9a36a28 ()
#2  0x00007f8594a0dce7 in __rust_maybe_catch_panic ()
#3  0x00007f85944f8a7d in _$LT$F$u20$as$u20$alloc..boxed..FnBox$LT$A$GT$$GT$::call_box::h867b6b87cb1e0efe ()
#4  0x00007f8594a03d63 in std::sys::thread::Thread::new::thread_start::h4c0ad33b336bc6ea ()
#5  0x00007f859312a0a4 in start_thread () from /lib/x86_64-linux-gnu/libpthread.so.0
#6  0x00007f8593b3a62d in clone () from /lib/x86_64-linux-gnu/libc.so.6

Thread 13 (Thread 0x7f85869fd700 (LWP 19002)):
#0  0x00007f859312e08f in pthread_cond_wait@@GLIBC_2.3.2 () from /lib/x86_64-linux-gnu/libpthread.so.0
#1  0x00007f8594497c52 in std::panicking::try::do_call::hb44672f9a9a36a28 ()
#2  0x00007f8594a0dce7 in __rust_maybe_catch_panic ()
#3  0x00007f85944f8a7d in _$LT$F$u20$as$u20$alloc..boxed..FnBox$LT$A$GT$$GT$::call_box::h867b6b87cb1e0efe ()
#4  0x00007f8594a03d63 in std::sys::thread::Thread::new::thread_start::h4c0ad33b336bc6ea ()
#5  0x00007f859312a0a4 in start_thread () from /lib/x86_64-linux-gnu/libpthread.so.0
#6  0x00007f8593b3a62d in clone () from /lib/x86_64-linux-gnu/libc.so.6

Thread 12 (Thread 0x7f85867fc700 (LWP 19003)):
#0  0x00007f859312e08f in pthread_cond_wait@@GLIBC_2.3.2 () from /lib/x86_64-linux-gnu/libpthread.so.0
#1  0x00007f8594497c52 in std::panicking::try::do_call::hb44672f9a9a36a28 ()
#2  0x00007f8594a0dce7 in __rust_maybe_catch_panic ()
#3  0x00007f85944f8a7d in _$LT$F$u20$as$u20$alloc..boxed..FnBox$LT$A$GT$$GT$::call_box::h867b6b87cb1e0efe ()
#4  0x00007f8594a03d63 in std::sys::thread::Thread::new::thread_start::h4c0ad33b336bc6ea ()
#5  0x00007f859312a0a4 in start_thread () from /lib/x86_64-linux-gnu/libpthread.so.0
#6  0x00007f8593b3a62d in clone () from /lib/x86_64-linux-gnu/libc.so.6

Thread 11 (Thread 0x7f85865fb700 (LWP 19004)):
#0  0x00007f859312e08f in pthread_cond_wait@@GLIBC_2.3.2 () from /lib/x86_64-linux-gnu/libpthread.so.0
#1  0x00007f8594497c52 in std::panicking::try::do_call::hb44672f9a9a36a28 ()
#2  0x00007f8594a0dce7 in __rust_maybe_catch_panic ()
#3  0x00007f85944f8a7d in _$LT$F$u20$as$u20$alloc..boxed..FnBox$LT$A$GT$$GT$::call_box::h867b6b87cb1e0efe ()
#4  0x00007f8594a03d63 in std::sys::thread::Thread::new::thread_start::h4c0ad33b336bc6ea ()
#5  0x00007f859312a0a4 in start_thread () from /lib/x86_64-linux-gnu/libpthread.so.0
#6  0x00007f8593b3a62d in clone () from /lib/x86_64-linux-gnu/libc.so.6

Thread 10 (Thread 0x7f85863fa700 (LWP 19005)):
#0  0x00007f859312e08f in pthread_cond_wait@@GLIBC_2.3.2 () from /lib/x86_64-linux-gnu/libpthread.so.0
#1  0x00007f8594497c52 in std::panicking::try::do_call::hb44672f9a9a36a28 ()
#2  0x00007f8594a0dce7 in __rust_maybe_catch_panic ()
#3  0x00007f85944f8a7d in _$LT$F$u20$as$u20$alloc..boxed..FnBox$LT$A$GT$$GT$::call_box::h867b6b87cb1e0efe ()
#4  0x00007f8594a03d63 in std::sys::thread::Thread::new::thread_start::h4c0ad33b336bc6ea ()
#5  0x00007f859312a0a4 in start_thread () from /lib/x86_64-linux-gnu/libpthread.so.0
#6  0x00007f8593b3a62d in clone () from /lib/x86_64-linux-gnu/libc.so.6

Thread 9 (Thread 0x7f85847ff700 (LWP 19006)):
#0  0x00007f8593b3ac03 in epoll_wait () from /lib/x86_64-linux-gnu/libc.so.6
#1  0x00007f859478e377 in mio::poll::Poll::poll::he70ab65962172840 ()
#2  0x00007f8594711914 in _$LT$ethcore_io..service..IoManager$LT$Message$GT$$GT$::start::h36cdf2cc85220541 ()
#3  0x00007f859470b6ec in std::panicking::try::do_call::hb9c61a122464756b ()
#4  0x00007f8594a0dce7 in __rust_maybe_catch_panic ()
#5  0x00007f8594710002 in _$LT$F$u20$as$u20$alloc..boxed..FnBox$LT$A$GT$$GT$::call_box::h6317e4e77ad42eb4 ()
#6  0x00007f8594a03d63 in std::sys::thread::Thread::new::thread_start::h4c0ad33b336bc6ea ()
#7  0x00007f859312a0a4 in start_thread () from /lib/x86_64-linux-gnu/libpthread.so.0
#8  0x00007f8593b3a62d in clone () from /lib/x86_64-linux-gnu/libc.so.6

Thread 8 (Thread 0x7f85841ff700 (LWP 19007)):
#0  0x00007f859312e08f in pthread_cond_wait@@GLIBC_2.3.2 () from /lib/x86_64-linux-gnu/libpthread.so.0
#1  0x00007f859481e979 in parking_lot::parking_lot::park::h3f0f2856450906df ()
#2  0x00007f859481f2f4 in parking_lot::raw_rwlock::RawRwLock::lock_exclusive_slow::hea0ddbd7ae7c26ca ()
#3  0x00007f8594464199 in _$LT$ethsync..api..SyncProtocolHandler$u20$as$u20$ethcore_network..NetworkProtocolHandler$GT$::read::h15e2229c8925a50a ()
#4  0x00007f859472b3a7 in _$LT$ethcore_network..host..Host$u20$as$u20$ethcore_io..IoHandler$LT$ethcore_network..host..NetworkIoMessage$GT$$GT$::stream_readable::he52fe4600b5b6b29 ()
#5  0x00007f859470aadd in std::panicking::try::do_call::h242fbf7fce9c0878 ()
#6  0x00007f8594a0dce7 in __rust_maybe_catch_panic ()
#7  0x00007f859470fd0f in _$LT$F$u20$as$u20$alloc..boxed..FnBox$LT$A$GT$$GT$::call_box::h3dc7f809c9c00e26 ()
#8  0x00007f8594a03d63 in std::sys::thread::Thread::new::thread_start::h4c0ad33b336bc6ea ()
#9  0x00007f859312a0a4 in start_thread () from /lib/x86_64-linux-gnu/libpthread.so.0
#10 0x00007f8593b3a62d in clone () from /lib/x86_64-linux-gnu/libc.so.6

Thread 7 (Thread 0x7f8583ffe700 (LWP 19008)):
#0  0x00007f859312e08f in pthread_cond_wait@@GLIBC_2.3.2 () from /lib/x86_64-linux-gnu/libpthread.so.0
#1  0x00007f859481e979 in parking_lot::parking_lot::park::h3f0f2856450906df ()
#2  0x00007f859481f51d in parking_lot::raw_rwlock::RawRwLock::lock_shared_slow::h1a93a7ea5b207a23 ()
#3  0x00007f8594804e1b in ethcore_util::kvdb::Database::get::h46e319c8c073b457 ()
#4  0x00007f8594652929 in _$LT$ethcore..blockchain..blockchain..BlockChain$u20$as$u20$ethcore..blockchain..blockchain..BlockProvider$GT$::is_known::h9a64c86e8e7b6f84 ()
#5  0x00007f859459b5c0 in _$LT$ethcore..client..client..Client$u20$as$u20$ethcore..client..traits..BlockChainClient$GT$::block_status::hba7f33f54b5877a7 ()
#6  0x00007f859445b789 in ethsync::chain::ChainSync::on_packet::h1ef8e78ca7250902 ()
#7  0x00007f8594463d2d in _$LT$ethsync..api..SyncProtocolHandler$u20$as$u20$ethcore_network..NetworkProtocolHandler$GT$::read::h15e2229c8925a50a ()
#8  0x00007f859472b3a7 in _$LT$ethcore_network..host..Host$u20$as$u20$ethcore_io..IoHandler$LT$ethcore_network..host..NetworkIoMessage$GT$$GT$::stream_readable::he52fe4600b5b6b29 ()
#9  0x00007f859470aadd in std::panicking::try::do_call::h242fbf7fce9c0878 ()
#10 0x00007f8594a0dce7 in __rust_maybe_catch_panic ()
#11 0x00007f859470fd0f in _$LT$F$u20$as$u20$alloc..boxed..FnBox$LT$A$GT$$GT$::call_box::h3dc7f809c9c00e26 ()
#12 0x00007f8594a03d63 in std::sys::thread::Thread::new::thread_start::h4c0ad33b336bc6ea ()
#13 0x00007f859312a0a4 in start_thread () from /lib/x86_64-linux-gnu/libpthread.so.0
#14 0x00007f8593b3a62d in clone () from /lib/x86_64-linux-gnu/libc.so.6

Thread 6 (Thread 0x7f8583dfd700 (LWP 19009)):
#0  0x00007f859312e08f in pthread_cond_wait@@GLIBC_2.3.2 () from /lib/x86_64-linux-gnu/libpthread.so.0
#1  0x00007f859481e979 in parking_lot::parking_lot::park::h3f0f2856450906df ()
#2  0x00007f859481f2f4 in parking_lot::raw_rwlock::RawRwLock::lock_exclusive_slow::hea0ddbd7ae7c26ca ()
#3  0x00007f8594466042 in _$LT$ethsync..api..SyncProtocolHandler$u20$as$u20$ethcore_network..NetworkProtocolHandler$GT$::disconnected::hd1b0c14589cbe5dc ()
#4  0x00007f85947267f1 in ethcore_network::host::Host::kill_connection::h6b85f55685a708f3 ()
#5  0x00007f859472af21 in _$LT$ethcore_network..host..Host$u20$as$u20$ethcore_io..IoHandler$LT$ethcore_network..host..NetworkIoMessage$GT$$GT$::stream_readable::he52fe4600b5b6b29 ()
#6  0x00007f859470aadd in std::panicking::try::do_call::h242fbf7fce9c0878 ()
#7  0x00007f8594a0dce7 in __rust_maybe_catch_panic ()
#8  0x00007f859470fd0f in _$LT$F$u20$as$u20$alloc..boxed..FnBox$LT$A$GT$$GT$::call_box::h3dc7f809c9c00e26 ()
#9  0x00007f8594a03d63 in std::sys::thread::Thread::new::thread_start::h4c0ad33b336bc6ea ()
#10 0x00007f859312a0a4 in start_thread () from /lib/x86_64-linux-gnu/libpthread.so.0
#11 0x00007f8593b3a62d in clone () from /lib/x86_64-linux-gnu/libc.so.6

Thread 5 (Thread 0x7f8583bfc700 (LWP 19010)):
#0  0x00007f859312e08f in pthread_cond_wait@@GLIBC_2.3.2 () from /lib/x86_64-linux-gnu/libpthread.so.0
#1  0x00007f859481e979 in parking_lot::parking_lot::park::h3f0f2856450906df ()
#2  0x00007f859481f2f4 in parking_lot::raw_rwlock::RawRwLock::lock_exclusive_slow::hea0ddbd7ae7c26ca ()
#3  0x00007f8594466460 in _$LT$ethsync..api..SyncProtocolHandler$u20$as$u20$ethcore_network..NetworkProtocolHandler$GT$::timeout::hadd639945f41b05a ()
#4  0x00007f8594731302 in _$LT$ethcore_network..host..Host$u20$as$u20$ethcore_io..IoHandler$LT$ethcore_network..host..NetworkIoMessage$GT$$GT$::timeout::haa8bcda0ac35f3c9 ()
#5  0x00007f859470ab6b in std::panicking::try::do_call::h242fbf7fce9c0878 ()
#6  0x00007f8594a0dce7 in __rust_maybe_catch_panic ()
#7  0x00007f859470fd0f in _$LT$F$u20$as$u20$alloc..boxed..FnBox$LT$A$GT$$GT$::call_box::h3dc7f809c9c00e26 ()
#8  0x00007f8594a03d63 in std::sys::thread::Thread::new::thread_start::h4c0ad33b336bc6ea ()
#9  0x00007f859312a0a4 in start_thread () from /lib/x86_64-linux-gnu/libpthread.so.0
#10 0x00007f8593b3a62d in clone () from /lib/x86_64-linux-gnu/libc.so.6

Thread 4 (Thread 0x7f857d5d6700 (LWP 19011)):
#0  0x00007f8593b3ac03 in epoll_wait () from /lib/x86_64-linux-gnu/libc.so.6
#1  0x00007f859478e377 in mio::poll::Poll::poll::he70ab65962172840 ()
#2  0x00007f85943e9d9b in _$LT$mio..event_loop..EventLoop$LT$H$GT$$GT$::run_once::ha9df8450a25f9dfc ()
#3  0x00007f85943f2ef0 in drop::h544f2a6c2aa4d6dd ()
#4  0x00007f85943e9ab8 in std::panicking::try::do_call::h169770713ec30198 ()
#5  0x00007f8594a0dce7 in __rust_maybe_catch_panic ()
#6  0x00007f85943f49b2 in _$LT$F$u20$as$u20$alloc..boxed..FnBox$LT$A$GT$$GT$::call_box::h0bc806a1cc7a666b ()
#7  0x00007f8594a03d63 in std::sys::thread::Thread::new::thread_start::h4c0ad33b336bc6ea ()
#8  0x00007f859312a0a4 in start_thread () from /lib/x86_64-linux-gnu/libpthread.so.0
#9  0x00007f8593b3a62d in clone () from /lib/x86_64-linux-gnu/libc.so.6

Thread 3 (Thread 0x7f857cfff700 (LWP 19012)):
#0  0x00007f859312e08f in pthread_cond_wait@@GLIBC_2.3.2 () from /lib/x86_64-linux-gnu/libpthread.so.0
#1  0x00007f859481e979 in parking_lot::parking_lot::park::h3f0f2856450906df ()
#2  0x00007f859481f51d in parking_lot::raw_rwlock::RawRwLock::lock_shared_slow::h1a93a7ea5b207a23 ()
#3  0x00007f8594804e1b in ethcore_util::kvdb::Database::get::h46e319c8c073b457 ()
#4  0x00007f8594654163 in _$LT$ethcore..blockchain..blockchain..BlockChain$u20$as$u20$ethcore..blockchain..blockchain..BlockProvider$GT$::block_hash::h1460c01581e91f9b ()
#5  0x00007f859459b41c in _$LT$ethcore..client..client..Client$u20$as$u20$ethcore..client..traits..BlockChainClient$GT$::block::hccf267e15d78f836 ()
#6  0x00007f8594241a0f in _$LT$ethcore_rpc..v1..impls..eth..EthClient$LT$C$C$$u20$S$C$$u20$M$C$$u20$EM$GT$$GT$::block::ha697b98ae5aa69ea ()
#7  0x00007f8594165114 in _$LT$jsonrpc_core..io..DelegateMethod$LT$T$C$$u20$F$GT$$u20$as$u20$jsonrpc_core..commander..MethodCommand$GT$::execute::h3447c0f8742a2ed3 ()
#8  0x00007f85944260a0 in jsonrpc_core::commander::Commander::execute_method::h07090f0af3da92ee ()
#9  0x00007f85944276db in jsonrpc_core::request_handler::RequestHandler::handle_call::h9a1b6bab46d5b7d3 ()
#10 0x00007f8594426951 in jsonrpc_core::request_handler::RequestHandler::handle_request::h1e7aef1c071a496d ()
#11 0x00007f859442a2f5 in jsonrpc_core::io::IoHandler::handle_request::h8dc6996b36c6a966 ()
#12 0x00007f85943e4327 in _$LT$json_ipc_server..nix..RpcServer$u20$as$u20$mio..handler..Handler$GT$::ready::h2b4d0a42f17b705b ()
#13 0x00007f85943dea1b in _$LT$mio..event_loop..EventLoop$LT$H$GT$$GT$::run_once::hc8bf41caa575f792 ()
#14 0x00007f85943de279 in std::panicking::try::do_call::h406cbc3db9e179e3 ()
#15 0x00007f8594a0dce7 in __rust_maybe_catch_panic ()
#16 0x00007f85943df975 in _$LT$F$u20$as$u20$alloc..boxed..FnBox$LT$A$GT$$GT$::call_box::he084c78a0091340c ()
#17 0x00007f8594a03d63 in std::sys::thread::Thread::new::thread_start::h4c0ad33b336bc6ea ()
#18 0x00007f859312a0a4 in start_thread () from /lib/x86_64-linux-gnu/libpthread.so.0
#19 0x00007f8593b3a62d in clone () from /lib/x86_64-linux-gnu/libc.so.6

Thread 2 (Thread 0x7f857cdfe700 (LWP 19013)):
#0  0x00007f8593b0bf2d in nanosleep () from /lib/x86_64-linux-gnu/libc.so.6
#1  0x00007f85949ed7e2 in std::thread::sleep::hc8d0c28ca14ef7ec ()
#2  0x00007f85941e99a1 in std::panicking::try::do_call::hd2d273b687e13de4 ()
#3  0x00007f8594a0dce7 in __rust_maybe_catch_panic ()
#4  0x00007f8594203e01 in _$LT$F$u20$as$u20$alloc..boxed..FnBox$LT$A$GT$$GT$::call_box::h5192c3482a2a8aea ()
#5  0x00007f8594a03d63 in std::sys::thread::Thread::new::thread_start::h4c0ad33b336bc6ea ()
#6  0x00007f859312a0a4 in start_thread () from /lib/x86_64-linux-gnu/libpthread.so.0
#7  0x00007f8593b3a62d in clone () from /lib/x86_64-linux-gnu/libc.so.6

Thread 1 (Thread 0x7f8594010b80 (LWP 18993)):
#0  0x00007f859312e08f in pthread_cond_wait@@GLIBC_2.3.2 () from /lib/x86_64-linux-gnu/libpthread.so.0
#1  0x00007f8594288cc8 in parity::run::execute::h2c782db893c97607 ()
#2  0x00007f859429b009 in parity::execute::h0874b2a12d4cc532 ()
#3  0x00007f85942a73fb in parity::main::h6e8352f588de7935 ()
#4  0x00007f8594a0dce7 in __rust_maybe_catch_panic ()
#5  0x00007f8594a04843 in std::rt::lang_start::h53bf99b0829cc03c ()
#6  0x00007f8593a73b45 in __libc_start_main () from /lib/x86_64-linux-gnu/libc.so.6
#7  0x00007f859411ac7c in _start ()

Last 100 lines of log:

2016-10-07 17:08:02  IO Worker #0 TRACE sync  23 -> GetBlockHeaders (number: 1920000, max: 1, skip: 0, reverse:false)
2016-10-07 17:08:02  IO Worker #0 TRACE sync  23: Returning cached fork header
2016-10-07 17:08:02  IO Worker #0 TRACE sync  -> GetBlockHeaders: returned 1 entries
2016-10-07 17:08:02  IO Worker #0 TRACE sync  23: Confirmed peer
2016-10-07 17:08:04  IO Worker #3 TRACE sync  == Connected 25: Parity/v1.3.1-beta/x86_64-linux-gnu/rustc1.11.0
2016-10-07 17:08:04  IO Worker #3 TRACE sync  == Disconnecting 25: Parity/v1.3.1-beta/x86_64-linux-gnu/rustc1.11.0
2016-10-07 17:08:04  IO Worker #3 TRACE sync  == Connected 92: Parity/v1.3.3-beta/x86_64-linux-gnu/rustc1.12.0
2016-10-07 17:08:05  IO Worker #2 TRACE sync  52 -> NewHashes (6 entries)
2016-10-07 17:08:05  IO Worker #2 TRACE sync  New block hash already in chain 5912178ea50eb7e780031bac4730e94fc030652f874874146e155dcf63dac122
2016-10-07 17:08:05  IO Worker #2 TRACE sync  New block hash already in chain 4d900983c3c22a6d2d728ab063710f0c417969392fbfa3498baaad7d77e96471
2016-10-07 17:08:05  IO Worker #2 TRACE sync  New unknown block hash 481d89d4368675ce304bbf9dc9fe4ebd3dd9ce784079cfeb6149cffc29412e02
2016-10-07 17:08:05  IO Worker #2 TRACE sync  Downloading blocks for new hashes
2016-10-07 17:08:05  IO Worker #2 TRACE sync  52 <- GetBlockHeaders: 128 entries starting from 481d…2e02
2016-10-07 17:08:05  IO Worker #2 TRACE sync  52 -> BlockHeaders (1 entries), state = NewBlocks
2016-10-07 17:08:05  IO Worker #2 TRACE sync  Inserted 1 headers
2016-10-07 17:08:05  IO Worker #2 TRACE sync  New header: 481d89d4368675ce304bbf9dc9fe4ebd3dd9ce784079cfeb6149cffc29412e02
2016-10-07 17:08:05  IO Worker #2 TRACE sync  Imported 0 of 0
2016-10-07 17:08:05  IO Worker #2 TRACE sync  52 <- GetBlockBodies: 1 entries starting from Some(481d89d4368675ce304bbf9dc9fe4ebd3dd9ce784079cfeb6149cffc29412e02)
2016-10-07 17:08:05  IO Worker #2 TRACE sync  Syncing with 3/3 peers
2016-10-07 17:08:05  IO Worker #3 TRACE sync  == Disconnecting 92: Parity/v1.3.3-beta/x86_64-linux-gnu/rustc1.12.0
2016-10-07 17:08:05  IO Worker #0 TRACE sync  52 -> BlockBodies (1 entries)
2016-10-07 17:08:05  IO Worker #0 TRACE sync  Got body 481d…2e02
2016-10-07 17:08:05  IO Worker #0 TRACE sync  Block queued 481d89d4368675ce304bbf9dc9fe4ebd3dd9ce784079cfeb6149cffc29412e02
2016-10-07 17:08:05  IO Worker #0 TRACE sync  Imported 1 of 1
2016-10-07 17:08:05  IO Worker #0 TRACE sync  Syncing with 3/3 peers
2016-10-07 17:08:05  IO Worker #0 TRACE sync  Sync complete
2016-10-07 17:08:05  IO Worker #2 TRACE sync  Sending NewHashes to [(23, 2396647), (65, 2396647)]
2016-10-07 17:08:05  IO Worker #2 TRACE sync  Sending NewBlocks to [(23, 2396647), (65, 2396647)]
2016-10-07 17:08:05  IO Worker #2 TRACE sync  Sent latest 2 blocks and 2 hashes to peers.
2016-10-07 17:08:05  IO Worker #2 TRACE sync  Sent 477 transactions to 3 peers.
2016-10-07 17:08:05  IO Worker #2 INFO import  Imported #2396648 481d…2e02 (0 txs, 0.00 Mgas, 0.38 ms, 1.04 KiB) + another 2 block(s)
2016-10-07 17:08:05  IO Worker #2 TRACE sync  65 -> GetBlockHeaders (hash: 7d94…8aab, max: 256, skip: 126, reverse:false)
2016-10-07 17:08:05  IO Worker #2 TRACE sync  -> GetBlockHeaders: returned 1 entries
2016-10-07 17:08:07  IO Worker #2 TRACE sync  == Connected 36: Parity/v1.3.3-beta/x86_64-linux-gnu/rustc1.12.0
2016-10-07 17:08:07  IO Worker #3 TRACE sync  == Disconnecting 36: Parity/v1.3.3-beta/x86_64-linux-gnu/rustc1.12.0
2016-10-07 17:08:08  IO Worker #1 TRACE sync  23 -> Transactions (1013 entries)
2016-10-07 17:08:08  IO Worker #1 TRACE sync  23 -> NewHashes (1 entries)
2016-10-07 17:08:08  IO Worker #1 TRACE sync  New block hash already in chain 481d89d4368675ce304bbf9dc9fe4ebd3dd9ce784079cfeb6149cffc29412e02
2016-10-07 17:08:11  IO Worker #1 TRACE sync  == Connected 68: Parity/v1.3.3-beta/x86_64-linux-gnu/rustc1.12.0
2016-10-07 17:08:12  IO Worker #3 TRACE sync  == Disconnecting 68: Parity/v1.3.3-beta/x86_64-linux-gnu/rustc1.12.0
2016-10-07 17:08:14  IO Worker #2 TRACE sync  23 -> Transactions (1019 entries)
2016-10-07 17:08:14  IO Worker #2 TRACE sync  23 -> Transactions (1 entries)
2016-10-07 17:08:16  IO Worker #1 TRACE sync  == Connected 77: Parity/v1.3.3-beta/x86_64-linux-gnu/rustc1.12.0
2016-10-07 17:08:17  IO Worker #2 TRACE sync  == Disconnecting 77: Parity/v1.3.3-beta/x86_64-linux-gnu/rustc1.12.0
2016-10-07 17:08:18  IO Worker #0 TRACE sync  23 -> Transactions (1020 entries)
2016-10-07 17:08:18  IO Worker #0 TRACE sync  23 -> Transactions (3 entries)
2016-10-07 17:08:19  IO Worker #0 TRACE sync  == Connected 78: Parity/v1.3.1-beta/x86_64-linux-gnu/rustc1.11.0
2016-10-07 17:08:19  IO Worker #0 TRACE sync  == Disconnecting 78: Parity/v1.3.1-beta/x86_64-linux-gnu/rustc1.11.0
2016-10-07 17:08:20  IO Worker #1 TRACE sync  == Connected 41: Parity/v1.3.1-beta/x86_64-linux-gnu/rustc1.11.0
2016-10-07 17:08:20  IO Worker #3 TRACE sync  == Disconnecting 41: Parity/v1.3.1-beta/x86_64-linux-gnu/rustc1.11.0
2016-10-07 17:08:21  IO Worker #3 TRACE sync  == Disconnecting 65: Parity/v1.3.2-beta/x86_64-linux-gnu/rustc1.11.0
2016-10-07 17:08:21  IO Worker #3 DEBUG sync  Disconnected 65
2016-10-07 17:08:21  IO Worker #3 TRACE sync  Syncing with 2/2 peers
2016-10-07 17:08:21  IO Worker #3 TRACE sync  Sync complete
2016-10-07 17:08:21  IO Worker #2 TRACE sync  == Connected 35: Parity/v1.3.3-beta/x86_64-linux-gnu/rustc1.12.0
2016-10-07 17:08:21  IO Worker #1 TRACE sync  == Disconnecting 35: Parity/v1.3.3-beta/x86_64-linux-gnu/rustc1.12.0
2016-10-07 17:08:21  IO Worker #3 TRACE sync  == Connected 71: Parity/v1.3.2-beta/x86_64-linux-gnu/rustc1.11.0
2016-10-07 17:08:22  IO Worker #2 TRACE sync  New peer 71 (protocol: 63, network: 1, difficulty: Some(71036104455512214166), latest:f97d…b5ed, genesis:d4e5…8fa3)
2016-10-07 17:08:22  IO Worker #2 DEBUG sync  Connected 71:Parity/v1.3.2-beta/x86_64-linux-gnu/rustc1.11.0
2016-10-07 17:08:22  IO Worker #2 TRACE sync  71 <- GetBlockHeaders: 1 entries starting from 1920000
2016-10-07 17:08:22  IO Worker #0 TRACE sync  71 -> GetBlockHeaders (number: 1920000, max: 1, skip: 0, reverse:false)
2016-10-07 17:08:22  IO Worker #0 TRACE sync  71: Returning cached fork header
2016-10-07 17:08:22  IO Worker #0 TRACE sync  -> GetBlockHeaders: returned 1 entries
2016-10-07 17:08:22  IO Worker #2 TRACE sync  23 -> Transactions (1 entries)
2016-10-07 17:08:22  IO Worker #1 TRACE sync  71: Confirmed peer
2016-10-07 17:08:23  IO Worker #1 TRACE sync  == Connected 10: Parity/v1.3.3-beta/x86_64-linux-gnu/rustc1.12.0
2016-10-07 17:08:23  IO Worker #3 TRACE sync  == Disconnecting 10: Parity/v1.3.3-beta/x86_64-linux-gnu/rustc1.12.0
2016-10-07 17:08:24  IO Worker #1 TRACE sync  == Connected 83: Parity/v1.3.1-beta/x86_64-linux-gnu/rustc1.11.0
2016-10-07 17:08:24  IO Worker #3 TRACE sync  71 -> GetBlockHeaders (hash: 7d94…8aab, max: 256, skip: 126, reverse:false)
2016-10-07 17:08:24  IO Worker #3 TRACE sync  -> GetBlockHeaders: returned 1 entries
2016-10-07 17:08:24  IO Worker #2 TRACE sync  == Disconnecting 83: Parity/v1.3.1-beta/x86_64-linux-gnu/rustc1.11.0
2016-10-07 17:08:25  IO Worker #3 TRACE sync  71 -> GetBlockHeaders (hash: 7d94…8aab, max: 256, skip: 126, reverse:false)
2016-10-07 17:08:25  IO Worker #3 TRACE sync  -> GetBlockHeaders: returned 1 entries
2016-10-07 17:08:26  IO Worker #1 TRACE sync  71 -> GetBlockHeaders (hash: 7d94…8aab, max: 256, skip: 126, reverse:false)
2016-10-07 17:08:26  IO Worker #1 TRACE sync  -> GetBlockHeaders: returned 1 entries
2016-10-07 17:08:27  IO Worker #3 TRACE sync  71 -> GetBlockHeaders (hash: 7d94…8aab, max: 256, skip: 126, reverse:false)
2016-10-07 17:08:27  IO Worker #3 TRACE sync  -> GetBlockHeaders: returned 1 entries
2016-10-07 17:08:28  IO Worker #1 TRACE sync  == Connected 83: Parity/v1.3.3-beta/x86_64-linux-gnu/rustc1.12.0
2016-10-07 17:08:28  IO Worker #1 TRACE sync  == Disconnecting 83: Parity/v1.3.3-beta/x86_64-linux-gnu/rustc1.12.0
2016-10-07 17:08:32  IO Worker #0 TRACE sync  71 -> GetBlockHeaders (hash: 7d94…8aab, max: 256, skip: 126, reverse:false)
2016-10-07 17:08:32  IO Worker #0 TRACE sync  -> GetBlockHeaders: returned 1 entries
2016-10-07 17:08:33  IO Worker #3 TRACE sync  == Connected 54: Parity/v1.3.3-beta/x86_64-linux-gnu/rustc1.12.0
2016-10-07 17:08:33  IO Worker #0 TRACE sync  == Disconnecting 54: Parity/v1.3.3-beta/x86_64-linux-gnu/rustc1.12.0
2016-10-07 17:08:35  IO Worker #0 TRACE sync  71 -> GetBlockHeaders (hash: 7d94…8aab, max: 256, skip: 126, reverse:false)
2016-10-07 17:08:35  IO Worker #0 TRACE sync  -> GetBlockHeaders: returned 1 entries
2016-10-07 17:08:35  IO Worker #0 TRACE sync  == Connected 98: Parity/v1.3.3-beta/x86_64-linux-gnu/rustc1.12.0
2016-10-07 17:08:35  IO Worker #3 TRACE sync  == Disconnecting 98: Parity/v1.3.3-beta/x86_64-linux-gnu/rustc1.12.0
2016-10-07 17:08:37  IO Worker #3 TRACE sync  71 -> GetBlockHeaders (hash: 7d94…8aab, max: 256, skip: 126, reverse:false)
2016-10-07 17:08:37  IO Worker #3 TRACE sync  -> GetBlockHeaders: returned 1 entries
2016-10-07 17:08:38  IO Worker #1 TRACE sync  == Connected 30: Parity/v1.3.3-beta/x86_64-linux-gnu/rustc1.12.0
2016-10-07 17:08:38  IO Worker #0 TRACE sync  == Disconnecting 30: Parity/v1.3.3-beta/x86_64-linux-gnu/rustc1.12.0
2016-10-07 17:08:39  IO Worker #0 TRACE sync  71 -> GetBlockHeaders (hash: 7d94…8aab, max: 256, skip: 126, reverse:false)
2016-10-07 17:08:39  IO Worker #0 TRACE sync  -> GetBlockHeaders: returned 1 entries
2016-10-07 17:08:41  IO Worker #2 TRACE sync  71 -> GetBlockHeaders (hash: 7d94…8aab, max: 256, skip: 126, reverse:false)
2016-10-07 17:08:41  IO Worker #2 TRACE sync  -> GetBlockHeaders: returned 1 entries
2016-10-07 17:08:42  IO Worker #1 TRACE sync  == Connected 25: Parity/v1.3.3-beta/x86_64-linux-gnu/rustc1.12.0
2016-10-07 17:08:42  IO Worker #3 TRACE sync  == Disconnecting 25: Parity/v1.3.3-beta/x86_64-linux-gnu/rustc1.12.0
2016-10-07 17:08:42  IO Worker #2 TRACE sync  == Connected 68: Parity/v1.3.2-beta/x86_64-linux-gnu/rustc1.11.0
2016-10-07 17:08:43  IO Worker #1 TRACE sync  23 -> NewHashes (5 entries)
2016-10-07 17:08:43  IO Worker #1 TRACE sync  New block hash already in chain 481d89d4368675ce304bbf9dc9fe4ebd3dd9ce784079cfeb6149cffc29412e02
@arkpar
Copy link
Collaborator

arkpar commented Oct 7, 2016

Please try upgrading to the latest version (1.3.4) and see if it still hangs

@gituser
Copy link
Author

gituser commented Oct 7, 2016

Please try upgrading to the latest version (1.3.4) and see if it still hangs

will do this in a moment and report back

@rphmeier rphmeier added F2-bug 🐞 The client fails to follow expected behavior. M4-core ⛓ Core client code / Rust. labels Oct 7, 2016
@gituser
Copy link
Author

gituser commented Oct 8, 2016

no, v1.3.4 didn't fix the issue.

it's still happening on Parity/v1.3.4-beta-a8b2cf9-20161006/x86_64-linux-gnu/rustc1.12.0

I'll try v1.3.5

@arkpar
Copy link
Collaborator

arkpar commented Oct 8, 2016

@gituser please run with -l sync=trace 2>parity.log And upload the log file if it happens again

@gituser
Copy link
Author

gituser commented Oct 8, 2016 •
edited
Loading

@arkpar, I did run with that flag when it crashed and stderr was redirected to the same file. Do you really need whole big logfile? I've pasted last 100 lines of it there are no errors.

However, I have an idea where bug might be residing, I'm running in exactly same VM parity-v1.3.3 for 2 days already without issues in the screen (without daemon mode). So this bug might be residing there. So worth checking daemon lib you're using (maybe update to latest?)

NOTE: v1.3.1 runs perfectly fine in daemon mode.

As for current situation: I've compiled latest v1.3.5 and now running it through start-stop-daemon, but not using daemon mode and without --log-file, both stdout and stderr are redirected to the logfile. Will see how it goes and report back.

@arkpar
Copy link
Collaborator

arkpar commented Oct 8, 2016

@gituser if that fails, try the db-testing branch

@gituser
Copy link
Author

gituser commented Oct 8, 2016

@arkpar, thank you, i'll try that if everything else fails.

So far v1.3.5 been running ok for ~ 7 hours without daemon mode.

@gituser
Copy link
Author

gituser commented Oct 10, 2016

so it's something to do with daemon mode.

on other VM exactly same v1.3.5 in daemon mode hanged, whilst in other VM just running parity without daemon works fine.

here is log

@gavofyork gavofyork added this to the 1.4 Civility milestone Oct 17, 2016
@tomusdrw tomusdrw self-assigned this Oct 17, 2016
@tomusdrw
Copy link
Collaborator

Daemon library is in the latest version, it was updated between 1.3.1 and 1.3.4, but the change is minor and is not affecting our use case.

So I think it's not related to deamon library but rather to ongoing attacks by that time. @gituser could you test the behaviour with 1.3.9 to see if it's still reproducible?

@gituser
Copy link
Author

gituser commented Nov 7, 2016

@tomusdrw sorry for longness, somehow didn't notice there was a reply to this issue.

I'm certain it's related to daemon mode. I'll test latest 1.3.10 and report back in few days.

@gavofyork gavofyork added Z0-unconfirmed 🤔 Issue might be valid, but it’s not yet known. and removed F2-bug 🐞 The client fails to follow expected behavior. labels Nov 7, 2016
@gavofyork gavofyork removed this from the 1.4 Civility milestone Nov 7, 2016
@gituser
Copy link
Author

gituser commented Nov 12, 2016

with 1.3.10 parity was stuck at block #2415298 forever, I'm not sure if it's related to the new HF, I'm just building 1.3.11 and re-syncing from scratch..

@gituser
Copy link
Author

gituser commented Nov 14, 2016 •
edited
Loading

I'm still trying to get syncd fully.

Here are some more observations:

  • If I run parity with daemon mode it syncs very slowly (sometimes there is simply no updates in the log for an hour or more).
  • If I run parity without daemon mode sync progress is there in the logs.

Also had 3 OOM on 1.3.11 during initial sync from scratch on 5GB + 1GB swap VM.

UPDATE: there is indeed an issue with daemon mode, verified on 1.3.11 - but this time parity is not stalling, it keeps giving messages about syncing blocks in the log, but in fact it's always behind and not fully synced, if i run without daemon mode - it works just fine.

@zet-tech
Copy link

What is status of this issue ? We currently (parity 1.4.10) face very similar problem. After some time (this time approx. after 2 days) parity stops responding properly to some RPC calls (e.g. getWork, block information, but for example peerCount works). Logs show continuous synchronisation but our mining software still gets the same work. It looks like parity is not fully synced. After restart there is no syncing, parity continues to work like it was fully synced and everything is back normal. This happened second time (previously on parity 1.4.9) during the last month.

I attach two log files: first from the moment where parity started to return the same work from getWork (block at height #3205515).
hang.txt

Second from the moment of the restart.
restart.txt

Issue is really hard to reproduce (2 times/month) so I cannot attach more detailed logs.

We run parity as systemd service with following config file:
parity.txt

@tomusdrw
Copy link
Collaborator

@zet-tech So the RPC server is responding to those requests but responses are incorrect?

It seems like it's a different issue then, most probably related to sync stalling for some reason.

If that's the case it would be very helpful if you could provide -lsync=trace logs from the moment when it happens.

As a workaround you may also check beta release and see if the issue is still present there.

@zet-tech
Copy link

zet-tech commented Feb 17, 2017 •
edited
Loading

@tomusdrw Yes, server responded to getWork but always with the same value for about 8h.

After giving it some thought, I agree that is must be some sync related problem. But if sync was stalled then after parity restart there should be some syncing to catch up with blockchain. It was not (see logs in restart.txt) and after the restart everything worked fine. It looks like parity continued to sync properly, but RPC was fixed at given block and was unaware of any further synced blocks.

Running beta in our environment is currently not possible due to some RPC changes.

I've configured new node with sync=trace but as I mentioned we probably need do wait some time (2-3 weeks) for this bug to reproduce.

@5chdn
Copy link
Contributor

5chdn commented Mar 3, 2017

@zet-tech were you able to reproduce this?

Please note, that 1.5.x will be the stable branch soon, so you might have to update your RPC calls anyways.

@zet-tech
Copy link

zet-tech commented Mar 4, 2017

(Un)Fortunately we are still waiting for this bug to reproduce with sync=trace enabled.

But maybe it wont because we started to restart parity every few days. If we manage to reproduce this error and gather some logs then we will file new issue.

@5chdn
Copy link
Contributor

5chdn commented Apr 11, 2017

Ok.

@5chdn 5chdn closed this as completed Apr 11, 2017
Sign up for free to subscribe to this conversation on GitHub. Already have an account? Sign in.
Assignees

@tomusdrw tomusdrw

Labels
M4-core ⛓ Core client code / Rust. Z0-unconfirmed 🤔 Issue might be valid, but it’s not yet known.
Projects
None yet
Milestone
No milestone
Development

No branches or pull requests
7 participants
@gituser @gavofyork @tomusdrw @arkpar @rphmeier @5chdn @zet-tech