19.16.2.2 (official build) crash

[kwai-code]

[alesapin]

@kwai-code Maybe it's possible to make a similar paste as text, not screenshot? Seems like you don't use our clickhouse-common-static-dbg package, so line numbers of source code files are hidden in the trace. It's possible to use addresses to deduce lines with addr2line tool, but copy them from the screenshot is not convenient.

[kwai-code]

sorry for paste screenshot

2019.11.13 22:03:41.428343 [ 157 ] {15D56D1912F3E289} <Error> virtual DB::UnionBlockInputStream::~UnionBlockInputStream(): Code: 394, e.displayText() = DB::Exception: Query was cancelled:
Code: 394, e.displayText() = DB::Exception: Query was cancelled (version 19.16.2.2 (official build)):
Code: 394, e.displayText() = DB::Exception: Query was cancelled (version 19.16.2.2 (official build)):
Code: 394, e.displayText() = DB::Exception: Query was cancelled (version 19.16.2.2 (official build)):
Code: 394, e.displayText() = DB::Exception: Query was cancelled (version 19.16.2.2 (official build)):
Code: 394, e.displayText() = DB::Exception: Query was cancelled (version 19.16.2.2 (official build)):
Code: 394, e.displayText() = DB::Exception: Query was cancelled (version 19.16.2.2 (official build)):
Code: 394, e.displayText() = DB::Exception: Query was cancelled (version 19.16.2.2 (official build)):
Code: 394, e.displayText() = DB::Exception: Query was cancelled (version 19.16.2.2 (official build)):
Code: 394, e.displayText() = DB::Exception: Query was cancelled (version 19.16.2.2 (official build)):
Code: 394, e.displayText() = DB::Exception: Query was cancelled (version 19.16.2.2 (official build)):
Code: 394, e.displayText() = DB::Exception: Query was cancelled (version 19.16.2.2 (official build)):
Code: 394, e.displayText() = DB::Exception: Query was cancelled (version 19.16.2.2 (official build)):
Code: 394, e.displayText() = DB::Exception: Query was cancelled (version 19.16.2.2 (official build)):
Code: 394, e.displayText() = DB::Exception: Query was cancelled (version 19.16.2.2 (official build)):
Code: 394, e.displayText() = DB::Exception: Query was cancelled (version 19.16.2.2 (official build)):
Code: 394, e.displayText() = DB::Exception: Query was cancelled (version 19.16.2.2 (official build)):
Code: 394, e.displayText() = DB::Exception: Query was cancelled (version 19.16.2.2 (official build)):
Code: 394, e.displayText() = DB::Exception: Query was cancelled (version 19.16.2.2 (official build)):
Code: 394, e.displayText() = DB::Exception: Query was cancelled (version 19.16.2.2 (official build)):
Code: 394, e.displayText() = DB::Exception: Query was cancelled (version 19.16.2.2 (official build)):
Code: 394, e.displayText() = DB::Exception: Query was cancelled (version 19.16.2.2 (official build)):
Code: 394, e.displayText() = DB::Exception: Query was cancelled (version 19.16.2.2 (official build)):
Code: 394, e.displayText() = DB::Exception: Query was cancelled (version 19.16.2.2 (official build)):
Code: 394, e.displayText() = DB::Exception: Query was cancelled (version 19.16.2.2 (official build)):
Code: 394, e.displayText() = DB::Exception: Query was cancelled (version 19.16.2.2 (official build)):
Code: 394, e.displayText() = DB::Exception: Query was cancelled (version 19.16.2.2 (official build)):
Code: 394, e.displayText() = DB::Exception: Query was cancelled (version 19.16.2.2 (official build)):
Code: 394, e.displayText() = DB::Exception: Query was cancelled (version 19.16.2.2 (official build)):
Code: 394, e.displayText() = DB::Exception: Query was cancelled (version 19.16.2.2 (official build)):
Code: 394, e.displayText() = DB::Exception: Query was cancelled (version 19.16.2.2 (official build)):
Code: 394, e.displayText() = DB::Exception: Query was cancelled (version 19.16.2.2 (official build)):
Code: 394, e.displayText() = DB::Exception: Query was cancelled (version 19.16.2.2 (official build)):
Code: 394, e.displayText() = DB::Exception: Query was cancelled (version 19.16.2.2 (official build)):
Code: 394, e.displayText() = DB::Exception: Query was cancelled (version 19.16.2.2 (official build)), Stack trace:

0. 0x555dce6e67b0 StackTrace::StackTrace() ?
1. 0x555dce6e6585 DB::Exception::Exception(std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const&, int) ?
2. 0x555dce3e75a7 ? ?
3. 0x555dd18f9707 DB::IBlockInputStream::read() ?
4. 0x555dd2019b44 DB::DistinctBlockInputStream::readImpl() ?
5. 0x555dd18f97f7 DB::IBlockInputStream::read() ?
6. 0x555dd18fe71a DB::LimitBlockInputStream::readImpl() ?
7. 0x555dd18f97f7 DB::IBlockInputStream::read() ?
8. 0x555dd1a710b2 DB::ParallelInputsProcessor<DB::UnionBlockInputStream::Handler>::loop(unsigned long) ?
9. 0x555dd1a71775 DB::ParallelInputsProcessor<DB::UnionBlockInputStream::Handler>::thread(std::shared_ptr<DB::ThreadGroupStatus>, unsigned long) ?
10. 0x555dd1a720ed ThreadFromGlobalPool::ThreadFromGlobalPool<void (DB::ParallelInputsProcessor<DB::UnionBlockInputStream::Handler>::*)(std::shared_ptr<DB::ThreadGroupStatus>, unsigned long), DB::ParallelInputsProcessor<DB::UnionBlockInputStream::Handler>*, std::shared_ptr<DB::ThreadGroupStatus>, unsigned long&>(void (DB::ParallelInputsProcessor<DB::UnionBlockInputStream::Handler>::*&&)(std::shared_ptr<DB::ThreadGroupStatus>, unsigned long), DB::ParallelInputsProcessor<DB::UnionBlockInputStream::Handler>*&&, std::shared_ptr<DB::ThreadGroupStatus>&&, unsigned long&)::{lambda()#1}::operator()() const ?
11. 0x555dce730d5c ThreadPoolImpl<std::thread>::worker(std::_List_iterator<std::thread>) ?
12. 0x555dd444e1e0 ? ?
13. 0x7f137770ae25 start_thread /usr/lib64/libpthread-2.17.so
14. 0x7f137702c34d clone /usr/lib64/libc-2.17.so
 (version 19.16.2.2 (official build))
2019.11.13 22:03:41.495778 [ 554 ] {} <Fatal> BaseDaemon: (version 19.16.2.2 (official build)) (from thread 157) Terminate called for uncaught exception:
Code: 24, e.displayText() = DB::Exception: Cannot write to ostream at offset 1650915, Stack trace:

0. 0x555dce6e67b0 StackTrace::StackTrace() ?
1. 0x555dce6e6585 DB::Exception::Exception(std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const&, int) ?
2. 0x555dce55d2e7 ? ?
3. 0x555dd25b5379 DB::WriteBufferFromHTTPServerResponse::nextImpl() ?
4. 0x555dd25bc2ae DB::WriteBufferValidUTF8::nextImpl() ?
5. 0x555dd25bc34a DB::WriteBufferValidUTF8::finish() ?
6. 0x555dd1ee61ef DB::WriteBufferValidUTF8::~WriteBufferValidUTF8() ?
7. 0x555dd1ee67fc DB::JSONRowOutputFormat::~JSONRowOutputFormat() ?
8. 0x555dd1ec3dd8 DB::OutputStreamToOutputFormat::~OutputStreamToOutputFormat() ?
9. 0x555dd1e7a708 DB::MaterializingBlockOutputStream::~MaterializingBlockOutputStream() ?
10. 0x555dce6e6698 std::_Sp_counted_base<(__gnu_cxx::_Lock_policy)2>::_M_release() ?
11. 0x555dd1b895fd ? ?
12. 0x555dce772a9a DB::IBlockInputStream::~IBlockInputStream()
2019.11.13 22:03:41.517041 [ 555 ] {} <Fatal> BaseDaemon: ########################################
2019.11.13 22:03:41.517099 [ 555 ] {} <Fatal> BaseDaemon: (version 19.16.2.2 (official build)) (from thread 157) Received signal Aborted (6).
2019.11.13 22:03:41.517110 [ 555 ] {} <Fatal> BaseDaemon:
2019.11.13 22:03:41.517129 [ 555 ] {} <Fatal> BaseDaemon: Stack trace: 0x7f1376f691f7 0x7f1376f6a8e8 0x555dce25afae 0x555dd43cc4a6 0x555dd447bbd9 0x555dd43cbe84 0x555dd448aef9 0x555dd448b157 0x555dce55f178 0x555dd25bc34a 0x555dd1ee61ef 0x555dd1ee67fc 0x555dd1ec3dd8 0x555dd1e7a708 0x555dce6e6698 0x555dd1b895fd 0x555dce772a9a 0x555dce772ae5 0x555dd1a6b3e6 0x555dce772ae5 0x555dd201b1b8 0x555dd1d7d2fd 0x555dd1902c55 0x555dd1b176c6 0x555dce6e6698 0x555dce43be29 0x555dce77a8ae 0x555dce77cc29 0x555dd260ba69
2019.11.13 22:03:41.517225 [ 555 ] {} <Fatal> BaseDaemon: 3. 0x7f1376f691f7 raise /usr/lib64/libc-2.17.so
2019.11.13 22:03:41.517236 [ 555 ] {} <Fatal> BaseDaemon: 4. 0x7f1376f6a8e8 abort /usr/lib64/libc-2.17.so
2019.11.13 22:03:41.517247 [ 555 ] {} <Fatal> BaseDaemon: 5. 0x555dce25afae ? ?
2019.11.13 22:03:41.521674 [ 555 ] {} <Fatal> BaseDaemon: 6. 0x555dd43cc4a6 __cxxabiv1::__terminate(void (*)()) ?
2019.11.13 22:03:41.521727 [ 555 ] {} <Fatal> BaseDaemon: 7. 0x555dd447bbd9 ? ?
2019.11.13 22:03:41.521736 [ 555 ] {} <Fatal> BaseDaemon: 8. 0x555dd43cbe84 __gxx_personality_v0 ?
2019.11.13 22:03:41.521749 [ 555 ] {} <Fatal> BaseDaemon: 9. 0x555dd448aef9 ? ?
2019.11.13 22:03:41.521756 [ 555 ] {} <Fatal> BaseDaemon: 10. 0x555dd448b157 _Unwind_Resume ?
2019.11.13 22:03:41.521762 [ 555 ] {} <Fatal> BaseDaemon: 11. 0x555dce55f178 ? ?
2019.11.13 22:03:41.521770 [ 555 ] {} <Fatal> BaseDaemon: 12. 0x555dd25bc34a DB::WriteBufferValidUTF8::finish() ?
2019.11.13 22:03:41.521779 [ 555 ] {} <Fatal> BaseDaemon: 13. 0x555dd1ee61ef DB::WriteBufferValidUTF8::~WriteBufferValidUTF8() ?
2019.11.13 22:03:41.521786 [ 555 ] {} <Fatal> BaseDaemon: 14. 0x555dd1ee67fc DB::JSONRowOutputFormat::~JSONRowOutputFormat() ?
2019.11.13 22:03:41.521792 [ 555 ] {} <Fatal> BaseDaemon: 15. 0x555dd1ec3dd8 DB::OutputStreamToOutputFormat::~OutputStreamToOutputFormat() ?
2019.11.13 22:03:41.521801 [ 555 ] {} <Fatal> BaseDaemon: 16. 0x555dd1e7a708 DB::MaterializingBlockOutputStream::~MaterializingBlockOutputStream() ?
2019.11.13 22:03:41.521811 [ 555 ] {} <Fatal> BaseDaemon: 17. 0x555dce6e6698 std::_Sp_counted_base<(__gnu_cxx::_Lock_policy)2>::_M_release() ?
2019.11.13 22:03:41.521817 [ 555 ] {} <Fatal> BaseDaemon: 18. 0x555dd1b895fd ? ?
2019.11.13 22:03:41.521824 [ 555 ] {} <Fatal> BaseDaemon: 19. 0x555dce772a9a DB::IBlockInputStream::~IBlockInputStream() ?
2019.11.13 22:03:41.521831 [ 555 ] {} <Fatal> BaseDaemon: 20. 0x555dce772ae5 DB::IBlockInputStream::~IBlockInputStream() ?
2019.11.13 22:03:41.521838 [ 555 ] {} <Fatal> BaseDaemon: 21. 0x555dd1a6b3e6 DB::UnionBlockInputStream::~UnionBlockInputStream() ?
2019.11.13 22:03:41.521845 [ 555 ] {} <Fatal> BaseDaemon: 22. 0x555dce772ae5 DB::IBlockInputStream::~IBlockInputStream() ?
2019.11.13 22:03:41.521851 [ 555 ] {} <Fatal> BaseDaemon: 23. 0x555dd201b1b8 DB::DistinctBlockInputStream::~DistinctBlockInputStream() ?
2019.11.13 22:03:41.521859 [ 555 ] {} <Fatal> BaseDaemon: 24. 0x555dd1d7d2fd DB::ExpressionBlockInputStream::~ExpressionBlockInputStream() ?
2019.11.13 22:03:41.521866 [ 555 ] {} <Fatal> BaseDaemon: 25. 0x555dd1902c55 DB::LimitBlockInputStream::~LimitBlockInputStream() ?
2019.11.13 22:03:41.521873 [ 555 ] {} <Fatal> BaseDaemon: 26. 0x555dd1b176c6 DB::ProcessListEntry::~ProcessListEntry() ?
2019.11.13 22:03:41.521879 [ 555 ] {} <Fatal> BaseDaemon: 27. 0x555dce6e6698 std::_Sp_counted_base<(__gnu_cxx::_Lock_policy)2>::_M_release() ?
2019.11.13 22:03:41.521886 [ 555 ] {} <Fatal> BaseDaemon: 28. 0x555dce43be29 ? ?
2019.11.13 22:03:41.521912 [ 555 ] {} <Fatal> BaseDaemon: 29. 0x555dce77a8ae DB::HTTPHandler::processQuery(Poco::Net::HTTPServerRequest&, HTMLForm&, Poco::Net::HTTPServerResponse&, DB::HTTPHandler::Output&) ?
2019.11.13 22:03:41.521920 [ 555 ] {} <Fatal> BaseDaemon: 30. 0x555dce77cc29 DB::HTTPHandler::handleRequest(Poco::Net::HTTPServerRequest&, Poco::Net::HTTPServerResponse&) ?
2019.11.13 22:03:41.521928 [ 555 ] {} <Fatal> BaseDaemon: 31. 0x555dd260ba69 Poco::Net::HTTPServerConnection::run() ?

[kevintop]

What is the cause of this problem? How to avoid?

[alexey-milovidov]

This is caused by unhandled exception in descrutor. Exception may happen when a client has disconnected and we cannot send data. The exception is not handled correctly.