Crash while publishing to the schema registry

[zimbatm]

Version & Environment

Redpanda version: vectorized/redpanda:v21.11.2 (docker image)

What went wrong?

The service crashed while publishing a new schema to the registry.

What should have happened instead?

Not crash? :)

How to reproduce the issue?

It seems to only happen when writing a lot of schemas in short successions in the registry. I will update the issue once I get a better repro.

Additional information

Please attach any relevant logs, backtraces, or metric charts.

Backtrace:

redpanda                | redpanda: /vectorized/include/boost/container/vector.hpp:1622: boost::container::vector::reference boost::container::vector<std::unique_ptr<seastar::reactor::task_queue>, boost::container::dtl::static_storage_allocator<std::unique_ptr<seastar::reactor::task_queue>, 16, 0, true>>::operator[](boost::container::vector::size_type) [T = std::unique_ptr<seastar::reactor::task_queue>, A = boost::container::dtl::static_storage_allocator<std::unique_ptr<seastar::reactor::task_queue>, 16, 0, true>, Options = void]: Assertion `this->m_holder.m_size > n' failed.
redpanda                | Aborting on shard 0.
redpanda                | Backtrace:
redpanda                |   0x380b3e8
redpanda                |   0x2a552ba699ff
redpanda                |   /opt/redpanda/lib/libc.so.6+0x3d291
redpanda                |   /opt/redpanda/lib/libc.so.6+0x268a3
redpanda                |   /opt/redpanda/lib/libc.so.6+0x26788
redpanda                |   /opt/redpanda/lib/libc.so.6+0x35a15
redpanda                |   0x37b964b
redpanda                |   0x1c11b15
redpanda                |   0x37ba9f4
redpanda                |   0x37bdeb7
redpanda                |   0x37bb295
redpanda                |   0x370d0b9
redpanda                |   0x370b150
redpanda                |   0x116ee94
redpanda                |   0x3b01d3c
redpanda                |   /opt/redpanda/lib/libc.so.6+0x27b74
redpanda                |   0x116bced
redpanda                | Segmentation fault on shard 0.
redpanda                | Backtrace:
redpanda                |   0x380acb8
redpanda                |   0x2a552ba699ff
redpanda                |   /opt/redpanda/lib/libc.so.6+0x26962
redpanda                |   /opt/redpanda/lib/libc.so.6+0x26788
redpanda                |   /opt/redpanda/lib/libc.so.6+0x35a15
redpanda                |   0x37b964b
redpanda                |   0x1c11b15
redpanda                |   0x37ba9f4
redpanda                |   0x37bdeb7
redpanda                |   0x37bb295
redpanda                |   0x370d0b9
redpanda                |   0x370b150
redpanda                |   0x116ee94
redpanda                |   0x3b01d3c
redpanda                |   /opt/redpanda/lib/libc.so.6+0x27b74
redpanda                |   0x116bced

[BenPope]

If you're publishing protobuf schema, then I'm 95% sure I have a reproducer for this, I'm working on a fix as top priority.

[flokli]

Yes, this is protobuf schemas.

[zimbatm]

I wasn't able to build a minimal repro so far but can reproduce this locally very easily.

What I was trying to do is dump and restore the schema registry so it can be migrated from cp-schema-registry to redpanda. Here are the two scripts that I am using: https://gist.github.com/b235775c6191abfee075ac4c4fd3d9af

There are currently 39 files generated by the sc-dump.sh script, each with different IDs and at version 1. Then I run the sc-restore.sh script and it fails randomly. If I add a sleep between each curl call the script doesn't fail (so far). Note that since I'm loading the same schema, I don't think it's generating new versions either.

[BenPope]

I wasn't able to build a minimal repro so far but can reproduce this locally very easily.

What I was trying to do is dump and restore the schema registry so it can be migrated from cp-schema-registry to redpanda. Here are the two scripts that I am using: https://gist.github.com/b235775c6191abfee075ac4c4fd3d9af

There are currently 39 files generated by the sc-dump.sh script, each with different IDs and at version 1. Then I run the sc-restore.sh script and it fails randomly. If I add a sleep between each curl call the script doesn't fail (so far). Note that since I'm loading the same schema, I don't think it's generating new versions either.

@zimbatm As a workaround, you may be able to dump and restore the raw records (perhaps something like MirrorMaker) on the _schemas topic. The "other" binary format should be supported for common message types, but there are some unsupported messages, such as those for the /mode endpoints. I'd be interested if you encounter any issues doing it that way.