Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Unsupported kafka api v3 (leave group) in PartitionMovementUpgradeTest.test_basic_upgrade #5868

Closed
dotnwat opened this issue Aug 5, 2022 · 9 comments · Fixed by #5869 or #6580
Closed

Unsupported kafka api v3 (leave group) in PartitionMovementUpgradeTest.test_basic_upgrade #5868

dotnwat opened this issue Aug 5, 2022 · 9 comments · Fixed by #5869 or #6580
Assignees
@mmaslankaprv
Labels
area/redpanda area/tests ci-disabled-test ci-failure kind/bug Something isn't working

Comments

@dotnwat
Copy link
Member

dotnwat commented Aug 5, 2022

https://buildkite.com/redpanda/redpanda/builds/13660#01826c94-abff-4f4f-ba8e-e64331d9b328

test_id:    rptest.tests.partition_movement_upgrade_test.PartitionMovementUpgradeTest.test_basic_upgrade
--
  | status:     FAIL
  | run time:   2 minutes 8.856 seconds
  |  
  |  
  | <BadLogLines nodes=docker-rp-23(20) example="ERROR 2022-08-05 07:01:42,886 [shard 0] rpc - server.cc:116 - kafka rpc protocol - Error[applying protocol] remote address: 172.16.32.22:49704 - std::runtime_error (Unsupported version 3 for leave group API)">
  | Traceback (most recent call last):
  | File "/usr/local/lib/python3.10/dist-packages/ducktape/tests/runner_client.py", line 135, in run
  | data = self.run_test()
  | File "/usr/local/lib/python3.10/dist-packages/ducktape/tests/runner_client.py", line 227, in run_test
  | return self.test_context.function(self.test)
  | File "/root/tests/rptest/services/cluster.py", line 48, in wrapped
  | self.redpanda.raise_on_bad_logs(allow_list=log_allow_list)
  | File "/root/tests/rptest/services/redpanda.py", line 1121, in raise_on_bad_logs
  | raise BadLogLines(bad_lines)
  | rptest.services.utils.BadLogLines: <BadLogLines nodes=docker-rp-23(20) example="ERROR 2022-08-05 07:01:42,886 [shard 0] rpc - server.cc:116 - kafka rpc protocol - Error[applying protocol] remote address: 172.16.32.22:49704 - std::runtime_error (Unsupported version 3 for leave group API)">
@dotnwat dotnwat added kind/bug Something isn't working ci-failure labels Aug 5, 2022
@jcsp
Copy link
Contributor

jcsp commented Aug 5, 2022

Similar failure on the same test, on a different group API

FAIL test: PartitionMovementUpgradeTest.test_basic_upgrade (7/37 runs)
failure at 2022-08-05T07:30:11.541Z:
in job https://buildkite.com/redpanda/redpanda/builds/13658#01826c85-9f41-44eb-9564-83cc5724d06e

@jcsp jcsp changed the title Unsupported kafka api v3 (leave group) in upgrade test Unsupported kafka api v3 (leave group) in PartitionMovementUpgradeTest.test_basic_upgrade Aug 5, 2022
@dotnwat dotnwat mentioned this issue Aug 5, 2022
Merged
5 tasks
mmaslankaprv added a commit to mmaslankaprv/redpanda that referenced this issue Aug 5, 2022
@mmaslankaprv
k/connection_context: use warn severity for unsupported api versions
74a2aff 
When client requests an unsupported API version it is not an unexpected
situation and should not be logged with ERROR severity. Introduced
separate exception handler for unsupported API version exceptions.

Fixes: redpanda-data#5868

Signed-off-by: Michal Maslanka <michal@redpanda.com>
@mmaslankaprv mmaslankaprv mentioned this issue Aug 5, 2022
Merged
5 tasks
mmaslankaprv added a commit to mmaslankaprv/redpanda that referenced this issue Aug 5, 2022
@mmaslankaprv
k/connection_context: use warn severity for unsupported api versions
6a2ece5 
When client requests an unsupported API version it is not an unexpected
situation and should not be logged with ERROR severity. Introduced
separate exception handler for unsupported API version exceptions.

Fixes: redpanda-data#5868

Signed-off-by: Michal Maslanka <michal@redpanda.com>
mmaslankaprv added a commit to mmaslankaprv/redpanda that referenced this issue Aug 5, 2022
@mmaslankaprv
k/connection_context: use warn severity for unsupported api versions
0aa3606 
When client requests an unsupported API version it is not an unexpected
situation and should not be logged with ERROR severity. Introduced
separate exception handler for unsupported API version exceptions.

Fixes: redpanda-data#5868

Signed-off-by: Michal Maslanka <michal@redpanda.com>
@dotnwat dotnwat mentioned this issue Aug 5, 2022
Merged
5 tasks
@rystsov
Copy link
Contributor

rystsov commented Aug 6, 2022

Another instance https://ci-artifacts.dev.vectorized.cloud/redpanda/01826f8f-f896-46fd-9a53-0a31b2af399b/vbuild/ducktape/results/2022-08-05--001/report.html

@rystsov rystsov mentioned this issue Aug 6, 2022
Merged
@dotnwat dotnwat mentioned this issue Aug 6, 2022
Merged
5 tasks
@BenPope
Copy link
Member

BenPope commented Aug 8, 2022

This seems similar: https://ci-artifacts.dev.vectorized.cloud/redpanda/018279f4-00b6-44e0-956b-20f174bc3a6e/vbuild/ducktape/results/2022-08-07--001/PartitionMovementUpgradeTest/test_basic_upgrade/138/

[INFO  - 2022-08-07 21:30:41,423 - runner_client - log - lineno:278]: RunnerClient: rptest.tests.partition_movement_upgrade_test.PartitionMovementUpgradeTest.test_basic_upgrade: Summary: <BadLogLines nodes=docker-rp-6(20) example="ERROR 2022-08-07 21:29:42,458 [shard 0] rpc - server.cc:116 - kafka rpc protocol - Error[applying protocol] remote address: 172.16.16.9:54822 - std::runtime_error (Unsupported version 5 for join group API)">
Traceback (most recent call last):
  File "/usr/local/lib/python3.10/dist-packages/ducktape/tests/runner_client.py", line 135, in run
    data = self.run_test()
  File "/usr/local/lib/python3.10/dist-packages/ducktape/tests/runner_client.py", line 227, in run_test
    return self.test_context.function(self.test)
  File "/root/tests/rptest/services/cluster.py", line 48, in wrapped
    self.redpanda.raise_on_bad_logs(allow_list=log_allow_list)
  File "/root/tests/rptest/services/redpanda.py", line 1121, in raise_on_bad_logs
    raise BadLogLines(bad_lines)
rptest.services.utils.BadLogLines: <BadLogLines nodes=docker-rp-6(20) example="ERROR 2022-08-07 21:29:42,458 [shard 0] rpc - server.cc:116 - kafka rpc protocol - Error[applying protocol] remote address: 172.16.16.9:54822 - std::runtime_error (Unsupported version 5 for join group API)">

@dotnwat dotnwat mentioned this issue Aug 8, 2022
Merged
@dotnwat dotnwat closed this as completed in e800be0 Aug 8, 2022
ballard26 pushed a commit to ballard26/redpanda that referenced this issue Aug 8, 2022
@mmaslankaprv @ballard26
k/connection_context: use warn severity for unsupported api versions
aea17fd 
When client requests an unsupported API version it is not an unexpected
situation and should not be logged with ERROR severity. Introduced
separate exception handler for unsupported API version exceptions.

Fixes: redpanda-data#5868

Signed-off-by: Michal Maslanka <michal@redpanda.com>
@BenPope
Copy link
Member

BenPope commented Aug 9, 2022 •
edited

Seen again - seems to be the same as the one I reported yesterday: https://buildkite.com/redpanda/redpanda/builds/13844#018281c1-1808-413b-b7e8-9e72e8d0082e

[INFO  - 2022-08-09 09:41:23,203 - runner_client - log - lineno:278]: RunnerClient: rptest.tests.partition_movement_upgrade_test.PartitionMovementUpgradeTest.test_basic_upgrade: FAIL: <BadLogLines nodes=docker-rp-20(35) example="ERROR 2022-08-09 09:40:44,134 [shard 0] rpc - server.cc:116 - kafka rpc protocol - Error[applying protocol] remote address: 172.16.16.7:44242 - std::runtime_error (Unsupported version 5 for join group API)">
Traceback (most recent call last):
  File "/usr/local/lib/python3.10/dist-packages/ducktape/tests/runner_client.py", line 135, in run
    data = self.run_test()
  File "/usr/local/lib/python3.10/dist-packages/ducktape/tests/runner_client.py", line 227, in run_test
    return self.test_context.function(self.test)
  File "/root/tests/rptest/services/cluster.py", line 48, in wrapped
    self.redpanda.raise_on_bad_logs(allow_list=log_allow_list)
  File "/root/tests/rptest/services/redpanda.py", line 1125, in raise_on_bad_logs
    raise BadLogLines(bad_lines)
rptest.services.utils.BadLogLines: <BadLogLines nodes=docker-rp-20(35) example="ERROR 2022-08-09 09:40:44,134 [shard 0] rpc - server.cc:116 - kafka rpc protocol - Error[applying protocol] remote address: 172.16.16.7:44242 - std::runtime_error (Unsupported version 5 for join group API)">

@BenPope BenPope reopened this Aug 9, 2022
@dotnwat
Copy link
Member Author

dotnwat commented Aug 9, 2022

Seen again - seems to be the same as the one I reported yesterday: https://ci-artifacts.dev.vectorized.cloud/redpanda/018281c1-1808-413b-b7e8-9e72e8d0082e/vbuild/ducktape/results/2022-08-09--001/PartitionMovementUpgradeTest/test_basic_upgrade/137/

[INFO  - 2022-08-09 09:41:23,203 - runner_client - log - lineno:278]: RunnerClient: rptest.tests.partition_movement_upgrade_test.PartitionMovementUpgradeTest.test_basic_upgrade: FAIL: <BadLogLines nodes=docker-rp-20(35) example="ERROR 2022-08-09 09:40:44,134 [shard 0] rpc - server.cc:116 - kafka rpc protocol - Error[applying protocol] remote address: 172.16.16.7:44242 - std::runtime_error (Unsupported version 5 for join group API)">
Traceback (most recent call last):
  File "/usr/local/lib/python3.10/dist-packages/ducktape/tests/runner_client.py", line 135, in run
    data = self.run_test()
  File "/usr/local/lib/python3.10/dist-packages/ducktape/tests/runner_client.py", line 227, in run_test
    return self.test_context.function(self.test)
  File "/root/tests/rptest/services/cluster.py", line 48, in wrapped
    self.redpanda.raise_on_bad_logs(allow_list=log_allow_list)
  File "/root/tests/rptest/services/redpanda.py", line 1125, in raise_on_bad_logs
    raise BadLogLines(bad_lines)
rptest.services.utils.BadLogLines: <BadLogLines nodes=docker-rp-20(35) example="ERROR 2022-08-09 09:40:44,134 [shard 0] rpc - server.cc:116 - kafka rpc protocol - Error[applying protocol] remote address: 172.16.16.7:44242 - std::runtime_error (Unsupported version 5 for join group API)">

@BenPope can you link to the buildkite job (not the artifacts) or the PR?

@dotnwat
Copy link
Member Author

dotnwat commented Aug 9, 2022

@BenPope was that from a 22.1.x run? Perhaps it's just that the fix needs to be backported?

@dotnwat
Copy link
Member Author

dotnwat commented Aug 9, 2022

@mmaslankaprv ^

@BenPope
Copy link
Member

BenPope commented Aug 9, 2022 •
edited

@BenPope can you link to the buildkite job (not the artifacts) or the PR?

Comment updated.

@BenPope was that from a 22.1.x run? Perhaps it's just that the fix needs to be backported?

It was dev

@mmaslankaprv
Copy link
Member

I guess we are going to keep seeing this unless we backport the change

mmaslankaprv added a commit to mmaslankaprv/redpanda that referenced this issue Sep 30, 2022
@mmaslankaprv
tests: allow logging unsupported api version entry when upgrading
8c0ddb0 
Older redpanda versions may not support some of the APIs in versions
that the client currently use. Added allowed error log entry to
tolerate this situation during an upgrade test.

Fixes: redpanda-data#5868

Signed-off-by: Michal Maslanka <michal@redpanda.com>
@mmaslankaprv mmaslankaprv mentioned this issue Sep 30, 2022
Merged
6 tasks
mmaslankaprv added a commit to mmaslankaprv/redpanda that referenced this issue Sep 30, 2022
@mmaslankaprv
tests: allow logging unsupported api version entry when upgrading
87424e7 
Older redpanda versions may not support some of the APIs in versions
that the client currently use. Added allowed error log entry to
tolerate this situation during an upgrade test.

Fixes: redpanda-data#5868

Signed-off-by: Michal Maslanka <michal@redpanda.com>
mmaslankaprv added a commit to mmaslankaprv/redpanda that referenced this issue Oct 3, 2022
@mmaslankaprv
tests: allow logging unsupported api version entry when upgrading
12ca38e 
Older redpanda versions may not support some of the APIs in versions
that the client currently use. Added allowed error log entry to
tolerate this situation during an upgrade test.

Fixes: redpanda-data#5868

Signed-off-by: Michal Maslanka <michal@redpanda.com>
mmaslankaprv added a commit to mmaslankaprv/redpanda that referenced this issue Oct 3, 2022
@mmaslankaprv
tests: allow logging unsupported api version entry when upgrading
7513db4 
Older redpanda versions may not support some of the APIs in versions
that the client currently use. Added allowed error log entry to
tolerate this situation during an upgrade test.

Fixes: redpanda-data#5868

Signed-off-by: Michal Maslanka <michal@redpanda.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees

@mmaslankaprv mmaslankaprv

Labels
area/redpanda area/tests ci-disabled-test ci-failure kind/bug Something isn't working
Projects
None yet
Milestone
No milestone
6 participants
@dotnwat @BenPope @jcsp @rystsov @mmedenjak @mmaslankaprv