Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Not all data is uploaded to S3 bucket in EndToEndTopicRecovery.test_restore #5474

Closed
dotnwat opened this issue Jul 14, 2022 · 9 comments · Fixed by #5544 or #5736
Closed

Not all data is uploaded to S3 bucket in EndToEndTopicRecovery.test_restore #5474

dotnwat opened this issue Jul 14, 2022 · 9 comments · Fixed by #5544 or #5736
Assignees
@LenaAn
Labels
area/cloud-storage Shadow indexing subsystem ci-failure kind/bug Something isn't working

Comments

@dotnwat
Copy link
Member

dotnwat commented Jul 14, 2022 •
edited

https://buildkite.com/redpanda/redpanda/builds/12555#0181fdac-cac8-4f45-b68c-aaf87809e4ab/1637-8802



test_id:    rptest.tests.e2e_topic_recovery_test.EndToEndTopicRecovery.test_restore.message_size=10000.num_messages=100000.recovery_overrides=.retention.bytes.1024
--
  | status:     FAIL
  | run time:   6 minutes 36.566 seconds
  |  
  |  
  | TimeoutError('Not all data is uploaded to S3 bucket')
  | Traceback (most recent call last):
  | File "/usr/local/lib/python3.9/dist-packages/ducktape/tests/runner_client.py", line 135, in run
  | data = self.run_test()
  | File "/usr/local/lib/python3.9/dist-packages/ducktape/tests/runner_client.py", line 227, in run_test
  | return self.test_context.function(self.test)
  | File "/usr/local/lib/python3.9/dist-packages/ducktape/mark/_mark.py", line 476, in wrapper
  | return functools.partial(f, *args, **kwargs)(*w_args, **w_kwargs)
  | File "/root/tests/rptest/services/cluster.py", line 35, in wrapped
  | r = f(self, *args, **kwargs)
  | File "/root/tests/rptest/tests/e2e_topic_recovery_test.py", line 145, in test_restore
  | wait_until(s3_has_all_data,
  | File "/usr/local/lib/python3.9/dist-packages/ducktape/utils/util.py", line 58, in wait_until
  | raise TimeoutError(err_msg() if callable(err_msg) else err_msg) from last_exception
  | ducktape.errors.TimeoutError: Not all data is uploaded to S3 bucket

See also #5241
@dotnwat dotnwat added kind/bug Something isn't working ci-failure area/cloud-storage Shadow indexing subsystem labels Jul 14, 2022
This was referenced Jul 14, 2022
Merged
Merged
@LenaAn
Copy link
Contributor

LenaAn commented Jul 19, 2022

another one https://buildkite.com/redpanda/redpanda/builds/12695#01821257-ffe7-4eea-a698-6050b414a48d

@dotnwat dotnwat mentioned this issue Jul 19, 2022
Merged
@BenPope BenPope mentioned this issue Jul 19, 2022
Merged
@VadimPlh
Copy link
Contributor

Again https://buildkite.com/redpanda/redpanda/builds/12972#018229b7-81cd-4954-a559-0a6c13b7cefb

@LenaAn
Copy link
Contributor

LenaAn commented Jul 25, 2022 •
edited

#5544 should fix it

UPD: put the right link

@jcsp
Copy link
Contributor

jcsp commented Jul 25, 2022

@LenaAn I think you meant to type a link to a PR, but it's a link back to this ticket

@jcsp jcsp mentioned this issue Jul 25, 2022
Merged
@mmedenjak mmedenjak mentioned this issue Jul 26, 2022
Merged
@abhijat abhijat mentioned this issue Jul 27, 2022
Merged
@ZeDRoman ZeDRoman mentioned this issue Jul 28, 2022
Merged
@VladLazar
Copy link
Contributor

Also seen in: https://buildkite.com/redpanda/redpanda/builds/13265#0182453f-2665-4897-8c20-755ee0fc74c3

@jcsp jcsp mentioned this issue Jul 28, 2022
Merged
@VadimPlh
Copy link
Contributor

Again https://buildkite.com/redpanda/redpanda/builds/13301#018248b0-0c96-4336-9387-bda3bcd7bb7b

@Lazin
Copy link
Contributor

Lazin commented Jul 29, 2022

I think it won't be fixed by #5544. I think that we just need to increase timeout or change the scale. The test generates a lot of small segments and simple not able to upload everything fast enough.

Lazin added a commit to Lazin/redpanda that referenced this issue Jul 29, 2022
@Lazin
ducktape: Fix topic recovery e2e test
53c3e2a 
Fixes redpanda-data#5474 'Not all data is uploaded to S3 bucket'.
The problem is caused by the fact that we're generating huge number of
segments in this test (message size was 10000, and number of messages
was 100000). This commit decreases the message size to 5000 and
increases the timeout for upload to 10 minutes.
@Lazin Lazin mentioned this issue Jul 29, 2022
Merged
5 tasks
@r-vasquez r-vasquez mentioned this issue Jul 29, 2022
Merged
@jcsp
Copy link
Contributor

jcsp commented Aug 1, 2022

I think it won't be fixed by #5544.

We didn't meant to close this via #5544 then, right?

This is failing 10% of the time over the weekend, e.g. https://buildkite.com/redpanda/redpanda/builds/13372#018252c5-c44e-416c-93fe-49609f5c1936

@jcsp jcsp reopened this Aug 1, 2022
@Lazin
Copy link
Contributor

Lazin commented Aug 1, 2022

Correct. This PR #5736 should fix that.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees

@LenaAn LenaAn

Labels
area/cloud-storage Shadow indexing subsystem ci-failure kind/bug Something isn't working
Projects
None yet
Milestone
No milestone
6 participants
@Lazin @dotnwat @jcsp @LenaAn @VadimPlh @VladLazar