Support unknown content-length for put #285
Conversation
Codecov Report
Additional details and impacted files@@ Coverage Diff @@
## main #285 +/- ##
==========================================
+ Coverage 88.13% 88.28% +0.14%
==========================================
Files 16 16
Lines 4341 4412 +71
==========================================
+ Hits 3826 3895 +69
- Misses 515 517 +2
|
| add_net_test_case(test_s3_put_object_no_content_length) | ||
| add_net_test_case(test_s3_put_object_single_part_no_content_length) |
There was a problem hiding this comment.
make sure we have tests that:
- create a 0-size Object
- create a noop part, after creating 1+ normal parts
There was a problem hiding this comment.
test for 0 size object added. it uncovered some issues that were fixed.
existing tests already hit path for noop part, due to tests having at most 3 parts and optimistic scheduling was set at min of 5
Co-authored-by: Waqar Ahmed Khan <waahm7@gmail.com>
Co-authored-by: Michael Graeb <graebm@amazon.com>
Co-authored-by: Michael Graeb <graebm@amazon.com>
|
Hey - been waiting a long time for something like this - thanks for implementing. But experiencing performance problem - can you tell me what happens between the time I complete my request, and the AsyncUpload actually completes? I'm using AWS SDK V2 S3 API to do Async upload with CRT Async Client - works without specifying upload size now due to this update, just there's a huge lag once I'm done. Not sure where to go with this or who to ask. Ends up being much slower start to finish than our old workaround for unknown content size. Thanks! |
|
Please create an issue. This pull-request is a weird place to have a long debugging conversation Please explain your workload:
|
|
But to explain what's going on under the hood... since we don't know how much data will be sent, we assume it will be large and treat it as a Multipart Upload. We send CreateMultipartUpload, then send an UploadPart for each 8MiB of data, then send CompleteMultipartUpload when it's done. If you're sending files smaller than 8MiB lower performance is expected (if we knew the length in advance, we'd just send 1 PutObject). We have ideas on how to optimize this, but it adds a lot of complexity. It's something we hope to tackle in the future. |
|
If it's the lag after the send has ended, I would check the integration between how you create the input and how it's passed from Java to C. As in C, we check the input_stream status to complete the MPU. |
|
Thanks for the quick response - will create an issue and if you think it's unrelated to aws-c-s3 then we can ship it to s3-transfer-manager. |
Issue #, if available:
Description of changes:
Support unknown content length for put
By submitting this pull request, I confirm that my contribution is made under the terms of the Apache 2.0 license.