s3: add info on multipart, fallback to PutObject, and completing uplo…

…ads (#1127)

Signed-off-by: Wesley Pettit <wppttt@amazon.com>

pipeline/outputs/s3.md

@@ -206,6 +206,43 @@ The following settings are recommended for this use case:
      use_put_object On
 ```
 
+## S3 Multipart Uploads
+
+With `use_put_object Off` (default), S3 will attempt to send files using multipart uploads. For each file, S3 first calls [CreateMultipartUpload](https://docs.aws.amazon.com/AmazonS3/latest/API/API_CreateMultipartUpload.html), then a series of calls to [UploadPart](https://docs.aws.amazon.com/AmazonS3/latest/API/API_UploadPart.html) for each fragment (targeted to be `upload_chunk_size` bytes), and finally [CompleteMultipartUpload](https://docs.aws.amazon.com/AmazonS3/latest/API/API_CompleteMultipartUpload.html) to create the final file in S3. 
+
+### Fallback to PutObject
+
+S3 [requires](https://docs.aws.amazon.com/AmazonS3/latest/userguide/qfacts.html) each [UploadPart](https://docs.aws.amazon.com/AmazonS3/latest/API/API_UploadPart.html) fragment to be at least 5,242,880 bytes, otherwise the upload is rejected. 
+
+Consequently, the S3 output must sometimes fallback to the [PutObject API](https://docs.aws.amazon.com/AmazonS3/latest/API/API_PutObject.html). 
+
+Uploads are triggered by three settings:
+1. `total_file_size` and `upload_chunk_size`: When S3 has buffered data in the `store_dir` that meets the desired `total_file_size` (for `use_put_object On`) or the `upload_chunk_size` (for Multipart), it will trigger an upload operation. 
+2. `upload_timeout`: Whenever locally buffered data has been present on the filesystem in the `store_dir` longer than the configured `upload_timeout`, it will be sent. This happens regardless of whether or not the desired byte size has been reached. Consequently, if you configure a small `upload_timeout`, your files may be smaller than the `total_file_size`. The timeout is evaluated against the time at which S3 started buffering data for each unqiue tag (that is, the time when new data was buffered for the unique tag after the last upload). The timeout is also evaluated against the [CreateMultipartUpload](https://docs.aws.amazon.com/AmazonS3/latest/API/API_CreateMultipartUpload.html) time, so a multipart upload will be completed after `upload_timeout` has elapsed, even if the desired size has not yet been reached. 
+
+If your `upload_timeout` triggers an upload before the pending buffered data reaches the `upload_chunk_size`, it may be too small for a multipart upload. S3 will consequently fallback to use the [PutObject API](https://docs.aws.amazon.com/AmazonS3/latest/API/API_PutObject.html).
+
+When you enable compression, S3 applies the compression algorithm at send time. The size settings noted above trigger uploads based on the size of buffered data, not the final compressed size. Consequently, it is possible that after compression, buffered data no longer meets the required minimum S3 [UploadPart](https://docs.aws.amazon.com/AmazonS3/latest/API/API_UploadPart.html) size. If this occurs, you will see a log message like:
+
+
+```
+[ info] [output:s3:s3.0] Pre-compression upload_chunk_size= 5630650, After compression, chunk is only 1063320 bytes, the chunk was too small, using PutObject to upload
+```
+
+If you encounter this frequently, use the numbers in the messages to guess your compression factor. For example, in this case, the buffered data was reduced from 5,630,650 bytes to 1,063,320 bytes. The compressed size is 1/5 the actual data size, so configuring `upload_chunk_size 30M` should ensure each part is large enough after compression to be over the min required part size of 5,242,880 bytes.
+
+The S3 API allows the last part in an upload to be less than the 5,242,880 byte minimum. Therefore, if a part is too small for an existing upload, the S3 output will upload that part and then complete the upload. 
+
+### upload_timeout constrains total multipart upload time for a single file
+
+The `upload_timeout` is evaluated against the [CreateMultipartUpload](https://docs.aws.amazon.com/AmazonS3/latest/API/API_CreateMultipartUpload.html) time. So a multipart upload will be completed after `upload_timeout` has elapsed, even if the desired size has not yet been reached. 
+
+### Completing uploads
+
+When [CreateMultipartUpload](https://docs.aws.amazon.com/AmazonS3/latest/API/API_CreateMultipartUpload.html) is called, an `UploadID` is returned. S3 stores these IDs for active uploads in the `store_dir`. Until [CompleteMultipartUpload](https://docs.aws.amazon.com/AmazonS3/latest/API/API_CompleteMultipartUpload.html) is called, the uploaded data will not be visible in S3. 
+
+On shutdown, S3 output will attempt to complete all pending uploads. If it fails to complete an upload, the ID will remain buffered in the `store_dir` in a directory called `multipart_upload_metadata`. If you restart the S3 output with the same `store_dir` it will discover the old UploadIDs and complete the pending uploads. The [S3 documentation](https://aws.amazon.com/blogs/aws-cloud-financial-management/discovering-and-deleting-incomplete-multipart-uploads-to-lower-amazon-s3-costs/) also has suggestions on discovering and deleting/completing dangling uploads in your buckets.  
+
 ## Worker support
 
 Fluent Bit 1.7 adds a new feature called `workers` which enables outputs to have dedicated threads. This `s3` plugin has partial support for workers. **The plugin can only support a single worker; enabling multiple workers will lead to errors/indeterminate behavior.**