Skip to content

Commit

Permalink
feat(firehose-destinations-alpha): support zero buffering (#28716)
Browse files Browse the repository at this point in the history
support zero buffering ([announcement](https://aws.amazon.com/about-aws/whats-new/2023/12/amazon-kinesis-data-firehose-zero-buffering/), [documentation](https://docs.aws.amazon.com/firehose/latest/dev/buffering-hints.html))

Closes #28714 

----

*By submitting this pull request, I confirm that my contribution is made under the terms of the Apache-2.0 license*
  • Loading branch information
humanzz committed Jan 16, 2024
1 parent c3802c4 commit db2e78e
Show file tree
Hide file tree
Showing 11 changed files with 439 additions and 20 deletions.
16 changes: 16 additions & 0 deletions packages/@aws-cdk/aws-kinesisfirehose-alpha/README.md
Original file line number Diff line number Diff line change
Expand Up @@ -299,6 +299,22 @@ new firehose.DeliveryStream(this, 'Delivery Stream', {
See: [Data Delivery Frequency](https://docs.aws.amazon.com/firehose/latest/dev/basic-deliver.html#frequency)
in the *Kinesis Data Firehose Developer Guide*.

Zero buffering, where Amazon Kinesis Data Firehose stream can be configured to not buffer data before delivery, is supported by
setting the "buffer interval" to 0.

```ts
// Setup zero buffering
declare const bucket: s3.Bucket;
const destination = new destinations.S3Bucket(bucket, {
bufferingInterval: Duration.seconds(0),
});
new firehose.DeliveryStream(this, 'ZeroBufferDeliveryStream', {
destinations: [destination],
});
```

See: [Buffering Hints](https://docs.aws.amazon.com/firehose/latest/dev/buffering-hints.html).

## Destination Encryption

Your data can be automatically encrypted when it is delivered to S3 as a final or an
Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -86,7 +86,7 @@ export interface CommonDestinationS3Props {
* The length of time that Firehose buffers incoming data before delivering
* it to the S3 bucket.
*
* Minimum: Duration.seconds(60)
* Minimum: Duration.seconds(0)
* Maximum: Duration.seconds(900)
*
* @default Duration.seconds(300)
Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -87,8 +87,8 @@ export function createBufferingHints(
}

const intervalInSeconds = interval?.toSeconds() ?? 300;
if (intervalInSeconds < 60 || intervalInSeconds > 900) {
throw new Error(`Buffering interval must be between 60 and 900 seconds. Buffering interval provided was ${intervalInSeconds} seconds.`);
if (intervalInSeconds > 900) {
throw new Error(`Buffering interval must be less than 900 seconds. Buffering interval provided was ${intervalInSeconds} seconds.`);
}
const sizeInMBs = size?.toMebibytes() ?? 5;
if (sizeInMBs < 1 || sizeInMBs > 128) {
Expand Down

Some generated files are not rendered by default. Learn more about how customized files appear on GitHub.

Original file line number Diff line number Diff line change
Expand Up @@ -680,6 +680,146 @@
"DependsOn": [
"DeliveryStreamS3DestinationRoleDefaultPolicy3015D8C7"
]
},
"ZeroBufferingDeliveryStreamS3DestinationRole606BB6E3": {
"Type": "AWS::IAM::Role",
"Properties": {
"AssumeRolePolicyDocument": {
"Statement": [
{
"Action": "sts:AssumeRole",
"Effect": "Allow",
"Principal": {
"Service": "firehose.amazonaws.com"
}
}
],
"Version": "2012-10-17"
}
}
},
"ZeroBufferingDeliveryStreamS3DestinationRoleDefaultPolicyC03FC12E": {
"Type": "AWS::IAM::Policy",
"Properties": {
"PolicyDocument": {
"Statement": [
{
"Action": [
"s3:Abort*",
"s3:DeleteObject*",
"s3:GetBucket*",
"s3:GetObject*",
"s3:List*",
"s3:PutObject",
"s3:PutObjectLegalHold",
"s3:PutObjectRetention",
"s3:PutObjectTagging",
"s3:PutObjectVersionTagging"
],
"Effect": "Allow",
"Resource": [
{
"Fn::GetAtt": [
"Bucket83908E77",
"Arn"
]
},
{
"Fn::Join": [
"",
[
{
"Fn::GetAtt": [
"Bucket83908E77",
"Arn"
]
},
"/*"
]
]
}
]
},
{
"Action": [
"logs:CreateLogStream",
"logs:PutLogEvents"
],
"Effect": "Allow",
"Resource": {
"Fn::GetAtt": [
"ZeroBufferingDeliveryStreamLogGroupFA68202E",
"Arn"
]
}
}
],
"Version": "2012-10-17"
},
"PolicyName": "ZeroBufferingDeliveryStreamS3DestinationRoleDefaultPolicyC03FC12E",
"Roles": [
{
"Ref": "ZeroBufferingDeliveryStreamS3DestinationRole606BB6E3"
}
]
}
},
"ZeroBufferingDeliveryStreamLogGroupFA68202E": {
"Type": "AWS::Logs::LogGroup",
"Properties": {
"RetentionInDays": 731
},
"UpdateReplacePolicy": "Retain",
"DeletionPolicy": "Retain"
},
"ZeroBufferingDeliveryStreamLogGroupS3Destination806CFD96": {
"Type": "AWS::Logs::LogStream",
"Properties": {
"LogGroupName": {
"Ref": "ZeroBufferingDeliveryStreamLogGroupFA68202E"
}
},
"UpdateReplacePolicy": "Retain",
"DeletionPolicy": "Retain"
},
"ZeroBufferingDeliveryStream7C2555C9": {
"Type": "AWS::KinesisFirehose::DeliveryStream",
"Properties": {
"DeliveryStreamType": "DirectPut",
"ExtendedS3DestinationConfiguration": {
"BucketARN": {
"Fn::GetAtt": [
"Bucket83908E77",
"Arn"
]
},
"BufferingHints": {
"IntervalInSeconds": 0,
"SizeInMBs": 5
},
"CloudWatchLoggingOptions": {
"Enabled": true,
"LogGroupName": {
"Ref": "ZeroBufferingDeliveryStreamLogGroupFA68202E"
},
"LogStreamName": {
"Ref": "ZeroBufferingDeliveryStreamLogGroupS3Destination806CFD96"
}
},
"CompressionFormat": "GZIP",
"ErrorOutputPrefix": "errorPrefix",
"Prefix": "regularPrefix",
"RoleARN": {
"Fn::GetAtt": [
"ZeroBufferingDeliveryStreamS3DestinationRole606BB6E3",
"Arn"
]
}
}
},
"DependsOn": [
"ZeroBufferingDeliveryStreamS3DestinationRoleDefaultPolicyC03FC12E"
]
}
},
"Mappings": {
Expand Down

Some generated files are not rendered by default. Learn more about how customized files appear on GitHub.

Some generated files are not rendered by default. Learn more about how customized files appear on GitHub.

Some generated files are not rendered by default. Learn more about how customized files appear on GitHub.

Loading

0 comments on commit db2e78e

Please sign in to comment.