Firehose & Kinesis Output plugin: time_key does not have milliseconds.

[taisyo7333]

Bug Report

Describe the bug

When I use firehose output plugin.
time_key does not have milliseconds , only seconds resolution.
time_key records is like the below.

 "@timestamp":"2020-12-04T06:16:04"

To Reproduce

• Rubular link if applicable:
• Example log message if applicable:
• Steps to reproduce the problem:

1. Provision Kinesis firehose and S3 bucket.
2. I use the follwing output config.
3. firehose forwards data to S3 bucket.
4. you can check logs in your S3 bucket.

[OUTPUT]
        Name             kinesis_firehose
        Match            *
        region           ${FLUENT_FIREHOSE_REGION}
        delivery_stream  ${FLUENT_FIREHOSE_DELIVERY_STREAM_NAME}
        time_key         @timestamp

Expected behavior

The time_key should have milliseconds resolution.

"@timestamp":"2020-12-04T06:16:04.789"

Screenshots

Your Environment

• Version used: v1.6.8
• Configuration:

[OUTPUT]
        Name             kinesis_firehose
        Match            *
        region           ${FLUENT_FIREHOSE_REGION}
        delivery_stream  ${FLUENT_FIREHOSE_DELIVERY_STREAM_NAME}
        time_key         @timestamp

• Environment name and version (e.g. Kubernetes? What version?):
  • kubernetes
    • Amazon EKS
    • v1.18.9-eks-d1db3
• Server type and version:
• Operating System and version:
  • Amazon Linux 2
• Filters and plugins:
  • Output kinesis_firehose plugin.

Additional context

we use firehose as logging architecture.

I think struct tm does not support milliseconds resolution.
This code uses struct tm.

https://github.com/fluent/fluent-bit/blob/master/plugins/out_kinesis_firehose/firehose_api.c#L233

[PettitWesley]

The code also uses strftime, which doesn't support anything smaller than seconds.

To support milliseconds you have to build a shim on top of strftime. I vaguely remember that some part of fluent bit supports millisecond format specifiers, I think it was %f, but I can't find any code from a cursory search.

[github-actions]

This issue is stale because it has been open 30 days with no activity. Remove stale label or comment or this will be closed in 5 days.

[github-actions]

This issue is stale because it has been open 30 days with no activity. Remove stale label or comment or this will be closed in 5 days.

[github-actions]

This issue was closed because it has been stalled for 5 days with no activity.

[github-actions]

This issue is stale because it has been open 90 days with no activity. Remove stale label or comment or this will be closed in 5 days. Maintainers can add the exempt-stale label.

[github-actions]

This issue was closed because it has been stalled for 5 days with no activity.

[PettitWesley]

Marking as enhancement so it won't keep getting closed

[PettitWesley]

This should be implemented in both kinesis_firehose and kinesis_streams when we get around to it

[Claych]

I'm working on this part, hopefully nanoseconds and milliseconds can be implemented for time_key by the end of this month.

[lubingfeng]

@Claych are we done with this one?

[matthewfala]

This will be added to aws-for-fluent-bit version 2.31.0, cherrypicked on top of Fluent Bit version 1.9.10.

Please see: #6486

[hackery]

@matthewfala looks like this can be closed (although I think you linked a different PR above, which is for out_s3) - but see #7538 which describes a bug in the sub-second formatting (printf format string needs a leading-zeros specifier like %03u but I don't know how that works vs. %I64u)

[PettitWesley]

@hackery Thanks, I have opened a new issue for the leading zero bug fix: aws/aws-for-fluent-bit#740