Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

elasticsearch:opensearch: add documentation for aws_service_name option #1032

Closed
wants to merge 1 commit into from
Closed

elasticsearch:opensearch: add documentation for aws_service_name option #1032

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
1 commit
Select commit
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
1 change: 1 addition & 0 deletions pipeline/outputs/elasticsearch.md
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -21,6 +21,7 @@ The **es** output plugin, allows to ingest your records into an [Elasticsearch](
| AWS\_STS\_Endpoint | Specify the custom sts endpoint to be used with STS API for Amazon OpenSearch Service | |
| AWS\_Role\_ARN | AWS IAM Role to assume to put records to your Amazon cluster | |
| AWS\_External\_ID | External ID for the AWS IAM Role specified with `aws_role_arn` | |
| AWS\_Service\_Name | Service name to be used in AWS Sigv4 signature. For integration with Amazon OpenSearch Serverless, set to `aoss`. See the [FAQ](opensearch.md#faq) section on Amazon OpenSearch Serverless for more information. | es |
Copy link
Contributor

@PettitWesley PettitWesley Mar 1, 2023 •
edited
Loading

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I'm worried this relative FAQ link won't work but I guess it does...

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I think it does work. I tried it in vscode and the reference was found. In the diff, it takes me to https://github.com/fluent/fluent-bit-docs/blob/aoss/pipeline/outputs/opensearch.md

| Cloud\_ID | If you are using Elastic's Elasticsearch Service you can specify the cloud\_id of the cluster running. The Cloud ID string has the format `<deployment_name>:<base64_info>`. Once decoded, the `base64_info` string has the format `<deployment_region>$<elasticsearch_hostname>$<kibana_hostname>`.
| |
| Cloud\_Auth | Specify the credentials to use to connect to Elastic's Elasticsearch Service running on Elastic Cloud | |
Expand Down
21 changes: 20 additions & 1 deletion pipeline/outputs/opensearch.md
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -21,10 +21,11 @@ The following instructions assumes that you have a fully operational OpenSearch
| AWS\_STS\_Endpoint | Specify the custom sts endpoint to be used with STS API for Amazon OpenSearch Service | |
| AWS\_Role\_ARN | AWS IAM Role to assume to put records to your Amazon cluster | |
| AWS\_External\_ID | External ID for the AWS IAM Role specified with `aws_role_arn` | |
| AWS\_Service\_Name | Service name to be used in AWS Sigv4 signature. For integration with Amazon OpenSearch Serverless, set to `aoss`. See the [FAQ](opensearch.md#faq) section on Amazon OpenSearch Serverless for more information. | es |
| HTTP\_User | Optional username credential for access | |
| HTTP\_Passwd | Password for user defined in HTTP\_User | |
| Index | Index name, supports [Record Accessor syntax](../../administration/configuring-fluent-bit/classic-mode/record-accessor.md) from 2.0.5 onwards. | fluent-bit |
| Type | Type name | \_doc |
| Type | Type name. This option is ignored if `Suppress_Type_Name` is enabled. | \_doc |
| Logstash\_Format | Enable Logstash format compatibility. This option takes a boolean value: True/False, On/Off | Off |
| Logstash\_Prefix | When Logstash\_Format is enabled, the Index name is composed using a prefix and the date, e.g: If Logstash\_Prefix is equals to 'mydata' your index will become 'mydata-YYYY.MM.DD'. The last string appended belongs to the date when the data is being generated. | logstash |
| Logstash\_DateFormat | Time format \(based on [strftime](http://man7.org/linux/man-pages/man3/strftime.3.html)\) to generate the second part of the Index name. | %Y.%m.%d |
Expand Down Expand Up @@ -157,3 +158,21 @@ Without this you will see errors like:
```text
{"error":{"root_cause":[{"type":"illegal_argument_exception","reason":"Action/metadata line [1] contains an unknown parameter [_type]"}],"type":"illegal_argument_exception","reason":"Action/metadata line [1] contains an unknown parameter [_type]"},"status":400}
```

### Fluent-Bit + Amazon OpenSearch Serverless <a id="#aws-opensearch-serverless"></a>
Amazon OpenSearch Serverless is an offering that eliminates your need to manage OpenSearch clusters. All existing Fluent Bit OpenSearch output plugin options work with OpenSearch Serverless. For Fluent Bit, the only difference is that you must specify the service name as `aoss` (Amazon OpenSearch Serverless) when you enable `AWS_Auth`:
```
AWS_Auth On
AWS_Region <aws-region>
AWS_Service_Name aoss
```

**Data Access Permissions**

When sending logs to OpenSearch Serverless, your AWS IAM entity needs [OpenSearch Serverless Data Access permisions](https://docs.aws.amazon.com/opensearch-service/latest/developerguide/serverless-data-access.html). Give your IAM entity the following data access permissions to your serverless collection:
```
aoss:CreateIndex
aoss:UpdateIndex
aoss:WriteDocument
```
With data access permissions, IAM policies are not needed to access the collection.