Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Use http-intake.logs for logs intake #521

Merged
merged 2 commits into from
Dec 21, 2021
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
18 changes: 10 additions & 8 deletions aws/logs_monitoring/README.md
Original file line number Diff line number Diff line change
Expand Up @@ -214,14 +214,16 @@ If you need to ship logs to multiple Datadog organizations or other destinations

### AWS PrivateLink support

You can run the Forwarder in a VPC by using AWS PrivateLink to connect to Datadog. Note that AWS PrivateLink can only be configured with Datadog organizations using the Datadog US site (i.e. datadoghq.com, not datadoghq.eu).
You can run the Forwarder in a VPC private subnet and send data to Datadog over AWS PrivateLink. Note that AWS PrivateLink can only be configured with [Datadog Sites](https://docs.datadoghq.com/getting_started/site/) hosted on AWS (i.e. datadoghq.com, not datadoghq.eu).

1. Follow the [setup instructions](https://docs.datadoghq.com/agent/guide/private-link/?tab=logs#create-your-vpc-endpoint) to add an endpoint to your VPC for Datadog's **API** service.
2. Follow the [same procedure](https://docs.datadoghq.com/agent/guide/private-link/?tab=logs#create-your-vpc-endpoint) to add a second endpoint to your VPC for Datadog's **Logs** service.
3. Follow the [same procedure](https://docs.datadoghq.com/agent/guide/private-link/?tab=logs#create-your-vpc-endpoint) once more to add a third endpoint to your VPC for Datadog's **Traces** service.
4. Unless the Forwarder is deployed to a public subnet, follow the [instructions](https://docs.aws.amazon.com/vpc/latest/userguide/vpce-interface.html#create-interface-endpoint) to add endpoints for Secrets Manager and S3 to the VPC, so that the Forwarder can access those services.
5. When installing the Forwarder with the CloudFormation template, set `DdUsePrivateLink`, `VPCSecurityGroupIds` and `VPCSubnetIds`.
6. Ensure the `DdFetchLambdaTags` option is disabled, because AWS VPC does not yet offer an endpoint for the Resource Groups Tagging API.
1. Follow the [instructions](https://docs.datadoghq.com/agent/guide/private-link/?tab=logs#create-your-vpc-endpoint) to add the Datadog `api`, `http-logs.intake` and `trace.agent` endpoints to your VPC.
2. Follow the [instructions](https://docs.aws.amazon.com/vpc/latest/userguide/vpce-interface.html#create-interface-endpoint) to add the AWS Secrets Manager and S3 endpoints to your VPC.
3. When installing the Forwarder with the CloudFormation template,
1. set `UseVPC` to `true`
2. set `VPCSecurityGroupIds` and `VPCSubnetIds` based on your VPC settings
3. set `DdFetchLambdaTags` to `false`, because AWS Resource Groups Tagging API doesn't support PrivateLink

NOTE: The `DdUsePrivateLink` option has been deprecated. It was previously used to instruct the Forwarder to use a special set of Datadog endpoints for intake. If you have `DdUsePrivateLink` enabled, keep it that way, unless you follow the instructions above to add the Datadog `api`, `http-logs.intake` and `trace.agent` endpoints to your VPC.

### AWS VPC and proxy support

Expand Down Expand Up @@ -362,7 +364,7 @@ To test different patterns against your logs, turn on [debug logs](#troubleshoot
`PermissionBoundaryArn`
: ARN for the Permissions Boundary Policy.

`DdUsePrivateLink`
`DdUsePrivateLink` (DEPRECATED)
: Set to true to enable sending logs and metrics via AWS PrivateLink. See https://dtdg.co/private-link.

`DdHttpProxyURL`
Expand Down
10 changes: 7 additions & 3 deletions aws/logs_monitoring/settings.py
Original file line number Diff line number Diff line change
Expand Up @@ -115,6 +115,7 @@ def get_env_var(envvar, default, boolean=False):
default="{}://trace.agent.{}".format("http" if DD_NO_SSL else "https", DD_SITE),
)

# The TCP transport has been deprecated, migrate to the HTTP intake.
if DD_USE_TCP:
DD_URL = get_env_var("DD_URL", default="lambda-intake.logs." + DD_SITE)
try:
Expand All @@ -125,14 +126,17 @@ def get_env_var(envvar, default, boolean=False):
except Exception:
DD_PORT = 10516
else:
DD_URL = get_env_var("DD_URL", default="lambda-http-intake.logs." + DD_SITE)
DD_URL = get_env_var("DD_URL", default="http-intake.logs." + DD_SITE)
DD_PORT = int(get_env_var("DD_PORT", default="443"))

## @param DD_USE_VPC
DD_USE_VPC = get_env_var("DD_USE_VPC", "false", boolean=True)

## @param DD_USE_PRIVATE_LINK - whether to forward logs via PrivateLink
## Overrides incompatible settings
# DEPRECATED. No longer need to use special endpoints, as you can now expose
# regular Datadog API endpoints `api`, `http-intake.logs` and `trace.agent`
# via PrivateLink. See https://docs.datadoghq.com/agent/guide/private-link/.
# @param DD_USE_PRIVATE_LINK - whether to forward logs via PrivateLink
# Overrides incompatible settings
#
DD_USE_PRIVATE_LINK = get_env_var("DD_USE_PRIVATE_LINK", "false", boolean=True)
if DD_USE_PRIVATE_LINK:
Expand Down
4 changes: 1 addition & 3 deletions aws/logs_monitoring/template.yaml
Original file line number Diff line number Diff line change
Expand Up @@ -157,7 +157,7 @@ Parameters:
AllowedValues:
- true
- false
Description: Set to true to deploy the Forwarder to a VPC and send logs, metrics, and traces via AWS PrivateLink. When set to true, must also set VPCSecurityGroupIds and VPCSubnetIds. Find more details from https://dtdg.co/private-link.
Description: DEPRECATED, DO NOT CHANGE. See README.md for details. Set to true to deploy the Forwarder to a VPC and send logs, metrics, and traces via AWS PrivateLink. When set to true, must also set VPCSecurityGroupIds and VPCSubnetIds.
DdUseVPC:
Type: String
Default: false
Expand Down Expand Up @@ -450,8 +450,6 @@ Resources:
- Ref: AWS::NoValue
DD_SITE:
Ref: DdSite
DD_:
Ref: DdSite
DD_TAGS:
Fn::If:
- SetDdTags
Expand Down