PrivateLink URLs are different from the DNS addresses exposed by the VPC Endpoints

[shshe]

Describe what happened:

Enabling DD_USE_PRIVATE_LINK=true on the AWS lambda function is supposed to set the DD URLs to use their PrivateLink addresses. I've added all of the VPC endpoints described here:
https://docs.datadoghq.com/agent/guide/private-link/?tab=metrics

The code block here sets DD_API_URL to https://pvtlink.api.datadoghq.com:
https://github.com/DataDog/datadog-serverless-functions/blob/master/aws/logs_monitoring/settings.py#L134-L149

However, I have no private DNS mapping for pvtlink.api.datadoghq.com after adding all the VPC endpoints.

Describe what you expected:

Setting DD_USE_PRIVATE_LINK=true should set URLs for private DNS addresses that exist.

Steps to reproduce the issue:

Enable DD_USE_PRIVATE_LINK=true. Start seeing some 2min timeouts (and getting some pretty expensive lambda charges).

Workaround:
I've had to set:

DD_USE_PRIVATE_LINK=false
DD_URL: api-pvtlink.logs.datadoghq.com
DD_API_URL: https://api.datadoghq.com

[tianchu]

@shshe Thanks for reporting this issue! I can confirm this is indeed a problem (ironically) due to our recent effort to simplify PrivateLink configurations for agent users DataDog/documentation#10669. New PrivateLink endpoints were configured so that our regular data intake endpoints would automatically resolve to the PrivateLink endpoints without extra configurations on the agent to override the URLs. However, the endpoint to intake logs from the Forwarder was apparently missed.

I don't think DD_API_URL: https://api.datadoghq.com is necessary though, because with DD_USE_PRIVATE_LINK=false, DD_API_URL would be set to https://api.datadoghq.com already

datadog-serverless-functions/aws/logs_monitoring/settings.py

Lines 107 to 110 in 6ad3c5c

	DD_API_URL = get_env_var(
	"DD_API_URL",
	default="{}://api.{}".format("http" if DD_NO_SSL else "https", DD_SITE),
	)

We are working on setting up a new PrivateLink endpoint for Forwarder log intake as well, and with that endpoint configured, DD_USE_PRIVATE_LINK would be fully deprecated for new installations of forwarder, because PrivateLink would work OOB without the need of redirecting to a different set of URLs.