User Agent 2.0 #2955

jonemo · 2023-05-26T00:03:21Z

This PR implements an updated User-Agent header format that is standardized across AWS SDKs. For example, the AWS Ruby SDK's equivalent PR is aws/aws-sdk-ruby#2854.

An example User-Agent header value before the change is:

Boto3/1.20.32 Python/3.9.13 Darwin/22.4.0 exec-env/AWS_Lambda_python3.7 Botocore/1.23.32

An example User-Agent header value after the change is:

Boto3/1.20.32 md/Botocore#1.23.32 ua/2.0 lang/python#3.7.16 os/linux#4.14.255-301-238.520.AMZN2.X86_64 exec-env/AWS_Lambda_python3.7

Various notes:

This PR includes the formerly separate (but then closed) change to add the AWS_SDK_UA_APP_ID config: Support for sdk_ua_app_id config setting #2942
Users who use the Config.user_agent will not see a change in their User-Agent header string
Both Session.user_agent_extra and Config.user_agent_extra continue to be appear at the end of the User-Agent header
Session.user_agent() is no longer used internally in Botocore but remains in place and unchanged for any third party users
In contrast to the Ruby SDK PR linked above, this PR does not include feature-related or any other request-level information in the User-Agent string.

codecov-commenter · 2023-05-26T15:29:25Z

Codecov Report

Patch coverage: 93.86% and project coverage change: +0.03 🎉

Comparison is base (8bd1a4d) 93.29% compared to head (10b11b6) 93.32%.

❗ Current head 10b11b6 differs from pull request most recent head d33a106. Consider uploading reports for the commit d33a106 to get more accurate results

❗ Your organization is not using the GitHub App Integration. As a result you may experience degraded service beginning May 15th. Please install the Github App Integration for your organization. Read more.

Additional details and impacted files

@@             Coverage Diff             @@
##           develop    #2955      +/-   ##
===========================================
+ Coverage    93.29%   93.32%   +0.03%     
===========================================
  Files           64       65       +1     
  Lines        13588    13742     +154     
===========================================
+ Hits         12677    12825     +148     
- Misses         911      917       +6

Impacted Files	Coverage Δ
botocore/config.py	`100.00% <ø> (ø)`
botocore/configprovider.py	`94.09% <ø> (ø)`
botocore/session.py	`97.64% <80.00%> (+0.74%)`	⬆️
botocore/client.py	`97.67% <83.33%> (-0.18%)`	⬇️
botocore/useragent.py	`94.73% <94.73%> (ø)`
botocore/args.py	`100.00% <100.00%> (ø)`

... and 2 files with indirect coverage changes

☔ View full report in Codecov by Sentry.
📢 Do you have feedback about the report comment? Let us know in this issue.

nateprewitt · 2023-06-02T20:22:37Z

botocore/args.py

@@ -66,13 +69,14 @@ def __init__(
        loader,
        exceptions_factory,
        config_store,
+        user_agent_creator,


I think this needs to be optional or we may end up breaking current callers. That may change how we use the attribute below.

bda23a9 is a refactor that prioritizes backwards compatible interfaces over everything (especially over avoiding duplication).

nateprewitt · 2023-06-02T20:24:56Z

botocore/args.py

@@ -249,7 +254,6 @@ def compute_client_args(
        return {
            'service_name': service_name,
            'parameter_validation': parameter_validation,
-            'user_agent': user_agent,


This may also be problematic, I don't immediately know how we do this cleanly though.

We can bring back this line together with lines 196-203 above where its value gets computed using the legacy method. There is no harm in returning the user_agent key in the dict, it simply won't get used by botocore code. Do you think that's preferable? I was worried the confusion caused by the dead code existing and an unused value being returned might outweigh any benefit to any third-party that may be relying on the user_agent dict key being present.

Looking at other libraries that may have issues with this, I can't find one that relies on user_agent. I agree this may cause unnecessary confusion, so let's opt to remove it.

botocore/args.py

botocore/config.py

botocore/useragent.py

nateprewitt · 2023-06-02T21:29:20Z

tests/functional/test_useragent.py

+
+
+@contextmanager
+def uacap_client(client):


Does this give us a testing efficiency over the HTTPStubber?

Somehow I couldn't figure out how to use HTTPStubber for this. Not sure if my requirements changed or I just got smarter since then, but your comment made me look at it again and it was straightforward. Done in eb7ded6.

tests/functional/test_useragent.py

nateprewitt · 2023-06-02T21:35:48Z

tests/functional/test_useragent.py

+    assert 'app/1234-' in uafields
+
+
+def test_boto3_behavior(patched_session, stubbed_list_buckets):


nit; We may want to be more specific than behavior.

Suggested change

def test_boto3_behavior(patched_session, stubbed_list_buckets):

def test_boto3_user_agent(patched_session, stubbed_list_buckets):

Done in eb7ded6

jonathan343 · 2023-06-06T17:15:19Z

botocore/useragent.py

+        ua_client = ua_session.with_client_config(Config(...))
+        ua_string = ua_request.to_string()


There seems to be inconsistent naming here. Both should be ua_client or ua_request.

I don't disagree with the naming being somewhat confusing. But I don't think your suggestions would clarify things. In fact your suggestions might be a sign that I successfully confused you!

ua_session is a UserAgentString object with session-level information only.

ua_client is a UserAgentString object with session-level and client-level information.

ua_string is a str.

jonathan343 · 2023-06-06T17:18:50Z

tests/functional/test_useragent.py

+            myclient.operation()
+        print(cap_client.captured_ua_string)
+    """
+    client.captured_ua_string = None


Would there be any value to add or replace this with a list that captures sequential ua_string's?

I ended up replacing this entire fixture in eb7ded6 to address another comment above. To answer the question though: I didn't see a reason to do more than one request in a single test because each request's UA is independent of the others.

nateprewitt

2 minor comments, otherwise I think this looks ready to go. Thanks @jonemo!

nateprewitt · 2023-06-20T20:10:48Z

botocore/args.py

@@ -249,7 +254,6 @@ def compute_client_args(
        return {
            'service_name': service_name,
            'parameter_validation': parameter_validation,
-            'user_agent': user_agent,


Looking at other libraries that may have issues with this, I can't find one that relies on user_agent. I agree this may cause unnecessary confusion, so let's opt to remove it.

botocore/useragent.py

nateprewitt

This reverts commit e5d52ae.

* release-1.29.162: Bumping version to 1.29.162 Update to latest models Revert "User Agent 2.0 (#2955)" (#2976) User Agent 2.0 (#2955) remove leftover print() statement (#2975) Fallback for situations where Python's fromtimestamp() raises OSError or OverflowError (#2972)

jonemo added 3 commits May 25, 2023 17:50

user agent header string functional tests

e1a165f

User Agent header v2

f9d6a96

drop ft/ components of user agent, drop with_request_params()

aa03f04

jonemo requested a review from nateprewitt May 26, 2023 00:03

jonemo marked this pull request as draft May 26, 2023 00:13

jonemo added 2 commits May 26, 2023 08:06

handle backwards compatibility case of user_agent_builder=None

87dfca1

updates to existing tests

a04c941

jonemo force-pushed the user-agent branch from 2edf9dd to a04c941 Compare May 26, 2023 15:20

jonemo marked this pull request as ready for review May 27, 2023 02:20

nateprewitt requested changes Jun 2, 2023

View reviewed changes

jonathan343 reviewed Jun 6, 2023

View reviewed changes

jonemo added 6 commits June 6, 2023 15:15

assorted typos and docstring fixes

34edd4b

refactor for fully backwards compatible private interfaces

bda23a9

include CRT version and machine architecture in ua string

e19e237

refactor to not have non-private _get_crt_version

66ad86a

notes re private interfaces

f08429b

use ClientHTTP Stubber

eb7ded6

jonemo requested review from nateprewitt and jonathan343 June 20, 2023 15:40

nateprewitt reviewed Jun 26, 2023

View reviewed changes

fix: use tuple for constants

b0a29f8

nateprewitt approved these changes Jun 26, 2023

View reviewed changes

changelog

d33a106

jonemo force-pushed the user-agent branch from 10b11b6 to d33a106 Compare June 27, 2023 04:02

jonemo merged commit e5d52ae into boto:develop Jun 27, 2023
31 checks passed

jonemo added a commit to jonemo/botocore that referenced this pull request Jun 27, 2023

Revert "User Agent 2.0 (boto#2955)"

8bbda59

This reverts commit e5d52ae.

jonemo mentioned this pull request Jun 27, 2023

Revert "User Agent 2.0 (#2955)" #2976

Merged

jonemo added a commit that referenced this pull request Jun 27, 2023

Revert "User Agent 2.0 (#2955)" (#2976)

df1904e

This reverts commit e5d52ae.

jonemo mentioned this pull request Jun 28, 2023

User Agent 2.0, second attempt #2977

Merged

This was referenced Jul 4, 2023

Support for setting UserAgent Prefix through environment variable aws/aws-sdk-java-v2#4162

Open

Support for setting UserAgent Prefix through environment variable aws/aws-sdk-java#3000

Closed

jonemo mentioned this pull request Jul 12, 2023

request compression #2959

Merged

kapilt mentioned this pull request Jul 17, 2023

breaking - aws - custodian user agent in botocore has changed cloud-custodian/cloud-custodian#8739

Closed

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

User Agent 2.0 #2955

User Agent 2.0 #2955

jonemo commented May 26, 2023

codecov-commenter commented May 26, 2023 •

edited

Loading

nateprewitt Jun 2, 2023

jonemo Jun 20, 2023 •

edited

Loading

nateprewitt Jun 2, 2023

jonemo Jun 19, 2023

nateprewitt Jun 20, 2023

nateprewitt Jun 2, 2023

jonemo Jun 20, 2023

nateprewitt Jun 2, 2023

jonemo Jun 20, 2023

jonathan343 Jun 6, 2023

jonemo Jun 19, 2023

jonathan343 Jun 6, 2023

jonemo Jun 20, 2023

nateprewitt left a comment

nateprewitt Jun 20, 2023

nateprewitt left a comment

		assert 'app/1234-' in uafields


		def test_boto3_behavior(patched_session, stubbed_list_buckets):

	def test_boto3_behavior(patched_session, stubbed_list_buckets):
	def test_boto3_user_agent(patched_session, stubbed_list_buckets):

		ua_client = ua_session.with_client_config(Config(...))
		ua_string = ua_request.to_string()

User Agent 2.0 #2955

User Agent 2.0 #2955

Conversation

jonemo commented May 26, 2023

codecov-commenter commented May 26, 2023 • edited Loading

Codecov Report

Choose a reason for hiding this comment

jonemo Jun 20, 2023 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

nateprewitt left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

nateprewitt left a comment

Choose a reason for hiding this comment

codecov-commenter commented May 26, 2023 •

edited

Loading

jonemo Jun 20, 2023 •

edited

Loading