[Console] Move humanReadableTuples into console lib & add tests
#981
+1,850
−215
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Signed-off-by: Mikayla Thompson <thomika@amazon.com>
Signed-off-by: Mikayla Thompson <thomika@amazon.com>
Signed-off-by: Mikayla Thompson <thomika@amazon.com>
mikaylathompson
requested review from
AndreKurait,
chelma,
gregschohn,
lewijacn,
okhasawn,
peternied and
sumobrian
as code owners
Codecov ReportAttention: Patch coverage is
Additional details and impacted files@@ Coverage Diff @@
## main #981 +/- ##
============================================
+ Coverage 80.80% 80.90% +0.10%
Complexity 2725 2725
============================================
Files 400 402 +2
Lines 15408 15582 +174
Branches 971 971
============================================
+ Hits 12450 12607 +157
- Misses 2383 2400 +17
Partials 575 575
Flags with carried forward coverage won't be shown. Click here to find out more. ☔ View full report in Codecov by Sentry. |
gregschohn
approved these changes
Sep 19, 2024
TrafficCapture/README.md
Outdated
| @@ -110,27 +110,46 @@ The body of the messages is sometimes gzipped which makes it difficult to repres | |||
| and responses is base64 encoded before it is logged. This makes the files stable, but not human-readable. | |||
|
|
|||
| We have provided a utility script that can parse these files and output them to a human-readable format: the bodies are | |||
| base64 decoded, un-gzipped if applicable, and parsed as JSON if applicable. They're then saved back to JSON format on disk. | |||
| base64 decoded and parsed as JSON if applicable. They're then saved back to JSON format to stdout or disk. | |||
There was a problem hiding this comment.
nit: s/or disk/or file/
| @@ -0,0 +1,49 @@ | |||
| { | |||
There was a problem hiding this comment.
nit - how is this tuple invalid? Maybe rename the file to be slightly more specific.
| @@ -0,0 +1,4 @@ | |||
| {"sourceRequest": {"Request-URI": "/", "Method": "GET", "HTTP-Version": "HTTP/1.1", "Host": "capture-proxy:9200", "User-Agent": "python-requests/2.32.3", "Accept-Encoding": "gzip, deflate, zstd", "Accept": "*/*", "Connection": "keep-alive", "Authorization": "Basic YWRtaW46YWRtaW4=", "body": ""}, "sourceResponse": {"HTTP-Version": {"keepAliveDefault": true}, "Status-Code": 200, "Reason-Phrase": "OK", "response_time_ms": 374, "content-type": "application/json; charset=UTF-8", "content-length": "538", "body": {"name": "3cc068ad54eb", "cluster_name": "docker-cluster", "cluster_uuid": "PIAawDwSQKSRLtoaXC-oWg", "version": {"number": "7.10.2", "build_flavor": "oss", "build_type": "docker", "build_hash": "747e1cc71def077253878a59143c1f785afa92b9", "build_date": "2021-01-13T00:42:12.435326Z", "build_snapshot": false, "lucene_version": "8.7.0", "minimum_wire_compatibility_version": "6.8.0", "minimum_index_compatibility_version": "6.0.0-beta1"}, "tagline": "You Know, for Search"}}, "targetRequest": {"Request-URI": "/", "Method": "GET", "HTTP-Version": "HTTP/1.1", "Host": "opensearchtarget", "User-Agent": "python-requests/2.32.3", "Accept-Encoding": "gzip, deflate, zstd", "Accept": "*/*", "Connection": "keep-alive", "Authorization": "Basic YWRtaW46bXlTdHJvbmdQYXNzd29yZDEyMyE=", "body": ""}, "targetResponses": [{"HTTP-Version": {"keepAliveDefault": true}, "Status-Code": 200, "Reason-Phrase": "OK", "response_time_ms": 341, "content-type": "application/json; charset=UTF-8", "content-length": "568", "body": {"name": "e6b3a4370763", "cluster_name": "docker-cluster", "cluster_uuid": "88zlik7ORFuFT78zduradA", "version": {"distribution": "opensearch", "number": "2.15.0", "build_type": "tar", "build_hash": "61dbcd0795c9bfe9b81e5762175414bc38bbcadf", "build_date": "2024-06-20T03:27:32.562036890Z", "build_snapshot": false, "lucene_version": "9.10.0", "minimum_wire_compatibility_version": "7.10.0", "minimum_index_compatibility_version": "7.0.0"}, "tagline": "The OpenSearch Project: https://opensearch.org/"}}], "connectionId": "0242acfffe12000b-0000000a-00000003-4c7bb8f4aefa3189-c0544dbf.0", "numRequests": 1, "numErrors": 0} | |||
There was a problem hiding this comment.
nit: consider pretty-formatting all of these, maybe one per file and put them into a directory structure. Then you can cons them together with the python equivalent of (cat foo/*).
It's tough to figure out what the errors are in each one. Being able to name each will let the developer know what the faults or examples are for too.
Signed-off-by: Mikayla Thompson <thomika@amazon.com>
Signed-off-by: Mikayla Thompson <thomika@amazon.com>
Signed-off-by: Mikayla Thompson <thomika@amazon.com>
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Description
I scrapped the old
humanReadableTuples.pyfile, added the logic to the console library, removed the unnecessary parts (the replayer now handles un-chunking & un-gzipping), and added tests.Follow-up task
(will create an issue for it)
Add a tuple conversion/check step to the E2E tests
Issues Resolved
Should fix roughly 150 lines of uncovered code, plus some bugs. I suspect there's a JIRA about this as well. (Edit: https://opensearch.atlassian.net/browse/MIGRATIONS-1325)
There were also two or three Sonarqube "code smells" associated with the former file that are resolved.
Testing
Unit tests and a integration test at the CLI command level that converts a real (small) tuple file generated by the replayer.
Check List
By submitting this pull request, I confirm that my contribution is made under the terms of the Apache 2.0 license.
For more information on following Developer Certificate of Origin and signing off your commits, please check here.