[BUG] Opensearch resposne not respecting client HTTP Headers (Transfer-Encoding: Chunked instead of Content-Type:gzip)

[NelsonBurton]

What is the bug?

I’m using the Rust client , and found the Opensearch response does not seem to be respecting HTTP Headers during content negotation, specifically when I ask for a gzip response to be sent, via, I instead get backed a Transfer-Encoding: chunked , which costs an extra ~20ms for 1.2Mb response, because it downloads chunks back, 8kb at a time.

For example, my request will include:

Accept-Encoding: gzip
Accept: */*

But my response Headers I get back are:

"content-type": "application/json; charset=UTF-8"
"transfer-encoding": "chunked"

One way to avoid the chunking, that I found, is to add these headers

Accept-Encoding:  "*"
Accept: "application/json, */*; q=0.01"

But this gets back a full JSON payload, and Opensearch hasn't compressed my result.

How can one reproduce the bug?

I have code to reproduce using the reqwest library, that I could share, as it prints Headers, but couldn't figure out how to easily print headers with the Opensearch-rs wrapper.

What is the expected behavior?

I should get back a Gzip payload, with content-encoding: gzip, content-type: application/json

Do you have any screenshots?

Here's my debug logs

live request headers: "content-type": "application/json"
live request headers: "accept-encoding": "gzip"
live request headers: "accept": "application/json; charset=UTF-8"
live request headers: "x-amz-date": "20231213T234744Z"
live request headers: "authorization": "AWS4-HMAC-SHA256 Credential=ASIAQLJP7JEYT5LBNX5S/20231213/us-east-1/es/aws4_request, SignedHeaders=accept;accept-encoding;content-type;host;x-amz-content-sha256;x-amz-date;x-amz-security-token, Signature=d39e048bc7db317940336a71f1d896b40c4b09087139d6b37de4c0655788f907"
live request headers: "x-amz-content-sha256": "752762f7389c59d43283964584fc2c9d869b955ea65c2b18ca960107a336e7bf"
live request headers: "x-amz-security-token": Sensitive

live response headers: "date": "Wed, 13 Dec 2023 23:47:44 GMT"
live response headers: "content-type": "application/json; charset=UTF-8"
live response headers: "transfer-encoding": "chunked"
live response headers: "connection": "keep-alive"
live response headers: "access-control-allow-origin": "*"
live response headers: "vary": "Accept-Encoding, User-Agent"

Do you have any additional context?

I tried all the Respone headers in this list:

        let accept_encodings = vec!("*", "gzip", "compress", "deflate", "br", "identity");
        let accept = vec!("*/*",  "application/*", "text/*", "application/json", "application/gzip", "application/x-gzip", "application/javascript");

But no combination worked.

[Xtansia]

Hi @NelsonBurton,

This is a behaviour of reqwest, if you refer to the documentation for the gzip option, it states that it removes the content-encoding header after decompressing the response: https://docs.rs/reqwest/latest/reqwest/struct.ClientBuilder.html#method.gzip

So I believe it is actually using gzip compression.