fix(hmac-auth): hmac-auth plugin sort array param

[kevinw66]

What this PR does / why we need it:

fixes #6313

Pre-submission checklist:

• Did you explain what problem does this PR solve? Or what new features have been added?
• Have you added corresponding test cases?
• Have you modified the corresponding document?
• Is this PR backward compatible? If it is not backward compatible, please discuss on the mailing list first

[tylitianrui]

i wonder that why not use table.new

[kevinw66]

Just keep the same code style like table keys

[tzssangglass]

Not necessary to keep old code style here. We can specify the initialization size when we create a new table by using core.table.new(narray, nhash)

[starsz]

Hi, @kevinw66.Can you add a test case to cover it.

[bisakhmondal]

Hi! Thanks for detecting the issue and the PR.
After looking into the codebase, I find the way of forming the canonical query string unnecessarily complex yet it's not accurate.
I would go with a simpler approach rather than pathing this up.

We can replace:

apisix/apisix/plugins/hmac-auth.lua

Lines 219 to 255 in 605141c

	if type(args) == "table" then
	local keys = {}
	local query_tab = {}
	for k, v in pairs(args) do
	core.table.insert(keys, k)
	end
	core.table.sort(keys)
	local field_val = get_conf_field(params.access_key, "encode_uri_params")
	core.log.info("encode_uri_params: ", field_val)
	local encode_or_not = do_nothing
	if field_val then
	encode_or_not = escape_uri
	end
	for _, key in pairs(keys) do
	local param = args[key]
	-- when args without `=<value>`, value is treated as true.
	-- In order to be compatible with args lacking `=<value>`,
	-- we need to replace true with an empty string.
	if type(param) == "boolean" then
	param = ""
	end
	-- whether to encode the uri parameters
	if type(param) == "table" then
	for _, val in pairs(param) do
	core.table.insert(query_tab, encode_or_not(key) .. "=" .. encode_or_not(val))
	end
	else
	core.table.insert(query_tab, encode_or_not(key) .. "=" .. encode_or_not(param))
	end
	end
	canonical_query_string = core.table.concat(query_tab, "&")
	end

with

apisix/apisix/plugins/aws-lambda.lua

Lines 123 to 130 in 605141c

	-- computing canonical query string
	local canonical_qs = {}
	for k, v in pairs(params.query) do
	canonical_qs[#canonical_qs+1] = ngx.unescape_uri(k) .. "=" .. ngx.unescape_uri(v)
	end
	tab_sort(canonical_qs)
	canonical_qs = tab_concat(canonical_qs, "&")

It's very simple, yet effective.
cc @spacewander to share your views.

[kevinw66]

Hi, @kevinw66.Can you add a test case to cover it.

Can you teach me how to run unit tests in my local machine ? I left a comment on issue.
Thanks.

[kevinw66]

Hi! Thanks for detecting the issue and the PR. After looking into the codebase, I find the way of forming the canonical query string unnecessarily complex yet it's not accurate. I would go with a simpler approach rather than pathing this up.

We can replace:

apisix/apisix/plugins/hmac-auth.lua

Lines 219 to 255 in 605141c

	if type(args) == "table" then
	local keys = {}
	local query_tab = {}
	for k, v in pairs(args) do
	core.table.insert(keys, k)
	end
	core.table.sort(keys)
	local field_val = get_conf_field(params.access_key, "encode_uri_params")
	core.log.info("encode_uri_params: ", field_val)
	local encode_or_not = do_nothing
	if field_val then
	encode_or_not = escape_uri
	end
	for _, key in pairs(keys) do
	local param = args[key]
	-- when args without `=<value>`, value is treated as true.
	-- In order to be compatible with args lacking `=<value>`,
	-- we need to replace true with an empty string.
	if type(param) == "boolean" then
	param = ""
	end
	-- whether to encode the uri parameters
	if type(param) == "table" then
	for _, val in pairs(param) do
	core.table.insert(query_tab, encode_or_not(key) .. "=" .. encode_or_not(val))
	end
	else
	core.table.insert(query_tab, encode_or_not(key) .. "=" .. encode_or_not(param))
	end
	end
	canonical_query_string = core.table.concat(query_tab, "&")
	end

with

apisix/apisix/plugins/aws-lambda.lua

Lines 123 to 130 in 605141c

	-- computing canonical query string
	local canonical_qs = {}
	for k, v in pairs(params.query) do
	canonical_qs[#canonical_qs+1] = ngx.unescape_uri(k) .. "=" .. ngx.unescape_uri(v)
	end
	tab_sort(canonical_qs)
	canonical_qs = tab_concat(canonical_qs, "&")

It's very simple, yet effective.
cc @spacewander to share your views.

This may still need to check table param

for k, v in pairs(args) do
    if type(v) == "table" then
        for _, sv in pairs(v) do
            canonical_qs[#canonical_qs+1] = ngx.unescape_uri(k) .. "=" .. ngx.unescape_uri(sv)
        end
    else
        canonical_qs[#canonical_qs+1] = ngx.unescape_uri(k) .. "=" .. ngx.unescape_uri(v)
    end
end

[kevinw66]

Hi! Thanks for detecting the issue and the PR. After looking into the codebase, I find the way of forming the canonical query string unnecessarily complex yet it's not accurate. I would go with a simpler approach rather than pathing this up.

We can replace:

apisix/apisix/plugins/hmac-auth.lua

Lines 219 to 255 in 605141c

	if type(args) == "table" then
	local keys = {}
	local query_tab = {}
	for k, v in pairs(args) do
	core.table.insert(keys, k)
	end
	core.table.sort(keys)
	local field_val = get_conf_field(params.access_key, "encode_uri_params")
	core.log.info("encode_uri_params: ", field_val)
	local encode_or_not = do_nothing
	if field_val then
	encode_or_not = escape_uri
	end
	for _, key in pairs(keys) do
	local param = args[key]
	-- when args without `=<value>`, value is treated as true.
	-- In order to be compatible with args lacking `=<value>`,
	-- we need to replace true with an empty string.
	if type(param) == "boolean" then
	param = ""
	end
	-- whether to encode the uri parameters
	if type(param) == "table" then
	for _, val in pairs(param) do
	core.table.insert(query_tab, encode_or_not(key) .. "=" .. encode_or_not(val))
	end
	else
	core.table.insert(query_tab, encode_or_not(key) .. "=" .. encode_or_not(param))
	end
	end
	canonical_query_string = core.table.concat(query_tab, "&")
	end

with

apisix/apisix/plugins/aws-lambda.lua

Lines 123 to 130 in 605141c

	-- computing canonical query string
	local canonical_qs = {}
	for k, v in pairs(params.query) do
	canonical_qs[#canonical_qs+1] = ngx.unescape_uri(k) .. "=" .. ngx.unescape_uri(v)
	end
	tab_sort(canonical_qs)
	canonical_qs = tab_concat(canonical_qs, "&")

It's very simple, yet effective.
cc @spacewander to share your views.

This may not work correctly in some cases, in ASCII table, the '=' symbol is behind number sequence, so if I pass param like a1a=2&a1a=1&a=2, it will sort as a1a=1&a1a=2&a=2, but I think the result a=2&a1a=1&a1a=2 may correct.

[spacewander]

Hi! Thanks for detecting the issue and the PR. After looking into the codebase, I find the way of forming the canonical query string unnecessarily complex yet it's not accurate. I would go with a simpler approach rather than pathing this up.

We can replace:

apisix/apisix/plugins/hmac-auth.lua

Lines 219 to 255 in 605141c

	if type(args) == "table" then
	local keys = {}
	local query_tab = {}
	for k, v in pairs(args) do
	core.table.insert(keys, k)
	end
	core.table.sort(keys)
	local field_val = get_conf_field(params.access_key, "encode_uri_params")
	core.log.info("encode_uri_params: ", field_val)
	local encode_or_not = do_nothing
	if field_val then
	encode_or_not = escape_uri
	end
	for _, key in pairs(keys) do
	local param = args[key]
	-- when args without `=<value>`, value is treated as true.
	-- In order to be compatible with args lacking `=<value>`,
	-- we need to replace true with an empty string.
	if type(param) == "boolean" then
	param = ""
	end
	-- whether to encode the uri parameters
	if type(param) == "table" then
	for _, val in pairs(param) do
	core.table.insert(query_tab, encode_or_not(key) .. "=" .. encode_or_not(val))
	end
	else
	core.table.insert(query_tab, encode_or_not(key) .. "=" .. encode_or_not(param))
	end
	end
	canonical_query_string = core.table.concat(query_tab, "&")
	end

with

apisix/apisix/plugins/aws-lambda.lua

Lines 123 to 130 in 605141c

	-- computing canonical query string
	local canonical_qs = {}
	for k, v in pairs(params.query) do
	canonical_qs[#canonical_qs+1] = ngx.unescape_uri(k) .. "=" .. ngx.unescape_uri(v)
	end
	tab_sort(canonical_qs)
	canonical_qs = tab_concat(canonical_qs, "&")

It's very simple, yet effective.
cc @spacewander to share your views.

Yes, the way in aws-lambda still needs to handle multiple values in the same query key.

[spacewander]

Remember to convert boolean to '' before sorting. table.sort only supports sorting in the same type.

[kevinw66]

done

[bisakhmondal]

This may still need to check table param

for k, v in pairs(args) do
    if type(v) == "table" then
        for _, sv in pairs(v) do
            canonical_qs[#canonical_qs+1] = ngx.unescape_uri(k) .. "=" .. ngx.unescape_uri(sv)
        end
    else
        canonical_qs[#canonical_qs+1] = ngx.unescape_uri(k) .. "=" .. ngx.unescape_uri(v)
    end
end

Ohh, Thanks! I missed the scenario : )
Btw, in that case, something still looks odd! We are considering one extra level nesting in query params. Query params with nesting level > 2 would fail.

For example: (random example, taken from web)

main_key[other_key][main_array][0][name]=Bob&main_key[other_key][main_array][0][nickname]=bobby&main_key[other_key][main_array][1][name]=Tom&main_key[other_key][main_array][1][nickname]=Tommy

{
    "main_key": {
        "other_key": {
            "main_array": [{
                    "name": "Bob",
                    "nickname": "bobby"
                },
                {
                    "name": "Tom",
                    "nickname": "Tommy"
                }
            ]
        }
    }
}

I think we need to perform a recursive DFS! Else it will introduce some sort of indeterminism

🔥➜ ~ resty -e 'print(ngx.escape_uri({["a"]="b"}))' 

table%3A%200x7f5aa8abc608
🔥➜ ~ resty -e 'print(ngx.escape_uri({["a"]="b"}))'

table%3A%200x7f7eca005608

[spacewander]

AFAIK, the ngx.req.get_uri_args only returns two level table at most.

Multiple occurrences of an argument key will result in a table value holding all the values for that key in order.

https://github.com/openresty/lua-nginx-module#ngxreqget_uri_args

Here is the source: https://github.com/openresty/lua-resty-core/blob/97d2e5e08d9ddf328e33d2b5f1710a9ce3d8bbd0/lib/resty/core/request.lua#L237