bug: environment variable values mixed in proxy-rewrite plugin

[jompu]

Current Behavior

The proxy-rewrite plugin uses value from wrong environment variable, when configured with standalone configuration.

The problem is that some times the values at step 2 are like this (which is wrong):
X-Some-String-Value: "astringvalue"
X-Some-String-Value-But-Different: "Different astringvalue"

Expected Behavior

Header values should be like this:
X-Some-String-Value: "astringvalue"
X-Some-String-Value-But-Different: "Different astringvaluebutdifferent"

Error Logs

No response

Steps to Reproduce

config.yaml

apisix:
  node_listen: 9999
  enable_ipv6: false
  enable_admin: false
  ssl:
    enable: false

deployment:
  role: data_plane
  role_data_plane:
    config_provider: yaml

apisix.yaml

upstreams:
  - id: echoserver
    pass_host: rewrite
    upstream_host: echo.free.beeceptor.com
    scheme: https
    nodes:
      - host: echo.free.beeceptor.com
        port: 443
        weight: 1
plugin_configs:
  - id: validaterequest
    plugins:
      request-validation:
        header_schema:
          type: object
          required: 
            - X-random-request-header
          properties:
            X-random-request-header:
              type: string
              pattern: '^${{SOME_STRING_VALUE}}$'
routes:
  - name: test
    uri: /test
    plugin_config_id: validaterequest
    plugins:
      proxy-rewrite:
        uri: /some/api/endpoint
        headers:
          set:
            X-Some-String-Value-But-Different: "Different ${{SOME_STRING_VALUE_BUT_DIFFERENT}}"
            X-Some-String-Value: ${{SOME_STRING_VALUE}}
    upstream_id: echoserver
#END

The issue does not happen always, anyway try below steps multiple times and it happens at some point. Quite often it happens at the first time though. The apisix.yaml is just an example that I have used to reproduce the problem.

1. Run container

docker run --env SOME_STRING_VALUE_BUT_DIFFERENT=astringvaluebutdifferent \
  --env SOME_STRING_VALUE=astringvalue \
  -v $(pwd)/config.yaml:/usr/local/apisix/conf/config.yaml:ro \
  -v $(pwd)/apisix.yaml:/usr/local/apisix/conf/apisix.yaml:ro \
  -p 9999:9999/tcp apache/apisix:3.9.0-debian

2. Run curl: curl -H "X-random-request-header: astringvalue" http://localhost:9999/test

• You should receive a json that shows headers set by the proxy-rewrite plugin.
• Header values should be like this:
  X-Some-String-Value: "astringvalue"
  X-Some-String-Value-But-Different: "Different astringvaluebutdifferent"
  • if this is the case, stop the container and go back to step 1

The problem is that some times the values at step 2 are like this (which is wrong):
X-Some-String-Value: "astringvalue"
X-Some-String-Value-But-Different: "Different astringvalue"

Restarting the pod or just reloading the configuration (e.g. by editing apisix.yaml file) will fix the problem.

A script to run steps repeatedly and stop when the issue happens:

#!/bin/bash

SOME_STRING_VALUE_BUT_DIFFERENT=astringvaluebutdifferent
SOME_STRING_VALUE=astringvalue

start_container () {
  docker run -d --env SOME_STRING_VALUE_BUT_DIFFERENT=$SOME_STRING_VALUE_BUT_DIFFERENT \
    --env SOME_STRING_VALUE=$SOME_STRING_VALUE \
    -v $(pwd)/config.yaml:/usr/local/apisix/conf/config.yaml:ro \
    -v $(pwd)/apisix.yaml:/usr/local/apisix/conf/apisix.yaml:ro \
    -p 9999:9999/tcp apache/apisix:3.9.0-debian
}

run_curl () {
  curl -s -H "X-random-request-header: astringvalue" http://localhost:9999/test
}

run_curl_get_header () {
  run_curl | jq -r ".headers[\"$1\"]"
}

counter=1
while :; do
  container_id=$(start_container)
  sleep 5

  butdifferentvalue=$(run_curl_get_header "X-Some-String-Value-But-Different")

  if [ "$butdifferentvalue" != "Different $SOME_STRING_VALUE_BUT_DIFFERENT" ]; then
    echo
    echo "X-Some-String-Value-But-Different header was set with value 'Different $SOME_STRING_VALUE', when it should be 'Different $SOME_STRING_VALUE_BUT_DIFFERENT'."
    echo "We had to repeate the test $counter times."
    echo; echo
    echo "Here is the wrong echoserver response:"
    echo
    run_curl
    
    docker rm -f $container_id > /dev/null
    break
  fi

  docker rm -f $container_id > /dev/null

  echo "Test succeeded, let's try again!"
  sleep 1
  counter=$((counter+1))
done

Environment

• Docker image: apache/apisix:3.9.0-debian

[hanqingwu]

Yes, Maybe I can reproduce, But I don't know why .
I will try to find out .

[hanqingwu]

when wrong case:
2024/04/06 08:21:30 [info] 43#43: *7 [lua] config_yaml.lua:136: routes items: [{"upstream_id":"echoserver","name":"test","uri":"/test","plugins":{"proxy-rewrite":{"uri":"/some/api/endpoint","headers":{"set":{"X-Some-String-Value":"\"astringvalue\"","X-Some-String-Value-But-Different":"Different \"astringvalue\""}}}}}], context: init_worker_by_lua*

when right case:
2024/04/06 08:24:45 [info] 34#34: *5 [lua] config_yaml.lua:136: routes items: [{"uri":"/test","upstream_id":"echoserver","plugins":{"proxy-rewrite":{"uri":"/some/api/endpoint","headers":{"set":{"X-Some-String-Value-But-Different":"Different \"astringvaluebutdifferent\"","X-Some-String-Value":"\"astringvalue\""}}}},"name":"test"}], context: init_worker_by_lua*

some wrong in read config .

[hanqingwu]

I have trace
file.lua

        local v = getenv(var) or default

        print("env key ", var, " env value ", v )

wrong case:
2024/04/06 09:21:06 [notice] 1#1: [lua] file.lua:73: env key SOME_STRING_VALUE_BUT_DIFFERENT env value "astringvalue"
right case:
2024/04/06 09:23:04 [notice] 1#1: [lua] file.lua:73: env key SOME_STRING_VALUE_BUT_DIFFERENT env value "astringvaluebutdifferent"

some times, os.getenv return wrong value ....

[jompu]

It works fine when I prefix them with some different random strings, like: ZCBSFN_SOME_STRING_VALUE and UITUFK_SOME_STRING_VALUE_BUT_DIFFERENT.

So it seems to be related to that the variable names start with same string.

But still a short variable names like VALUE_A and VALUE_B works.

[jompu]

Actually it seems to be matter of just if the prefix or suffix of the variable name is different it works. But if other variable name is just a subset of the other, then it fails.

[zhoujiexiong]

@jompu

I tested with the case below on v3.2 with no problem.
And I will test on v3.9.0/v.3.8.0 later.

use t::APISIX 'no_plan';

repeat_each(100);

$ENV{SOME_STRING_VALUE_BUT_DIFFERENT} = 'astringvaluebutdifferent';
$ENV{SOME_STRING_VALUE} = 'astringvalue';

our $yaml_config = <<_EOC_;
apisix:
  node_listen: 1984
deployment:
  role: data_plane
  role_data_plane:
    config_provider: yaml
_EOC_

run_tests();

__DATA__

=== TEST 1: apisix_issue_11121
--- main_config
env SOME_STRING_VALUE_BUT_DIFFERENT;
env SOME_STRING_VALUE;
--- yaml_config eval: $::yaml_config
--- apisix_yaml
upstreams:
  - id: 1
    nodes:
      - host: 127.0.0.1
        port: 1980
        weight: 1
routes:
  - uri: /hello
    upstream_id: 1
    plugins:
      response-rewrite:
        headers:
          set:
            X-Some-String-Value-But-Different: "Different ${{SOME_STRING_VALUE_BUT_DIFFERENT}}"
            X-Some-String-Value: ${{SOME_STRING_VALUE}}

#END
--- request
GET /hello
--- response_headers
X-Some-String-Value-But-Different: Different astringvaluebutdifferent
X-Some-String-Value: astringvalue

[zhoujiexiong]

@jompu
Run the test case on v3.8.0 & v3.9.0 are also OK.
May be I should try your way. :D

apache/apisix:3.8.0-debian
apache/apisix:3.9.0-debian

# apisix version
/usr/local/openresty//luajit/bin/luajit ./apisix/cli/apisix.lua version
3.9.0

# prove -I. t/plugin/apisix_issue_11121.t
t/plugin/apisix_issue_11121.t .. nginx: [warn] [lua] config_yaml.lua:117: read_apisix_yaml(): config file /usr/local/apisix/t/servroot/conf/apisix.yaml reloaded.
t/plugin/apisix_issue_11121.t .. ok     
All tests successful.
Files=1, Tests=400, 16 wallclock secs ( 0.15 usr  0.01 sys +  0.71 cusr  0.28 csys =  1.15 CPU)
Result: PASS

# apisix version
/usr/local/openresty//luajit/bin/luajit ./apisix/cli/apisix.lua version
3.8.0

# prove -I. t/plugin/apisix_issue_11121.t 
t/plugin/apisix_issue_11121.t .. nginx: [warn] [lua] config_yaml.lua:118: read_apisix_yaml(): config file /usr/local/apisix/t/servroot/conf/apisix.yaml reloaded.
t/plugin/apisix_issue_11121.t .. ok     
All tests successful.
Files=1, Tests=400, 16 wallclock secs ( 0.16 usr  0.00 sys +  0.70 cusr  0.31 csys =  1.17 CPU)
Result: PASS