Using template processor corrupts response HTML if it includes %

[rorosaurus]

I got this library successfully serving index.htm over SPIFFS, using the example provided in the readme: request->send(SPIFFS, "/www/index.htm", "text/html");

Then I tried to use the template processor to alter some variables inside the HTML: request->send(SPIFFS, "/www/index.htm", String(), false, processor);

With the processor from the readme:

String processor(const String& var)
{
  if(var == "HELLO_FROM_TEMPLATE")
    return F("Hello world!");
  return String();
}

This correctly replaces the %HELLO_FROM_TEMPLATE% with Hello world!, but it appears to corrupt the output HTML as well. For example, the submit button is now missing, as well as some CSS.

I suspect this is related to some of the inline styles I wrote, which use %, such as: style='width: 100%;height:80%'

However, I wouldn't expect this behavior, since the template processor should just pass through any string that doesn't match, right?

... As I write that, it becomes obvious to me that the example provided in the readme returns an empty String if var doesn't match the template string. So anything "surrounded" with % signs gets gobbled up.

Changing this to return var; recovers some of the expected HTML, but not everything. Since the %'s on both sides get stripped, the var variable only contains what was between them. Trying return "%" + var + "%"; actually seems to reset the ESP32 (triggering a RTOS watchdog).

Is there an easier way for me to ensure unexpected parts of my HTML are not altered? Maybe we can clarify that in the README?

Example code

[rorosaurus]

Just confirming: if we return "%" + var + "%";, we get stuck in an infinite loop where the template processor keeps trying to process the same thing over and over. It seems very determined to remove any % signs from the HTML.

Here's what it is trying to process: %; height: 100%

I can work around this by moving to an external stylesheet, so this is not a blocking issue. :)

[cjcr]

I can confirm this issue.

The workaround you can do is put the CSS code in other file and not use the processor. In this way it works... but isn't the best solution. We hope he can fix it.

[rorosaurus]

I can confirm the workaround. :) I was hoping to keep everything inline for efficiency (and also since it's for a captive portal, it's a little annoying to route exceptions).

[me21]

Does escaping percent signs like %% work for you?
Or, you can provide your own value for template delimiter characters by defining TEMPLATE_PLACEHOLDER preprocessor symbol to, for example, '$' (use dollar sign instead of percent).

[rorosaurus]

That is a useful reference, thank you! Would that be worth adding to the readme? Or did I miss it?

[me21]

Yes, it probably should be added to readme, thank you

[meditant]

Hello,

I have the same issue : ...min-width: 100%;">%CFGFRM%... and wrong response !

the response with std TEMPLATE_PLACEHOLDER at % :
…style="padding-right: 0;padding-left: 0;min-width: 100CFGFRM%
…

and the response with std TEMPLATE_PLACEHOLDER at % but with in html ...min-width: 100%;">%%CFGFRM%%... :
...style="padding-right: 0;padding-left: 0;min-width: 100<form class="text-left d-inline-flex...

and the good response with TEMPLATE_PLACEHOLDER at $ :
…style="padding-right: 0;padding-left: 0;min-width: 100%;"><form class="text-left d-inline-flex…

Best,
Franck

[Pablo2048]

Hi,
correct syntax is min-width: 100%%;">%CFGFRM% IMO because escaped % chracter is after 100 number...

[me21]

Yes, escaping % sign should work too. Let me know if it doesn't.

[meditant]

Hello,

Thanks for your response !

I try to change to min-width: 100%%;">%CFGFRM% but it's not work too, i have a strange return :
<div class="col-xl-6 d-inline-flex d-xl-flex align-self-center flex-wrap justify-content-lg-center align-items-lg-start justify-content-xl-center align-items-xl-start" style="padding-right: 0;padding-left: 0;min-width: 100%;"><form class="text-left d-inline-flex flex-nowrap" action="maj" method="post" target="_self" style="width: 100;padding: 5px;margin: 0;"><input class="form-control" type="hidden" name="ID" value="1"><input class="form-control" type="text" name="PROC" value="Kallitype" required="" minlength="1" maxlength="20" style="width: 100;margin-right: 10px;"><button class="btn btn-primary" type="submit" name="enregistrer" value="Enregistrer" style="margin-right: 10px;width: 20;" onclick="return confirm('Êtes-vous sûr de vouloir supprimer ce procédé ?')">Supprimer`

and the form is partial

And the same code with TEMPLATE_PLACEHOLDER at $ :

<div class="col-xl-6 d-inline-flex d-xl-flex align-self-center flex-wrap justify-content-lg-center align-items-lg-start justify-content-xl-center align-items-xl-start" style="padding-right: 0;padding-left: 0;min-width: 100%;"><form class="text-left d-inline-flex flex-nowrap" action="maj" method="post" target="_self" style="width: 100%;min-width: 50%;padding: 5px;margin: 0;"><input class="form-control" type="hidden" name="ID" value="1"><input class="form-control" type="text" name="PROC" value="Kallitype" required="" minlength="1" maxlength="20" style="width: 100%;margin-right: 10px;"><input class="form-control d-xl-flex justify-content-xl-start" type="number" name="UUV" value="250" min="1" max="9999" step="1" required="" style="width: 25%;margin-right: 10px;"><button class="btn btn-primary" type="submit" name="enregistrer" value="Enregistrer" style="margin-right: 10px;width: 20%;" onclick="return confirm('Êtes-vous sûr de vouloir enregistrer ?')">Enregistrer</button><button class="btn btn-danger" type="submit" name="supprimer" value="Supprimer" style="background-color: rgb(255,15,0);width: 20%;" onclick="return confirm('Êtes-vous sûr de vouloir supprimer ce procédé ?')">Supprimer</button></form></div>

I try to change 100% to 99%, but same way

Best,
Franck

[Pablo2048]

Hi Franck,
can you please give complete MCVE so we can test it out? I was using %% escaping in my templated HTML generator, and I don't remember any problem...

[meditant]

Hi Pablo,

Thanks for you interest.

A link to download a zip with all the elements https://www.franck-rondot.com/telechargement/WebServerTest_MCVE.zip

Best,
Franck

[stale]

[STALE_SET] This issue has been automatically marked as stale because it has not had recent activity. It will be closed in 14 days if no further activity occurs. Thank you for your contributions.

[meditant]

I think this issues need to still open it's not solved !

[stale]

[STALE_CLR] This issue has been removed from the stale queue. Please ensure activity to keep it openin the future.

[MrKalach]

Ok, the issue is still there, in my case I have next behavior:
next css:

        table.graphs {
            table-layout: auto;
            width: 100%;
        }
        canvas{
            border: 1px solid black;
            display: inline-block;
        }
        table.io {
            font-family: arial, sans-serif;
            border-collapse: collapse;
            width: 50%;
        }
        td.io, th.io {
            font-size: 1.5rem;
            border: 1px solid #000000;
            text-align: center;
            padding: 8px;
        }

becomes to

        table.graphs {
            table-layout: auto;
            width: 100;
        }
        td.io, th.io {
            font-size: 1.5rem;
            border: 1px solid #000000;
            text-align: center;
            padding: 8px;
        }

So, the next part was removed:

        %;
        }
        canvas{
            border: 1px solid black;
            display: inline-block;
        }
        table.io {
            font-family: arial, sans-serif;
            border-collapse: collapse;
            width: 50%

Possible workaround: return original str in template processor by default.

[MrKalach]

I have no rights to create a PR with the fix, so somebody with rights pls do:

diff --git a/src/WebResponses.cpp b/src/WebResponses.cpp
index a22e991..f1d35a8 100644
--- a/src/WebResponses.cpp
+++ b/src/WebResponses.cpp
@@ -389,7 +389,16 @@ size_t AsyncAbstractResponse::_fillBufferAndProcessTemplates(uint8_t* data, size
   // Search for template placeholders
   uint8_t* pTemplateStart = data;
   while((pTemplateStart < &data[len]) && (pTemplateStart = (uint8_t*)memchr(pTemplateStart, TEMPLATE_PLACEHOLDER, &data[len - 1] - pTemplateStart + 1))) { // data[0] ... data[len - 1]
-    uint8_t* pTemplateEnd = (pTemplateStart < &data[len - 1]) ? (uint8_t*)memchr(pTemplateStart + 1, TEMPLATE_PLACEHOLDER, &data[len - 1] - pTemplateStart) : nullptr;
+    uint8_t* pTemplateEnd = nullptr;
+    // A template can only be a one-liner
+    for (uint8_t *chr_ptr = pTemplateStart + 1; chr_ptr < &data[len]; ++chr_ptr) {
+      if (*chr_ptr == '\n') {
+        break;
+      } else if (*chr_ptr == TEMPLATE_PLACEHOLDER) {
+        pTemplateEnd = chr_ptr;
+        break;
+      }
+    }
     // temporary buffer to hold parameter name
     uint8_t buf[TEMPLATE_PARAM_NAME_LENGTH + 1];
     String paramName;

cc @me-no-dev

[MrKalach]

@Pablo2048 maybe you have rights? ^^^

[Pablo2048]

Unfortunately no - we have to wait for @me-no-dev

[MrKalach]

@me-no-dev proposal fix #737

[ilorevilo]

the proposal fixes the issue for me for single occurences of % in one line. However, for e.g. margin: 2% 5%; I need to escape each % to prevent its replacement.

[MrKalach]

@ilorevilo the proper fix should be something next: Define acceptable symbols for template name, for example [a-zA-Z_0-9], then it will be working like a charm. But then I think it will break the backward compatibility with old code.

[cems2]

I'm also having a probelm with Processor. using it with Arduino.cc compiler on ESP8266 LinLo noneMCU board. (As a check I Just downloaded a new copy of the ESPAsyncWebserver library today May 8 2020, but same problems. )

The issue manifests differently than the above and doesn't seem related to stray % signs, so it may not be the same and point to a larger problem

what is happening: I have three variable Fields (%FIELD%) which are the sole text inside each of their %FIELD1% tags. Into two of them processor() inserts an HTML table and and into one of them a simple string of plain text.
When the web page opens intially I see is that all three fields render correctly.

But... then I do an assyncronous XHTTP request update of all three Span elements by their ID tags the simple text field renders fine but the other two tables get additional copies of the other fields table in them! So if Span id= "foo1" is supposed to be html "Table 1" and Span3 is supposed to be Table3. What I see is Span1 is table1 plus also table 3 and span3 is table3 plus also table 1.

The tables themselves do not contain any percent signs. (there are css modifiers that might contain percent signs defined elsewhere but nothing obvious.)

I can attach images if my description of the Spans getting mixed like that isn't clear. But to repeat, after the initial page load that applies the proccessing() step the page looks correct. But after the first XHTTP Request rerites the span context the spans are incorrect. the XHTTP doesn't invoke Processor(), that only happens on the first page load

However, if I turn off the processing() then I see the %FIELD% text in the initial page load but then after the XML Request update the correct information appears and doesn't get duplicated.

As a workaround I have turned off the processing() and now just added javascript which runs once at page load to immediately update the fields in addition to the asyncronous xhttp updates that update it later. This works.

So it seems like processing() may have a significant bug in it.

[cems2]

// =========================================
// Main (root) WEB Page HTML
// =========================================



const char index_html[] PROGMEM = R"rawliteral(
<!DOCTYPE HTML><html>
<head>
  <meta name="viewport" content="width=device-width, initial-scale=1">
  <link rel="stylesheet" href="https://use.fontawesome.com/releases/v5.7.2/css/all.css" integrity="sha384-fnmOCqbTlWIlj8LyTjo7mOUStjsKC4pOpQbqyi7RrhN7udi9RwhKkMHpvLbHG9Sr" crossorigin="anonymous">
  <style>
    html {
     font-family: Arial;
     display: inline-block;
     margin: 0px auto;
     text-align: center;
    }
    table { 
      font-size: 1.2rem;
    }
      table.center {
        margin-left:auto; 
        margin-right:auto;
      }
      tr:nth-child(even) {background-color: #f2f2f2;}
      td {
        padding: 10px;
        text-align: center;
        vertical-align: bottom;
      }
      th {
        padding: 5px;
      }
    h2 { font-size: 3.0rem; }
    p { font-size: 3.0rem; }
    .units { font-size: 1.2rem; }
    .ds-labels{
      font-size: 1.5rem;
      vertical-align:middle;
      padding-bottom: 15px;
    }
  </style>
</head>
<body>
  <h2>Farmageddon<br>Sixty61 Server</h2>
  <p>
    <i class="fas fa-thermometer-half" style="color:#059e8a;"></i> 
    <span class="ds-labels">Local Sensors</span> 
    <span id="/localHTML">%LOCAL%</span>
  </p>

 
  <p>
    <i class="fas fa-thermometer-half" style="color:#059e8a;"></i> 
    <span class="ds-labels">Temperature Remotes</span>
    <span id="/temperaturef">%TEMPERATUREF%</span>
   
  </p>
  <h6>Spacesheep Enterprises LtD.<h6>
</body>

<script>
  function fillit(fieldname,rurl) {
    var xhttp = new XMLHttpRequest();
    xhttp.onreadystatechange = function() {
      if (this.readyState == 4 && this.status == 200) {
        document.getElementById(fieldname).innerHTML = this.responseText;
      }
    };
 
  
    xhttp.open("GET", rurl, true);
    xhttp.send();
   };
   

setInterval(function ( ) {
  fillit("/localHTML","/localHTML");
}, 10000) ;

setInterval(function ( ) {
  fillit("/temperaturef","/temperaturef"); 
}, 10000) ;



setInterval(function ( ) {
   fillit("/light","/light");
}, 10000) ;
</script>

</html>)rawliteral";


// =========================================
// Create Async WebServer
// =========================================

// Create AsyncWebServer object on port 80
AsyncWebServer server(80);



// Replaces placeholder with DHT values
String processor(const String& var){ 
  String a;
  if(var == "LOCAL"){
    a=readDSTemperatureH();
  }
  else if(var == "TEMPERATUREF"){
    a= readLocalHTML();
  }

  else if(var == "LIGHT"){
    a= readLight();
  }
  return a;

}
void setup_webasync() {
  
  // Connect to Wi-Fi
  WiFi.begin(ssid, password);
  Serial.println("Connecting to WiFi");
  while (WiFi.status() != WL_CONNECTED) {
    delay(500);
    Serial.print("."+WiFi.status());
  }
  Serial.println();
  
  // Print ESP Local IP Address
  Serial.println(WiFi.localIP());

  // This is the main entry point for the standalone webpage 
  // Route for root / web page

// THIS IS ONLY PLACE PROCESSOR IS USED
 server.on("/", HTTP_GET, [](AsyncWebServerRequest *request){
    request->send_P(200, "text/html", index_html, processor);
  }); 
  
  
  
    // The following are utilities called from the async javascript functions
    // in the main web page to provide updates dynamically on the page


    server.on("/temperaturef", HTTP_GET, [](AsyncWebServerRequest *request){
    request->send_P(200, "text/plain", readDSTemperatureH().c_str());
  });

    server.on("/localHTML", HTTP_GET, [](AsyncWebServerRequest *request){
   request->send_P(200, "text/plain", readLocalHTML().c_str());
  });
  

  // Start server
  server.begin();
}



void setup() {
    // Serial port for debugging purposes
  Serial.begin(115200);
  while (!Serial) {
    delay(10); // hang out until serial port opens
  }
  Serial.println("Serial Comms init");
  
 // setup_Local();
  
 
  setup_webasync();

}
void loop(){
  while (1) {
      Serial.println("Serial Comms test");
      delay(100000);
  }
  
}

[cems2]

Quick tangential question too:(since the webserver is actually doing all the request handling, is it better or worse to have loop() be empty or have it just do some long delays. It seems like leaving it empty just is going to waste cycles calling loop() repeatedly to do nothing at all. Putting a delay in there is going to put it effectively to sleep and just set some timer to interrupt back to the main tthrea periodically, wasting fewer cycles presumably?? or not?

[MrKalach]

@cems2 can you show the final HTML, which you see in your web browser? (ctrl+U in chrome)

[philbowles]

This is a "pilot errror" or "GIGO" issue: For anyone who disagrees, please explain how any code can tell the difference between "%validreplacer%" and "%; height: 100%" they are both strings delimited at both ends by a % - the exact pattern that is being serached for...

Given that you probably don't have a variable anywhere called "; height: 100" - how can you possibly expect the function to perform correctly?

Including style info inline is bad technique in the first place and deprecated for many reasons (things like this being just one of dozens) - simple solution: Put all style information in attached stylesheet and don't put stray % characters in your html - then there is no problem to "solve"

[cems2]

Hello, sorry for the delay. Had to find a moment to go back to my non-working code with the processor() call to regenerate the error. Below I will paste the ESP8266 code, then the the HTML as seen by the browser, then an image of the result. What You will see here is the %TEMPERATUREF% variable has been substituted for a table. When you do this initially nothing bad seems to happen but the moment the external request tries to replace the content of the same span it malfunctions.

The key thing is this---

1. if I remove the Processor() call, it works perfectly. But to do that I hard to use a javascript client side to do the span substitutions of content rather than using processor on the server side.
2. If I use the processor() but just insert plain text rather than an HTML table then it also works perfectly.
   It only malfuctions when Processor inserts a table.

const char index_html[] PROGMEM = R"rawliteral(
<!DOCTYPE HTML><html>
<head>
  <meta name="viewport" content="width=device-width, initial-scale=1">
  <link rel="stylesheet" href="https://use.fontawesome.com/releases/v5.7.2/css/all.css" integrity="sha384-fnmOCqbTlWIlj8LyTjo7mOUStjsKC4pOpQbqyi7RrhN7udi9RwhKkMHpvLbHG9Sr" crossorigin="anonymous">
  <style>
    html {
     font-family: Arial;
     display: inline-block;
     margin: 0px auto;
     text-align: center;
    }
    table { 
      font-size: 1.2rem;
    }
      table.center {
        margin-left:auto; 
        margin-right:auto;
      }
      tr:nth-child(even) {background-color: #f2f2f2;}
      td {
        padding: 10px;
        text-align: center;
        vertical-align: bottom;
      }
      th {
        padding: 5px;
      }
    h2 { font-size: 3.0rem; }
    p { font-size: 3.0rem; }
    .units { font-size: 1.2rem; }
    .ds-labels{
      font-size: 1.5rem;
      vertical-align:middle;
      padding-bottom: 15px;
    }
  </style>
</head>
<body>
  <h2>Farmageddon<br>Sixty61 Server</h2>
  <p>
    <i class="fas fa-thermometer-half" style="color:#059e8a;"></i> 
    <span class="ds-labels">Local Sensors</span> 
    <span id="/localHTML">%LOCAL%</span>
  </p>

 
  <p>
    <i class="fas fa-thermometer-half" style="color:#059e8a;"></i> 
    <span class="ds-labels">Temperature Remotes</span>
    <span id="/temperaturef">%TEMPERATUREF%</span>
   
  </p>
  <h6>Fish & Chips Enterprises Ltd.<h6>
</body>

<script>
  function fillit(fieldname,rurl) {
    var xhttp = new XMLHttpRequest();
    xhttp.onreadystatechange = function() {
      if (this.readyState == 4 && this.status == 200) {
        document.getElementById(fieldname).innerHTML = this.responseText;
      }
    };
 
  
    xhttp.open("GET", rurl, true);
    xhttp.send();
   };
   
 fillit("/localHTML","/localHTML");
 fillit("/temperaturef","/temperaturef"); 
 
setInterval(function ( ) {
  fillit("/localHTML","/localHTML");
}, 10000) ;

setInterval(function ( ) {
  fillit("/temperaturef","/temperaturef"); 
}, 10000) ;


</script>

</html>)rawliteral";


// =========================================
// Create Async WebServer
// =========================================

// Create AsyncWebServer object on port 80
AsyncWebServer server(80);


//  Sadly this "proccessor" feature seems to have a bug in the ESPAsyncWeb server
// So I have replaced  server functions with client side javascript 
// in the main webpage for now.  Maybe someday the library will get debugged.

// Replaces placeholder with DHT values
String processor(const String& var){  // not used due to library bug
  String a;
  if(var == "LOCAL"){
    a=readDSTemperatureFHTML();
  }
  else if(var == "TEMPERATUREF"){
    a= readLocalHTML();
  }

  else if(var == "LIGHT"){
    a= dataLocal.light;
  }

  return a;
 // return String("");
}

now the HTML from the browser. this seems to be okay

<!DOCTYPE HTML><html>
<head>
  <meta name="viewport" content="width=device-width, initial-scale=1">
  <link rel="stylesheet" href="https://use.fontawesome.com/releases/v5.7.2/css/all.css" integrity="sha384-fnmOCqbTlWIlj8LyTjo7mOUStjsKC4pOpQbqyi7RrhN7udi9RwhKkMHpvLbHG9Sr" crossorigin="anonymous">
  <style>
    html {
     font-family: Arial;
     display: inline-block;
     margin: 0px auto;
     text-align: center;
    }
    table { 
      font-size: 1.2rem;
    }
      table.center {
        margin-left:auto; 
        margin-right:auto;
      }
      tr:nth-child(even) {background-color: #f2f2f2;}
      td {
        padding: 10px;
        text-align: center;
        vertical-align: bottom;
      }
      th {
        padding: 5px;
      }
    h2 { font-size: 3.0rem; }
    p { font-size: 3.0rem; }
    .units { font-size: 1.2rem; }
    .ds-labels{
      font-size: 1.5rem;
      vertical-align:middle;
      padding-bottom: 15px;
    }
  </style>
</head>
<body>
  <h2>Farmageddon<br>Sixty61 Server</h2>
  <p>
    <i class="fas fa-thermometer-half" style="color:#059e8a;"></i> 
    <span class="ds-labels">Local Sensors</span> 
    <span id="/localHTML"><table class="center">
 <tr>
		<td> Temperature[F] </td>
		<td> Location </td>
		<td> Sensor ID </td>
</tr>
</table></span>
  </p>

 
  <p>
    <i class="fas fa-thermometer-half" style="color:#059e8a;"></i> 
    <span class="ds-labels">Temperature Remotes</span>
    <span id="/temperaturef"><table class="center">
	<tr>
		<td> Temperature<sup class="units">&deg;<smaller>F<smaller></sup> </td>
		<td> Humidity<sup class="units"><smaller>%<smaller></sup> </td>
		<td> Pressure<sup class="units"><smaller>hPa<smaller> </td>
		<td> Light<sup class="units"><smaller>arb<smaller></td>
	</tr>
	<tr>
		<td> --off--<sup class="units">&deg;</sup> </td>
		<td> --off-- </td>
		<td> --off-- </td>
		<td> 333 </td>
	</tr>
</table>
</span>
   
  </p>
  <h6>Fish & Chips Enterprises Ltd.<h6>
</body>

<script>
  function fillit(fieldname,rurl) {
    var xhttp = new XMLHttpRequest();
    xhttp.onreadystatechange = function() {
      if (this.readyState == 4 && this.status == 200) {
        document.getElementById(fieldname).innerHTML = this.responseText;
      }
    };
 
  
    xhttp.open("GET", rurl, true);
    xhttp.send();
   };
   
 fillit("/localHTML","/localHTML");
 fillit("/temperaturef","/temperaturef"); 
 
setInterval(function ( ) {
  fillit("/localHTML","/localHTML");
}, 10000) ;

setInterval(function ( ) {
  fillit("/temperaturef","/temperaturef"); 
}, 10000) ;


</script>

</html>

[cems2]

ANd here is what it looks like if I remove the call to processor and don't make any other changes

Here is the html seen by the broswer

<!DOCTYPE HTML><html>
<head>
  <meta name="viewport" content="width=device-width, initial-scale=1">
  <link rel="stylesheet" href="https://use.fontawesome.com/releases/v5.7.2/css/all.css" integrity="sha384-fnmOCqbTlWIlj8LyTjo7mOUStjsKC4pOpQbqyi7RrhN7udi9RwhKkMHpvLbHG9Sr" crossorigin="anonymous">
  <style>
    html {
     font-family: Arial;
     display: inline-block;
     margin: 0px auto;
     text-align: center;
    }
    table { 
      font-size: 1.2rem;
    }
      table.center {
        margin-left:auto; 
        margin-right:auto;
      }
      tr:nth-child(even) {background-color: #f2f2f2;}
      td {
        padding: 10px;
        text-align: center;
        vertical-align: bottom;
      }
      th {
        padding: 5px;
      }
    h2 { font-size: 3.0rem; }
    p { font-size: 3.0rem; }
    .units { font-size: 1.2rem; }
    .ds-labels{
      font-size: 1.5rem;
      vertical-align:middle;
      padding-bottom: 15px;
    }
  </style>
</head>
<body>
  <h2>Farmageddon<br>Sixty61 Server</h2>
  <p>
    <i class="fas fa-thermometer-half" style="color:#059e8a;"></i> 
    <span class="ds-labels">Local Sensors</span> 
    <span id="/localHTML">%LOCAL%</span>
  </p>

 
  <p>
    <i class="fas fa-thermometer-half" style="color:#059e8a;"></i> 
    <span class="ds-labels">Temperature Remotes</span>
    <span id="/temperaturef">%TEMPERATUREF%</span>
   
  </p>
  <h6>Fish & Chips Enterprises Ltd.<h6>
</body>

<script>
  function fillit(fieldname,rurl) {
    var xhttp = new XMLHttpRequest();
    xhttp.onreadystatechange = function() {
      if (this.readyState == 4 && this.status == 200) {
        document.getElementById(fieldname).innerHTML = this.responseText;
      }
    };
 
  
    xhttp.open("GET", rurl, true);
    xhttp.send();
   };
   
 fillit("/localHTML","/localHTML");
 fillit("/temperaturef","/temperaturef"); 
 
setInterval(function ( ) {
  fillit("/localHTML","/localHTML");
}, 10000) ;

setInterval(function ( ) {
  fillit("/temperaturef","/temperaturef"); 
}, 10000) ;


</script>

</html>

And for completeness here is the webserver setup code, showing how I comment out the processor() and use the line below it instead.

/* server.on("/", HTTP_GET, [](AsyncWebServerRequest *request){
    request->send_P(200, "text/html", index_html, processor);
  });  // processor feature is broken in ESPAsyncWebServer library so we do the following instead
 */
    server.on("/", HTTP_GET, [](AsyncWebServerRequest *request){
    request->send_P(200, "text/html", index_html); 
  });

      server.on("/index.html", HTTP_GET, [](AsyncWebServerRequest *request){
    request->send_P(200, "text/html", index_html); 
  });
  
    // The following are utilities called from the async javascript functions
    // in the main web page to provide updates dynamically on the page
    server.on("/temperaturef", HTTP_GET, [](AsyncWebServerRequest *request){
    request->send_P(200, "text/plain", readDSTemperatureFHTML().c_str());
  });

    server.on("/localHTML", HTTP_GET, [](AsyncWebServerRequest *request){
   request->send_P(200, "text/plain", readLocalHTML().c_str());
  });
  
    server.on("/lightYAML", HTTP_GET, [](AsyncWebServerRequest *request){
    request->send_P(200, "text/plain", readLightYAML(" ").c_str());
  });
  
     // This is the one you will use to poll this device from your raspberry pi
  //  it gives a database style reply rather than HTML
    server.on("/allYAML", HTTP_GET, [](AsyncWebServerRequest *request){
   request->send_P(200, "text/plain", readAllYAML(" ").c_str());
  });
  
    // and for other uses these provide YAML for each sensor group separately.
    server.on("/localYAML", HTTP_GET, [](AsyncWebServerRequest *request){
   request->send_P(200, "text/plain", readLocalYAML(" ").c_str());
  });


    server.on("/temperaturefYAML", HTTP_GET, [](AsyncWebServerRequest *request){
   request->send_P(200, "text/plain", readDSTemperatureFYAML(" ").c_str());
  });


    // used mainly for debugging
    server.on("/temperaturec", HTTP_GET, [](AsyncWebServerRequest *request){
    request->send_P(200, "text/plain", readDSTemperatureC().c_str());
  });
  

  // Start server
  server.begin();
  Serial.println("WebAsync setup complete");

[cems2]

Comment: It's hard for me to see how this is a processor() bug. It appears the processor is creating the code I expect to see filled in with the table. So I'm hoping this isn't my own stupidity here. The duplicated table parts that show up do not show up until the javascript does a replace on the span contents. But the browser HTML doesn't show what that looks like.

However is seems the span content subsitution works just fine when processor is commented out. SO it's really hard to figure out this intereraction.

Since the problem only occurs when processor() is used and the substituiton is a table it's confusing.

[me21]

I don't understand why the browser displays the table twice. There are no repeats in the HTML response. Looks like the span content substitution goes crazy if the span already includes table tag.

But there is another thing which looks like it needs fixing: the processor function. Consider:

if(var == "LOCAL"){
    a=readDSTemperatureFHTML();
  }
  else if(var == "TEMPERATUREF"){
    a= readLocalHTML();
  }

In that snippet, you return readDSTemperatureFHTML() for LOCAL variable and readLocalHTML() for TEMPERATUREF variable. Is that correct? Based on the function names, it looks like they should be swapped.

[cems2]

You are correct about the swapped logic (oops). And furthermore that's interesting as well since it would be half way to explaining why the tables are showing up in the wrong place. (local table duplicated into the remote field, and vica versa. ) However I say half way because the span-replacement should simply nuke the contents and re-write it. What it looks like it is doing is this (even though this should nto happen). 1. with processor() 1. variables in spans' content gets rewritten (to unintendedly wrong tables) 2. javascript span re-write simply pre-pends the replacement text to the content of the span rather than erasing it. (erase/overwrite is expected from document.getElementById(fieldname).innerHTML = this.responseText;). 2. Without processor() 1. Varables in span content remain %VARIABLENAME% 2. javascript span re-write overwrties these as expected So... this seems to say processor is doing it's job correctly but something weird is happening with the spans not behaving as expected. By expected, I mean this javascript is doing what I intend it to ``` function fillit(fieldname,rurl) { var xhttp = new XMLHttpRequest(); xhttp.onreadystatechange = function() { if (this.readyState == 4 && this.status == 200) { document.getElementById(fieldname).innerHTML = this.responseText; } }; ``` so either innerHTML isn't what I think it is (everything between the span tags) or this.responseText does a pre-pend which would be had to beleive

[me21]

Use divs instead of spans. Span is inline element, it should not contain block elements such as table.

[cems2]

Aha! I was too quick to say there's no issue with processor()

Apparently, I think, processor malfunctions when it encounters a lone % sign in the text. (a user error) But then fails to find subsequent variables after this.

There's actually two different errors here. One is a malfunction caused by the user and is correctable and the other is more mysterious.

Evidently it does not require a double %VARIABLE% and just a %VARIABLE will still match!. my guess is that it is looking either for a match of bracketing percent signs, or a single percent sign plus a certain fixed number of characters in length.

As a result it will match any lone percent sign in the text and try to process it.

Now at this point the users processor() command will get this value and try to process it. If it does not recognize the variable a typical user written processor() function might return a blank string which would basically erase some part of the HTML!!! Or it could return the matched string but then this would re-insert the % sign again and cause an infinite recursion.

I chose to let it erase the bit of HTML that contained the rogue % sign.

But this then revealed a mysterious error: The processor did not find the second actual variable.

Weird.

Example:

I created a form with two variables in it %LOCAL% and %TEMPERATUREF%

THe substitution for %LOCAL% accidentally on my part had a "%" symbol as part of the text (Relative Humitdiy percentage).

here's what I see when I have processor print out what the variable name it is matching.

Processor: Var: LOCAL :::::   

So it found the variable %LOCAL% and it will now substitute the following table which happens to contain the character %

<table class="center">
	<tr>
		<td> Temperature<sup class="units">&deg;<smaller>F<smaller></sup> </td>
		<td> Humidity<sup class="units"><smaller>%<smaller></sup> </td>
		<td> Pressure<sup class="units"><smaller>hPa<smaller> </td>
		<td> Light<sup class="units"><smaller>arb<smaller></td>
	</tr>
	<tr>
		<td> --off--<sup class="units">&deg;</sup> </td>
		<td> --off-- </td>
		<td> --off-- </td>
		<td> 0 </td>
	</tr>
</table>

You can see the rogue % sign bracketed by tags in the above text block.
Next I see processing is called a second time and it says the following

Processor: Var: <smaller></sup> </td>
		<td> Pr :::::

This is unexpected. The processor has been called with the variable named literally

<smaller></sup> </td>
		<td> Pr

inspecting the text block that was inserted we see that there wasn't a trailing % sign after the letters "Pr" above. so evidently processors parser must be matching anything of that text length that begins with a percent sign and doesn't need the close-%.

So obviously that was kind of a dumb move on my part to put in a percent. But then a second error emerges:

the processing stops! it fails to continue hunting for the final variable "%TEMPERATUREF%

Here's the code:
first the processor() I wrote.

String processor(const String& var){  // not used due to library bug
  String a;
  Serial.println("Processor: Var: " + var+" :::::");
  if(var == "LOCAL"){
    a= readLocalHTML();
    Serial.println ("substituting LOCAL "+var+ " for "+ a+":;:;:;");
  }
  else if(var == "TEMPERATUREF"){
    a=readDSTemperatureFHTML();
    Serial.println ("substituting TEMPERATUREF"+var+ " for "+ a+":;:;:;");
  }

  else if(var == "LIGHT"){
    a= dataLocal.light;
    Serial.println ("substituting LIGHT"+var+ " for "+a+":;:;:;");
  }

  return a;
 // return String("");
}

next the HTML string it is parsing (same as before)

const char index_html[] PROGMEM = R"rawliteral(
<!DOCTYPE HTML><html>
<head>
  <meta name="viewport" content="width=device-width, initial-scale=1">
  <link rel="stylesheet" href="https://use.fontawesome.com/releases/v5.7.2/css/all.css" integrity="sha384-fnmOCqbTlWIlj8LyTjo7mOUStjsKC4pOpQbqyi7RrhN7udi9RwhKkMHpvLbHG9Sr" crossorigin="anonymous">
  <style>
    html {
     font-family: Arial;
     display: inline-block;
     margin: 0px auto;
     text-align: center;
    }
    table { 
      font-size: 1.2rem;
    }
      table.center {
        margin-left:auto; 
        margin-right:auto;
      }
      tr:nth-child(even) {background-color: #f2f2f2;}
      td {
        padding: 10px;
        text-align: center;
        vertical-align: bottom;
      }
      th {
        padding: 5px;
      }
    h2 { font-size: 3.0rem; }
    p { font-size: 3.0rem; }
    .units { font-size: 1.2rem; }
    .ds-labels{
      font-size: 1.5rem;
      vertical-align:middle;
      padding-bottom: 15px;
    }
  </style>
</head>
<body>
  <h2>Farmageddon<br>Sixty61 Server</h2>
  <p>
    <i class="fas fa-thermometer-half" style="color:#059e8a;"></i> 
    <span class="ds-labels">Local Sensors</span> 
    <span id="/localHTML">%LOCAL%</span>
  </p>

 
  <p>
    <i class="fas fa-thermometer-half" style="color:#059e8a;"></i> 
    <span class="ds-labels">Temperature Remotes</span>
    <span id="/temperaturef">%TEMPERATUREF%</span>
   
  </p>
  <h6>Fish & Chips Enterprises Ltd.<h6>
</body>

<script>
  function fillit(fieldname,rurl) {
    var xhttp = new XMLHttpRequest();
    xhttp.onreadystatechange = function() {
      if (this.readyState == 4 && this.status == 200) {
        document.getElementById(fieldname).innerHTML = this.responseText;
      }
    };
 
  
    xhttp.open("GET", rurl, true);
    xhttp.send();
   };
   
 fillit("/localHTML","/localHTML");
 fillit("/temperaturef","/temperaturef"); 
 
setInterval(function ( ) {
  fillit("/localHTML","/localHTML");
}, 10000) ;

setInterval(function ( ) {
  fillit("/temperaturef","/temperaturef"); 
}, 10000) ;


</script>

</html>)rawliteral";

[cems2]

Suggested way to change the code to avoid this pitfall.

1. set a maximum variable name length
2. require a close-% tag on every variable.
3. And have the parser skip over this text if the close-% is missing.

That combination would not handle every possible user screw-up but would work for many likely cases where the user has a percent sign in a web page text.

[stale]

[STALE_SET] This issue has been automatically marked as stale because it has not had recent activity. It will be closed in 14 days if no further activity occurs. Thank you for your contributions.

[stale]

[STALE_DEL] This stale issue has been automatically closed. Thank you for your contributions.