-
Notifications
You must be signed in to change notification settings - Fork 7.7k
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
String corruption on deployments with php-fpm 8.1.6 #8739
Comments
Hello @nono-gdv, I just wanted to drop by and add that at Mollie, we are experiencing the exact same issue. But we use To me, it looks like the interned strings cache is read while it is being written to.
or
Note that the cut off is a different places. Our error reporting uses JSON as a transport so I am unable to provide the exact bytes following the valid characters. We've tried increasing the interned strings buffer but to no avail. |
Adding more context to what @willemstuursma reported (I work at the same company)... We enabled verbose To make things more interesting, the tools we have in place to monitor opcache show a lot of free memory on that buffer (not even 50% is used). @cmb69 @iluuu1994 is there anything we could provide to help on finding the root cause? |
@lcobucci See:
Is it possible you're setting the INI value to Error handling / logging was improved in #9260 but only for PHP 8.2. |
We're setting it to |
Do you have a sufficiently large |
I believe so (it's set to |
Hi @cmb69 we have upgraded from PHP 8.1.9 to 8.1.10 and the problem went away. |
Unforunately, I don't have definitive way to replicate this, but I'm seeing this issue on PHP 8.1.27. I've noticed it twice in the past week, but seeing what happen, I think it's very likely it has happened before.
The opcache stats show |
I'm not sure if it makes a difference, but we use symlinks for our directories. On deployment, we change |
I just noticed the same issue as I mentioned above in #8739 (comment) (bullet 2). The same class had the issue of the last letter of the constant being removed, though it was a different constant. This time, it was a constant with name This is on PHP 8.1.29. However, I already raised memory since last time and the stats from
|
Referring the bullet points from #8739 (comment) |
Thanks, @nielsdos. Both involve using # Assume web root is as `/var/www/site/` and is a symlink to `/var/www/site_a`.
cd /var/www
rsync -a --delete /code/to/deploy/ site_b/
ln -s site_b ~/site_new
chown -h apache:apache ~/site_new
sudo mv -Tf ~/site_new /var/www/site After we've done that, we run From what I've seen, it's the Apache version of opcache that gets corrupted. What's the alternative to |
I'll try to look into the opcache_reset issue tomorrow, but can't make any promises.
Yes indeed, until the opcache_reset bug you experience is fixed this would be the way to go I think.
I don't think you can easily force a reset here. |
Thanks, @nielsdos. Fortunately, it's not all too often that it happens. |
@nielsdos We've been getting this on a daily basis as well. So far I've seen this only with PHP 8.1. This is the case witih PHP 8.1.29
So instead of php it would just cache phpp (an extra P at the end). The only solution to resolve the issue at that time is to reset OPcache or reload the PHP-FPM. Here is one more case where it adds a unique character:
I can confirm the issue is not related to Remi PHP as we've experienced the same with the non-remi php build. This ones a case with PHP 8.1.17.
It is hard to replicate the issue which prevents us from gathering more information. It just happens randomly on its own when opcache_reset is called. |
I'm working on figuring this out, #14471 is definitely related as both show the same symptoms. So far I figured out that |
I identified two race conditions with requests firing and the restart code. |
I identified a third issue but haven't fully managed to understand that one yet.
|
Description
Hello,
We experience seemingly random in-memory corruption of code when deploying new versions of our PHP app on php-fpm 8.1.6 on Linux. Here is a sample error message:
This example happens in the class loader from composer. The beginning of the file name is correct, the end is obviously not.
Another example (another release on another server):
This bug is not easily reproducible, it happens in random-looking places in the code, sometimes but not always, on some servers but not others (we have 10 app servers, the bug appears on one server at a time, very rarely two, not on every release but more than half of them).
When it happens, it looks like there is only one corrupted chunk of data; depending on the affected data, it may only cause a few real errors or basically prevent the code from working on the server at all. It generates errors for a few minutes. Restarting the fpm daemon fixes it; reloading it or running
opcache_reset()
does not.We will try to reproduce it on a staging server; at the moment I can only say it appears under load and not under light testing.
It may or may not be related to #8731 (same app, same servers, seemingly fixed by downgrading to 8.0.19).
I know this report is not really helpful, but I'm at a loss how to make it more useful.
Downgrading to PHP 8.0.19 seems to fix the issue.
PHP Version
PHP 8.1.6 (cli) (built: May 17 2022 16:49:19) (NTS)
Operating System
Debian 10
The text was updated successfully, but these errors were encountered: