php_fpm: Sporadic error "Failed to get metrics"

[steffenweber]

Requesting the PHP-FPM status URL can sporadically fail with an HTTP 503 error "scoreboard already in used".

See: https://github.com/php/php-src/blob/d84ef967424343abcc48b8f945d5e7807f1c1e09/sapi/fpm/fpm/fpm_status.c#L96

Maybe Datadog could retry once or twice before raising an error?

ERROR (php_fpm.py:91): Failed to get metrics from http://localhost/fpm-status.
  Error 503 Server Error: Service Unavailable for url: http://localhost/fpm-status?json=True
ERROR (__init__.py:829): Check 'php_fpm' instance #0 failed
  Traceback (most recent call last):
    File "/opt/datadog/5.25.0/agent/checks/__init__.py", line 812, in run
      self.check(copy.deepcopy(instance))
    File "/opt/datadog/latest/venv/lib/python2.7/site-packages/datadog_checks/php_fpm/php_fpm.py", line 74, in check
      raise status_exception  # pylint: disable=E0702
  HTTPError: 503 Server Error: Service Unavailable for url: http://localhost/fpm-status?json=True

[masci]

Hi @steffenweber thanks for the detailed report.
Not sure how often and for how long we can incur in that lock but a retry policy seems easy to add, going to schedule some work, will keep this issue updated.

[masci]

@steffenweber a fix was merged on master and will be shipped with the next agent minor release.

[steffenweber]

Thank you! 😃

[masci]

Closing for now, feel free to reopen if the fix shipped in #1581 doesn't work

[steffenweber]

The fix works! 👍