Nicholas,
Thank you for the extensive reply. We actually had another site kicked off for a backup that began running from 23:30 and was running for around 30 minutes when the problems began. At the time, it was the only backup that was running on the server.
I passed along your second possibility to the hosting provider and they are exploring it, but they came back with the following report while it is still being investigated.
Please let me know if anything stands out that may be causing an issue.
-------
Step 1.) PHP-FPM Max Children limit of 10 reached 1646 times for lambertvillenj.org on June 1st, 2022
Log path: /opt/cpanel/ea-php74/root/usr/var/log/php-fpm/error.log
* Example error:
[root@cloudvpsserver ~]# grep -i max_children /opt/cpanel/ea-php74/root/usr/var/log/php-fpm/error.log|grep lambertvillenj|grep 01-Jun-2022|tail -1
[01-Jun-2022 23:30:02] WARNING: [pool lambertvillenj_org] server reached max_children setting (10), consider raising it
* Number of times that errors was logged on June 1st 2022:
[root@cloudvpsserver ~]# grep -i max_children /opt/cpanel/ea-php74/root/usr/var/log/php-fpm/error.log|grep lambertvillenj|grep 01-Jun-2022|wc -l
1646
TO RESOLVE THIS ISSUE:
I adjusted the PHP-FPM settings for lambertvillenj.org
* WHM => MultiPHP Manager => User domain Settings => lambertvillenj.org => PHP-FPM Settings
Max Requests: 20 => 200
Max Children: 10 => 20
Process Idle Timeout: 10 => 5
1A.) I
raised the "
Max Requests" to increase how many requests can be served by an existing "Max Child" before it is recycled.
1B.) I
raised the "
Max Children" to allow for more concurrent connections / requests
1C.) I
lowered the "
Process Idle Timeout" to make idle processes cycle faster, rather than sitting unused.
==========
Step 2.) Checked the PHP-FPM php error log, but no recent errors were recorded:
PHP-FPM PHP error log: /home/lambertvillenj/logs/lambertvillenj_org.php.error.log
[root@cloudvpsserver ~]# tail -1 /home/lambertvillenj/logs/lambertvillenj_org.php.error.log
[21-May-2022 09:34:43 UTC] PHP Warning: session_start(): Failed to read session data: user (path: /var/cpanel/php/sessions/ea-php73) in /home/lambertvillenj/public_html/libraries/joomla/session/handler/native.php on line 260
[root@cloudvpsserver ~]#
NO ISSUES TO ADDRESS
==========
Step 3.) Several Apache Timeout errors for lambertvillenj.org
Apache error log: /var/log/apache2/error_log
[root@cloudvpsserver ~]# grep lambertvillenj /var/log/apache2/error_log
...etc...
[Thu Jun 02 00:01:21.672176 2022] [proxy_fcgi:error] [pid 14824:tid 46962644735744] (70007)The timeout specified has expired: [client 74.105.2.160:36570] AH01075: Error dispatching request to : (polling), referer: https://lambertvillenj.org/
[Thu Jun 02 00:01:41.360365 2022] [proxy_fcgi:error] [pid 1271:tid 46962604812032] (70007)The timeout specified has expired: [client 74.105.2.160:36624] AH01075: Error dispatching request to : (polling), referer: https://lambertvillenj.org/
[Thu Jun 02 00:02:07.468748 2022] [proxy_fcgi:error] [pid 1381:tid 46962611115776] (70007)The timeout specified has expired: [client 52.91.90.57:17008] AH01075: Error dispatching request to : (polling), referer: https://lambertvillenj.org/resident/calendar/2570
[Thu Jun 02 00:02:49.621652 2022] [proxy_fcgi:error] [pid 1380:tid 46962604812032] (70007)The timeout specified has expired: [client 74.105.2.160:37638] AH01075: Error dispatching request to : (polling), referer: https://lambertvillenj.org/
[Thu Jun 02 00:02:52.198788 2022] [proxy_fcgi:error] [pid 1381:tid 46962596407040] (70007)The timeout specified has expired: [client 74.105.2.160:37646] AH01075: Error dispatching request to : (polling), referer: https://lambertvillenj.org/
[Thu Jun 02 00:03:30.000546 2022] [proxy_fcgi:error] [pid 14824:tid 46962598508288] (70007)The timeout specified has expired: [client 74.105.2.160:36932] AH01075: Error dispatching request to : (polling), referer: https://lambertvillenj.org/
[Thu Jun 02 00:04:13.246408 2022] [proxy_fcgi:error] [pid 14824:tid 46962499417856] (70007)The timeout specified has expired: [client 74.105.2.160:37034] AH01075: Error dispatching request to : (polling), referer: https://lambertvillenj.org/
TO RESOLVE THIS ISSUE:
I checked the current Apache configuration, and found no Timeout value set. When a value is not defined, the "default value" is used, in this case the "Default Timeout" is "60"
* WHM => Apache Configuration => Include Editor => PreVirtual Host Include => All Versions:
* Original settings:
KeepAlive On
KeepAliveTimeout 2
MaxKeepAliveRequests 1500
<IfModule mpm_event_module>
StartServers 6
MinSpareThreads 150
MaxSpareThreads 250
ServerLimit 32
ThreadsPerChild 50
MaxRequestWorkers 1600
MaxConnectionsPerChild 10000
</IfModule>
* Adjusted Settings:
KeepAlive On
KeepAliveTimeout 2
MaxKeepAliveRequests 1500
Timeout 300
<IfModule mpm_event_module>
StartServers 6
MinSpareThreads 150
MaxSpareThreads 250
ServerLimit 32
ThreadsPerChild 50
MaxRequestWorkers 1600
MaxConnectionsPerChild 10000
</IfModule>
I increased the
Apache Timeout from
60 to
300 (
5 minutes) which is fairly standard, especially when you see a lot of those "
(70007)The timeout specified has expired" errors.