Hello,
Beginning around 11:20 UTC today, Cloudflare experienced widespread degradation to traffic in our network. We deeply apologize for this disruption. We have published a blog detailing exactly what happened. The majority of the impact was resolved by 14:30 UTC, with all remaining downstream impacted services being fully operational at 17:06 UTC.
In short, this incident occurred due to a latent bug related to a configuration file for our bot management service. There is no evidence of an attack or malicious activity causing the issue.
Bot management detection sits in the critical path for traffic flow. When any request hits our network, the bot management module should load, but due to the bug, the configuration file grew beyond an expected size of entries. This triggered a crash in the software system that handles traffic for a number of Cloudflare's services. Customers experienced HTTP 500 status code errors as many of Cloudflare's services suffered an outage.
The Cloudflare team diagnosed the issue and reverted to an earlier version of the bot detection module configuration that was below the limits. We also modified the core proxy service so it would not fail in this way if limits are reached again.
For a more detailed description and timeline of the incident, and the steps we took to address it, please review our blog.
Sincerely,
Cloudflare Team
A new security patch being applied requires a reboot.
There was a series of emergency patches that needed to be applied immediately. The system will need to reboot several times.
Some customers are experiencing the following issue. It is being investigated.
Error: phpinimng failed: Job for plesk-php74-fpm.service failed because the control process exited with error code. See "systemctl status plesk-php74-fpm.service" and "journalctl -xe" for details.
ERROR:php_fpm:Failed to start plesk-php74-fpm service, pause before retry
Job for plesk-php74-fpm.service failed because the control process exited with error code. See "systemctl status plesk-php74-fpm.service" and "journalctl -xe" for details.
Failed to start plesk-php74-fpm service
There will be a maintenance reboot of ALL of our 3CX VoIP servers. The actual reboot will take only 5 minutes but we've allowed extra time should an issue arise.
We are aware of an issue related to all services within amazon web services us-east-2 region. We have many services including web and voip in that region. We are working on answers from aws as well as internet carriers regarding connectivity. We know we can access services from remote locations but not from florida
UPDATE: we mistakenly referred to us-east-1, it is us-east-2 and it is not accessible from the United States, we are only able to reach the aws console for that region via vpn thru the UK
UPDATE: 3:26 aws started investigating regional outage.
UPDATE 3:38 isdown https://isdown.app/integrations/aws posted the first public acknowledgement of this connectivity issue.
UPDATE 4:00 it appears services are coming back online. we will continue to monitor the situation for today before considering any changes to OUR setup.
Between 11:
We have identified an issue stopping users from logging into the web panel for one of our servers. (us-east-2a-2) We anticipate the issue being resolved by Friday September 8, 2022 12:00PM EST
UPDATE Friday September 8, 2022 11:33AM
Our hosting panel vendor has issued a fix to remedy the issue. However, the server must be taken offline temporarily to do so. To assure the lowest impact possible, we have scheduled this to take place Sunday September 11, 2022 from 1:00AM - 3:00AM EST.
An unexpected deployment of new Apache Server configuration files stopped service for some customers on the us-east-2a-2.hostlabs.pro server cluster. The issue was resolved by level II support around 10:30am. The root is under investigation. All services are now restored and operating as expected.
Trying to stop service httpd... attempt to stop service httpd - service doesn't exist (missing unit file or not executable control script) /var/log/plesk/rc_actions.log: START pleskrc INFO: [Fri Mar 4 10:09:33 EST 2022]: Service: apache, Action: stop Trying to stop service httpd... attempt to stop service httpd - service doesn't exist (missing unit file or not executable control script) ***** problem report ***** Warning: stop service httpd failed STOP pleskrc START pleskrc INFO: [Fri Mar 4 10:09:33 EST 2022]: Service: apache, Action: status attempt to status service httpd - service doesn't exist (missing unit file or not executable control script) STOP pleskrc These symptoms are indicate that the Apache service failed to start due to a CentOS systemd bug: https://bugzilla.redhat.com/show_bug.cgi?id=1439989 This behavior has been registered by the Plesk development team as bug PPPM-7414. This bug is no longer present in up-do-date OSes, but there still are several factors that can cause this issue to resurface. For example, it is recommended to keep all the system packages updated, especially those related to systemd, dbus and kernel.
Will you look at the CSS for performancePups? I had to do some CSS on the in theme options, because the new calendar plugin is very limited on styling.
My issue is that when I use the arrows to go to the next month, the calendar defaults back to the original style.
An unscheduled reboot of the hosting system will occur tonight (12-16-2019) at 8:30. Services will be unavailable for up to 10 minutes.
Below is a real-time overview of our servers where you can check if there's any known issues.
| Server Name | HTTP | FTP | POP3 | PHP Info | Server Load | Uptime |
|---|---|---|---|---|---|---|
| soldier.host.burnserv.net | PHP Info |