This notice is to inform our valued customers and the public about a critical incident affecting MySQL databases on USHost247.com.
Last night, around 8:51 PM EST, a routine yum repository update inadvertently upgraded the MySQL packages on our server to versions incompatible with cPanel. Specifically, MySQL was upgraded from version 8.4.4 to 8.4.5, and the older mysql-community-* packages were replaced with mysql-cloud-* equivalents.
While initial checks suggested the update was successful and the system had stabilized, further investigation revealed that the MySQL databases are currently missing.
We have determined that the root cause of this issue was the unexpected upgrade of MySQL to the mysql-cloud-* packages. Following this, cPanel ran several MySQL-related checks which may have temporarily impacted availability.
We have made several attempts to rectify this situation:
Currently, the MySQL data appears to be corrupted, and our website is experiencing ongoing database connection errors. We have actively been attempting to repair the existing MySQL 8.4 installation. However, due to the complexities and incompatibilities introduced by the Cloud version, these repair attempts have been unsuccessful, and we have encountered errors during database recovery.
Our Immediate Action Plan:
Given the current state, we are now proceeding with the following critical steps:
We will continue to provide updates on our progress. Thank you for your patience and understanding as we work to restore full service.
Last night, around 8:51 PM EST, a routine yum repository update inadvertently upgraded the MySQL packages on our server to versions incompatible with cPanel. Specifically, MySQL was upgraded from version 8.4.4 to 8.4.5, and the older mysql-community-* packages were replaced with mysql-cloud-* equivalents.
While initial checks suggested the update was successful and the system had stabilized, further investigation revealed that the MySQL databases are currently missing.
We have determined that the root cause of this issue was the unexpected upgrade of MySQL to the mysql-cloud-* packages. Following this, cPanel ran several MySQL-related checks which may have temporarily impacted availability.
We have made several attempts to rectify this situation:
- Rollback Attempt: An attempt to roll back the MySQL changes was unsuccessful.
- Downgrade Attempt: We attempted to downgrade MySQL from version 8.4 to 8.0, as the MySQL Community Edition is not available for version 8.4. This downgrade unfortunately failed due to compatibility issues.
- Reinstallation of MySQL 8.4: As a result of the failed downgrade, we reinstalled MySQL 8.4. This action unfortunately led to the installation of the MySQL 8.4 Cloud version, which introduced significant and incompatible changes to the data directory, database log files, and socket files compared to both our previous MySQL environment and the current cPanel configuration. This situation arose after the removal of a MySQL upgrade lock.
Currently, the MySQL data appears to be corrupted, and our website is experiencing ongoing database connection errors. We have actively been attempting to repair the existing MySQL 8.4 installation. However, due to the complexities and incompatibilities introduced by the Cloud version, these repair attempts have been unsuccessful, and we have encountered errors during database recovery.
Our Immediate Action Plan:
Given the current state, we are now proceeding with the following critical steps:
- Reinstallation of MySQL 8.0: We will be reinstalling MySQL version 8.0 on the server.
- Database and User Recreation: Following the reinstallation, we will recreate all databases and user accounts.
- Data Restoration from Backups: We will then restore your data from the most recent backups available on our server.
We will continue to provide updates on our progress. Thank you for your patience and understanding as we work to restore full service.
