As an IT professional for 25+ years I'm trying to understand why a site, with the level of demand this one has, is being run on a single box and maintained by a couple of guys. No matter how competent you may be that's an old school approach.
If you're using your own server in a colo then you really need to be running RAID and the colo should have another box ready to hot swap to. With all due respect, 24 hrs down time and the need for a manual rebuild is pretty amateur with the options you have available to you.
At the very least write a script and ftp it off site several times a day.
# Dump SQL data
/usr/bin/mysqldump -uUSER -pPASS --all-databases --opt -l --result-file=/backup/mysql/mysqld ump.sql
# Compress sql dump
tar zcf /backup/mysqldump.sql.tar.gz /backup/mysql
# UPLOAD TO FTP (DD deletes on successful upload)
ncftpput -f ftplogin.cfg -DD /remote_path /backup/2010_12_4.tar.gz
# EMAIL TO MAILBOX
uuencode /home/user/backup/$DATE.tar.gz Some_Hosting_SQL_Dbases.$DATE.t ar.gz | mail -s "Some Hosting SQL Database Backup" [email protected]