| Author |
 Topic  |
|
|
DennisMCSE
Moderator
Canada
2400 Posts
Status: offline |
 Posted - 07/16/2012 : 1:08:26 PM
|
We had a fire at work that caused us to shut down since last Wednesday. We've been doing cleanup and I've had to restore dozens of computers that were damaged in the fire. We also had to do an emergency power down of the server room (and hope that the servers came back up). We are back up in record time, thanks to having mostly current backups for our equipment.
Having backups are nice, but if the local backup location is damaged, well that's the same as no backups (need offsite storage). And if you have old backups, well that's almost the same as no backups, because you are sitting around updating program files, sorting outdated data files, etc., to get the computer to a current state. And if you don't test your backups, that's almost the same as having no backups, because if you can't be positive that your backup can be restored, then you are out of luck as well.
We had current backups as well as offsite backups. A few of the current backups were corrupted, so had to restore older backups and then update the software, which luckily wasn't that old. And lukily I tested the retores just a few weeks ago, so was confident they worked.
So backups are nice, but you need to have a recovery plan as well. We did and I'm so thankfull that we did and that it worked, or it would have been a RGE (Resume Generating Event) for me.
|
DennisMCSE
Blog: http://itprofirewalker.wordpress.com/
Follow me on Twitter http://twitter.com/Firewalker96
|
|
|
Mark Minasi
Chief cook and bottle washer
USA
10658 Posts
Status: offline |
Posted - 07/16/2012 : 10:21:09 PM
|
| Glad you were one of the "awake" guys on the DR, Dennis... but I'm not surprised.<g> |
Mark
tweetin' at mminasi |
 |
|
|
Curt
Moderator
USA
6652 Posts
Status: offline |
Posted - 07/17/2012 : 12:09:12 AM
|
Wow, that was extensive.
Seem like you saw the dangers well in advance.
I had to restore data for a client today.
They would never done it on their own. Too foolish to care I guess.
I did that backup of code last year and it and got it off the server and moved some of it off network.
Like you Dennis, I saw trouble ahead.
Congrads on coming through with solutions.
|
Curt Spanburgh
Microsoft Certified Business Solution Specialist.
Dynamics CRM MVP
Contributing Editor, Windows IT Pro
He that is walking with wise persons will become wise, but he that is having dealings with the stupid ones will fare badly.
Proverbs 13:20
|
|
|
|
Curt
Moderator
USA
6652 Posts
Status: offline |
Posted - 07/17/2012 : 12:10:19 AM
|
That reminds me.
I have to test DPM Sql restores.
|
Curt Spanburgh
Microsoft Certified Business Solution Specialist.
Dynamics CRM MVP
Contributing Editor, Windows IT Pro
He that is walking with wise persons will become wise, but he that is having dealings with the stupid ones will fare badly.
Proverbs 13:20
|
|
|
|
wkasdo
Administrator
Netherlands
7405 Posts
Status: offline |
Posted - 07/17/2012 : 06:22:19 AM
|
DR is underrated, even at the enterprise level. One of the things I do is to help enterprise customers build a forest recovery plan, using a production backup. Every so often, we discover that the backup cannot be restored for various reasons:
* different hardware (original server is gone)
* backup corrupt or incomplete
* systems state restore fails because if target OS is wrongly configured
* etc.
So you need:
1. a verified backup
2. prove that you can restore it for the scenario you need (different hardware?)
3. make sure you have that backup somewhere safe, like in two different and secure locations.
It sounds so easy. Apparently it's not. |
Make it as simple as you can, but not simpler -- Albert Einstein |
|
|
|
DennisMCSE
Moderator
Canada
2400 Posts
Status: offline |
Posted - 07/17/2012 : 3:04:55 PM
|
One of the things that helped us was for some of the computers, we used Acronis Backup & Recovery 11 with the Universal Restore option. The Universal Restore allows the restore to different hardware (as long as you have the drivers for the new hardware to inject into the image).
But again, you have to Test, then Test again, then Test another time just to make sure it works.
|
DennisMCSE
Blog: http://itprofirewalker.wordpress.com/
Follow me on Twitter http://twitter.com/Firewalker96
|
|
|
|
aval
Honorable But Hopeless Addict
USA
3276 Posts
Status: offline |
Posted - 07/18/2012 : 9:52:30 PM
|
Besides nightly backups of data to tape, I'll perform an image backup of a Windows 2008 (R2) domain controller and our Windows 2008 Exchange server.
This is weekly, using a set of 3 external hard drives.
I'm confident I could restore to the same server, or identical hardware but reading the above I'm thinking yes, if the entire building is burnt to a crisp (completely destroyed by fire for those not familiar with the expression), I think I could have some problems.
First, even with Acronis or similar 3rd party solutions, how do you deal with potentially different disk layout?
The problem with Windows Full Server Backup is that it's expecting identical disk configuration. No problem if you are restoring to the same equipment (after severe corruption or replacement of a RAID controller or something like that).
One mitigating factor, concerning fire, is that our buildings have a fire detection system linked to a company that monitors the premises so if an alarm goes off, they'll alert the professional (not volunteer) fire department that gets here typically in less than 5 minutes for other 911 calls (people getting sick- nowdays someone faints and you call 911).
Of course, a fire fighter will tell you a fire can spread pretty far in 5 minutes.
So how are you guys backing up Active Directory and Exchange in particular for complete disaster recovery - and potentially to entirely different equipment?
And what about Windows 2003 servers?
For those (and I still have two that "count"), I assumed I would have to 1) rebuild, 2) perform system state restore, 3) restore data from backup tapes.
Having practiced restoring our SQL databases to a different server (for testing purposes), I know it can be done but a complete rebuild from install CD can take time.
And could Cloud backups work for full server recovery? I would think that backup duration - and restore duration - would take forever, even with some sort of differential method built-in.
|
|
|
|
aval
Honorable But Hopeless Addict
USA
3276 Posts
Status: offline |
Posted - 07/18/2012 : 10:05:40 PM
|
Also, while we do store backups tapes and external hard drives in separate locations (different sets in safes in two different buildings at opposite ends of our "campus"), and while I do test the tape backups almost daily (may skip a day), how are you testing your full server backups - if you backup your servers that way?
And how do you determine the data on the backup tape is valid?
I'll perform a restore and if I can open some Word documents (for example) selected at random, I consider the data valid. So I'm not just relying on the "backup completed successfully" message in Event Viewer.
Moreover, in the course of the year, we need to send copies of our SQL based database to the vendor of the software that runs on top of it. I take advantage of this to test our backup by sending them a compressed, encrypted and password protected copy of the database. This copy comes from the previous nights backup tape. Data's never been bad in 10 + years.
So trying to be as objective as possible, I'd say I do a decent job protecting the data itself. As for being able to put everything back together *quickly* after the loss of an entire buidling - and the servers inside... that's another story. |
|
|
| |
Topic |
|
|
|
The reason to have a Disaster Recovery Plan
