All, Technology

Exchange (2003) & Windows (2003) Full Recovery from “Cold” Backup

Having just successfully completed a “worst case” email data restore / recovery from Exchange 2003 backup, I feel it would be useful to pull all the sources of info I used to do this together into one concise guide, for anyone who ever needs to do likewise!

Scenario:

Total loss of Exchange 2003 email server. Recovery required from backup data ONLY.
(Does NOT require; Original server, same hardware, or backups of system state, log files, OS / program files, etc. This process only requires copies / backups of EDB / STM files, even if they are out of date / corrupt / fail to mount).

Process:

1. Allocate an ISOLATED system. (Tape over the network ports!)

Ensure system is NOT connected to your domain at ANY time, you could connect to the Internet directly if you can bypass your network / domain, this guide assumes you can’t, if you EVER allow this isolated recovery system to connect to your live domain you will cause SERIOUS problems with your active directory domain controllers.

2. Install Windows Server & Patches from media.

To same level / service packs etc as original server(s), if you don’t have this data then don’t worry, but do ensure you give the server the same machine name as original email server.
It’s also a good idea to ensure IO systems are optimised (due to the huge workload undertaken by the system during this process) by using latest drivers / performance config for at least the SATA / IDE / SCSI / Disk / Tape bus(es), and the rest of the system where possible.

3. Install the Microsoft Loop Back Adapter.

Download the Loopback adapter from Microsoft
Configure with the same IP address as original email server and set to point to self for DNS, disable all other network adapters.

4. Configure server as a replica of your Active Directory server(s).

Setup Active Directory using the same domain name as the original email server belonged to, and accepting DNS server setup defaults when prompted.

5. Install Exchange & Patches from media.

Ensure you follow the “Pre-install checklist” from the welcome screen of the CD. Alternatively you can use the downloadable tools found here. Also check that its setup with / to the same level / service packs etc as original server(s), if you don’t have this data then it’s not necessarily a show stopper, however if this process fail, you may need to try applying service packs in order and re-attempting below steps each time.

6. Setup Exchange as a replica of your original Exchange setup.

Only “Storage Group” structure / names need to be duplicated (Default is “First Storage Group”), ensure the Storage Group is NOT marked as “this database can be overwritten by a restore”. (“Database” tab of the Mailbox store’s “Properties” window)).
Also Ensure the PATHs for this storage group is set to a drive with enough space!

7. Create a “Recovery Storage Group” to mirror your primary storage group.

Ensure this Recovery Storage Group IS marked as “can be overwritten by a restore”, as above. Ensure the PATHs for this storage group is also set to a drive with enough space!

8. Perform a restore of your email databases from your backup media.

It is strongly recommended that you copy the backup file to the server before attempting a restore. restoring from media directly often times out. For performance purposes it is also highly advantageous to copy the backup source to a separate disk / array from the one you will be restoring to. Only the EDB & STM files are needed in the worst case. If you also have the log files you can use these to roll forward from the backup point. This restore may look like it will overwrite the existing Storage Group, but as long as Step 6 was followed, it will, in fact, restore into your new Recovery Storage Group. Restoring logs can cause additional complexity, so if you aren’t going to roll back / forward then it is often much simpler to only restore the mailstore or public folder store only. To do this navigate down the “tree” in the restore dialog of NTBackup until you see “Mailstore”, “Public Folders”, “Restore Logs” and ensure you select the bare minimum you need to restore for the specific case you are dealing with.

9. Repair your restored databases.

It is always a good idea, and often necessary to “repair” the newly restored database before trying to mount it.
Again, it is highly recommended, for performance reasons, that you repair / defragment the databases from one drive / array to another and copy back as necessary. (Or at least set the temporary db path to a separate disk / array).
To do this;
i) “Eseutil /p {Path to EDB file}” (Repair the databases / remove corruption)
Can take 12 hours +.
ii) “Eseutil /d {Path to EDB file} /t {Path to Temp EDB file}” (Defragment the databases) (This may not be necessary in an urgent situation)
Can take 12 hours + (This is likely to take roughly the same amount of time as step 9.i. above).
You will need 110% of the size of your database (EDB + STM) as free space on the actual drive where this will occur. If you do not have enough space to complete this task on the isolated Exchange server, you can move the database files to an alternate location and run the defragmentation there. To do this, copy the files Eseutil.exe, Ese.dll, Jcb.dll, Exosal.dll, and Exchmem.dll files from the Exchange Server to the remote location along with the restored Exchange Database (EDB & STM).
ii) “isinteg -s servername -fix -test alltests” (This may not be necessary in an urgent situation)
Can take up to 1 hour per run. Keep re-running until ALL results return 0 “Errors” and 0 “Fixes”. If possible, also try to fix any warnings. This usually takes 2-4 runs. Don’t worry if it’s more.

10. Process / “Roll forward” any log files you have / want to use. (Optional)

11. Mount the “Recovery Storage Group” (Via Exchange System Manager).

(You should see all recovered mailboxes now via the standard “Mailboxes” view within this group)

12. EXMERGE against the Recovery Storage Group.

Download EXMERGE from Microsoft
Extract and then open EXMERGE, selecting “Extrac or Import (Two Step Procedure)”, then select the “Recovery Storage Group” and click Next, then in the “Two Step Procedure” dialogue box, select “Step 1: Extract data from and Exchange Server Mailbox”. Fill in your server name and leave the domain name blank. Click Next. In the “Database Selection” dialogue box select the “Recovery Mailbox Store”. Click Next. This will produce an instant error. Don’t worry, that’s a necessary step! Now view the log file created (in the same directory as the executable) and identify / note the msEMGuid of each mailbox you need to recover, which will be text like;
“msExchMailboxGuid A\BC\D1\23\4E\5FG\HI6\J7\K8\91\2L\M3\45\67.” To process this into a usable form follow the steps below;
(i) (Isolate ID) “A\BC\D1\23\4E\5FG\HI6\J7\K8\91\2L\M3\45\67”
(ii) (Replace \ with Space) “A BC D1 23 4E 5FG HI6 J7 K8 91 2L M3 45 67”
(iii) (Convert to 16 “units”) “A BC D1 23 4E 5F G HI 6 J7 K8 91 2L M3 45 67”
(If you have 3 characters together then add a space before the last of the three, if you have 4 characters together then add a space before both the 3rd and 4rth characters (i.e. “ABC” would become “AB C” and ABCD would become “AB C D”.)
(iv) (Convert “lone” characters to hex) “41 BC D1 23 4E 5F 67 HI 36 J7 K8 91 2L M3 45 67”
(http://www.dolcevie.com/js/converter.html is a good simple converter to give you character by character conversion).

13. Create AD accounts (And Exchange mailboxes) for each user whom you’d like to recover.

Passwords do not need to match original passwords, use anything you like. Ensure users mailboxes are actually created by logging into each of their webmail accounts via the the server itself: “http:\\1.1.1.1\exchange” (where 1.1.1.1 is the recovery server’s own IP address).

14. Use ADSIEdit to change each users current “msExchMailboxGuid”

Download ADSIEdit from Microsoft
Extract and open ADSIEdit and navigate down the “Domain” tree until you find the relevant user. Then right click on them and choose “Properties”, finally find and double click on the user’s “msExchMailboxGuid” attribute and update it to match their ID as retrieved from the EXMERGE log file (Step 10, above).)

15. Restart the “MicrosoftExchangeSA” Service.

Make sure that both the “Primary” and “Recovery” mail stores a mounted following this step.

16. Re-run EXMERGE using the same options as Step 10.

Follow step 10, above, until you reach the point where the error is generated, instead of the error / after the error, continue by selecting the individual mailboxes you want to recover or press the “Select All” button to select all mailboxes found in the store. Click Next. On the “Locale Selection” dialogue box select the Locale that you would like to use to search the mailboxes.
Click Next. Specify the path to the folder where you want to place the .PST files. Click Next. Allow the EXMERGE to complete, timescales for this are entirely dependant on size and number of mailboxes being restored, allow at least 30 minutes per mailbox, and once complete you will have a fully portable PST of each selected users full mailbox!

17. Breath a big sigh of relief, pat yourself on the back, explain how difficult and involved the process was to anyone who will listen and buy yourself a donut (replace with preferred treat if donut not applicable).

(Please don’t forget to leave me a comment concerning the bits I no doubt missed!)

2 thoughts on “Exchange (2003) & Windows (2003) Full Recovery from “Cold” Backup

  1. Just glad I could help someone. 🙂
    The reason I posted this was because I had such trouble with the existing guides out there on the web.

Leave a Reply

Your email address will not be published. Required fields are marked *

This site uses Akismet to reduce spam. Learn how your comment data is processed.