How to repair a corrupted MS Exchange database

It's almost disaster when your email database shows corruption and you have to pull out all stops to get it back up again. When we faced the problem last week with our Microsoft Exchange 2003 email server there were a lot of lessons to be learnt.

Symptoms

One of the logical disks on the Exchange started running out of space. When we analysed the situation, we found that the exchange transaction logs were not getting cleared. We checked event logs in the exchange server and found the following ominous looking error log.

Event ID: 474
Category: Logging/Recovery
Source: ESE
Description: Information Store (1416) Second Storage Group: The database page read from the file 
"K:\EXCHSRVR\Second Storage Group\Priv2.edb" at offset 9591296000 (0x000000023baf9000) (database page
2341624 (0x23BAF8)) for 4096 (0x00001000) bytes failed verification due to a page checksum mismatch. 
The expected checksum was 9094876751199153140 (0x7e377e37ef2f2bf4) and the actual checksum was 
3108421473592972935 (0x2b2354dc71b49687).  The read operation will fail with error -1018 (0xfffffc06).
If this condition persists then please restore the database from a previous backup.  This problem is
likely due to faulty hardware. Please contact your hardware vendor for further assistance diagnosing
the problem. 

For more information, click http://www.microsoft.com/contentredirect.asp.

For more information, see Help and Support Center at http://go.microsoft.com/fwlink/events.asp.

Microsoft has covered this error (the -1018) in much detail here.

Diagnosis

Based on the assessment from our SAN vendor after running the system report, we found out that the one of the physical disks in our SAN had gone bad and was showing bad blocks. Also because of this particular error, the exchange backups were failing and the transaction logs were not getting cleared after backups.

Solution

First thing we did was to get the damaged disk offline and use up the spare disk in the SAN. We contacted Microsoft for this issue and there were two options facing us. As the error log above clearly says it is a hardware issue, Microsoft suggested that we first check the database consistency by running a backup after the disk was replaced. As expected, the error was more integral to the database. The database was corrupted. We were faced with only one options at that time. Bring the email servers down and run the exchange database repair utility (ESEUTIL) to do a hard repair of the database, defragment the database, check the integrity and then hope for the best. This is because the repair process "repairs" the database by dropping the faulty pages (called as leafs) from the database. Which in lay man's terms means that there is a possibility of 0 to 100% of data loss.

We took the down time and took 3 copies of backups:

• Exmerge extract of every mail store using the Exmerge utility.
  
• An offline flat file backup on tape using Veritas Netbackup
• An offline flat file backup using NTBackup.

Once the backup was completed, we ran the repair utilities. The syntax for the utilities is:

REPAIR
C:\Program files\Exchsrvr\bin>eseutil /p "c:\program files\exchsrvr\mdbdata\priv1.edb"
Note: The absolute path is needed for both the eseutil and the database. You could also run the utility
from the database location only difference would be that you would have to give the absolute path of the
"eseutil" utility instead of the database

DEFRAGMENTATION
C:\Program files\Exchsrvr\bin>eseutil /d "c:\program files\exchsrvr\mdbdata\priv1.edb"

It is the same utility but with a different switch.

DATA INTEGRITY
C:\Program Files\Exchsrvr\bin>isinteg -s <servername> -fix -test alltests

This utility checks the data integrity. Microsoft recommends that you run the isinteg utility multiple 
times in case there are fixes in integrity check process till the time the utility reports no fixes done.

Once the process was completed, I was surprised to see the size of one of the databases reduce from greater than 35Gb to 286Mb. We ran the backups after the repair was done and thankfully the backups completed successfully and it was truncating the logs as expected. We also checked the mailbox sizes and items counts after the activity was done comparing it with the same parameters before the activity and we did not find any differences in the mailbox sizes or item counts.

Conclusion

Thankfully we did not have to restore any of the mailbox from the backups we had taken. But taking a restorable backup is a very essential insurance policy every one should keep before such activities are performed. This also helped us identify the checks and balances we need to put in place to ensure that the systems are humming away in a healthy state. Now I check the transaction logs every day for correct truncating and event logs for errors (I know this should be done by default). So far so good.