A day before yesterday, I was asked if an Exchange Server 2010 mailbox server which is a part of a DAG can be recovered using disaster recovery setup options.
Due to considerable changes in the ways Exchange 2010 handles the clustering components it is little tricky to recover a crashed server that was a part of some DAG in an exchange organization. In Exchange 2007 you could run setup.com /m:recovercms to recover a clustered mailbox server node. But, in exchange server 2010 you have to do it a little different way. Let us take a look at how to do this. It is highly recommended that you make a note of your database copy deployment layout before you go ahead.
Let us say you have a 2 member DAG with a database:
DAG Name: DAG01
Database: Mailbox Database
In my case the server E2K10SP102 crashed and the OS could not be recovered at all.
1. Remove Database Copies Configuration
When you add a member into DAG and deploy database copies across the members, this information is stored in active directory. Since this information is not removed automatically when either of the members a DAG crash or encounter an irrecoverable problem we need to clean this information first. During the cleanup process you essentially remove the information of the database copy that was configured to be peer of the server that is no longer available.
To perform the cleanup of the database copy run below command using EMS
Remove-MailboxDatabaseCopy –Identity “Mailbox DatabaseE2K10SP102”
Right after you hit enter you will be prompted to confirm your action and you might also see your Management Shell screen filled up with warning messages. This is totally okay. In ideal situations, the server will try to clean up this information from another member’s local configurations as well. Since your other server is no longer available the local configuration data from other node will not be cleaned up. This will delete the database copy information from AD though. You need to repeat this for every database copy if you have more than one.
2. Remove DAG Member
Next step is to remove the crashed DAG member from the configuration.
Remove-DatabaseAvailabilityGroupServer –Identity DAG01 –MailboxServer E2K10SP102 –ConfigurationOnly
ConfigurationOnly is very important here. If you miss this parameter in the command the server will try to contact the other node to remove the cluster service configuration and will fail since the other server is already gone. Keep in mind that we are removing all of this information from AD and not the other node so this parameter should not be missed.
3. Remove Cluster Service Configurations
As we all know, although DAG do not use failover clustering mechanism; it still uses MSCS failover clustering components. With said that there are some cluster service configurations in service control database and quorum needing cleanup before the crashed server could be brought back into the DAG. To clean up cluster service configurations open command prompt as a privileged user if you have UAC enabled on the server and simply type below command:
cluster.exe DAG01 /Node E2K10SP102 /Evict
Once you have completed the cleanup. Next steps are to rebuild the server, add it back to DAG and configure database copies.
4. Rebuild the Server– If you have completed fixing your hardware issues (If at all the server had any 🙂 ), it is time to install the OS similar to what patch levels and drivers what you have on the currently working node.
5. Reset the computer account – After your server OS is built correctly and you have made sure the path level is matching to the currently working box, reset the computer account of this server in AD and join the computer with same name in the domain.
6. Install Software Prerequisites for Exchange 2010 -If you had more than one server roles installed on the server you should install all applicable prerequisites for each server role.
7. Recover the Server – Insert Exchange 2010 RTM / SP1 disk (depending upon what release of Exchange you are running) and install exchange server binaries using command line setup. Run setup.com /m:recoverserver from command prompt.
8. Add the server to DAG – After the installation completes, reboot the server once and add the server in DAG using below cmdlet
Add-DatabaseAvailabilityGroupServer –Identity DAG01 –MailboxServer E2K10SP102
This will initiate the cluster service configuration on local as well as remote computer.
9. Add Mailbox Database Copies – If you have reached this step that means you are almost done and need to execute following command to database copy configuration completed.
Add-MailboxDatabaseCopy –Identity “Mailbox Database” –MailboxServer E2K10SP102 –ActivationPreference :2
Done! You just need to monitor how the replication of this copy is going; using Get-MailboxDatabaseCopyStatus –Identity “Mailbox Database”
Hope this information helps!