Evaluating DR Criteria - Part 5

Part V: DR Testing and Drills

This is part 5 in a series of articles on understanding and addressing Disaster Recovery criteria. In this article we will discuss the often overlooked and neglected requirements and needs for DR drill and test.

Part 1 was an introduction and overview.

Part 2 covered the often overlooked personnel plan and site location.

Part 3 explained the importance of application categorization to drive the correct RPO and RTO needs and the resulting impact to economies.

Part 4 raised the question of a converged DR and backup solution and emphasized the importance of a product that is automatically adaptable to changes in production.

Similar to other criteria in previous articles, thinking ahead about your testing and drill needs and drivers can not only save you a lot of time and money but is critical to a successful failover during a real DR event.

Modern technology allows DR deployments and steady state operations that are far more economical than bygone years. That extends to DR testing and drills. Obviously testing your strategy is important. It's also important to test it with a modicum of frequency. It's interesting that regardless of how many times a DR drill was conducted there is always something to update or refine in the runbook. And production operations change, servers are reconfigured, added, deleted. With regular testing, oversights and changes are highlighted and can be corrected prior to an actual event.

Before we delve into some specifics, we should address what is Disaster Recovery and what it is not.

Difference Between DR and Backup

Disaster Recovery and Backup are two very different functions. Backup is more about having copies of your data, usually with some retention policy, that can be available when needed. Single file restore is an assumed part of backup with fairly rapid recovery from a recent backup and longer recovery from an archive.

Disaster Recovery may have backup features, but it's intended to restore servers, complete environments, and datacenters in their entirety to a completely functional state to resume all production operations.

Backup without any DR semantics will not restore production operations in any reasonable timeframe in terms of meeting business needs, especially in the modern world. If you have a backup solution without DR take the time to restore an important set of applications and see how long it takes. For an Enterprise environment restoring from backup alone will take 3 weeks to 3 months best case.

Difference Between High Availability and DR

Another common mistake is confusing High Availability and DR. High Availability protects specific mission critical applications from a localized failure typically using application specific mechanisms. HA recovery is also typically deployed in the same datacenter or environment as the primary application.

This is a very different function than recovering a wide scope of applications and servers in a completely different geography.

Another example to illustrate the difference between HA and DR is that a properly designed DR plan will include replication of the entire HA function to the target environment. This is different than using the HA function as your DR plan. The initial recovery plan may not include enabling HA in the DR site, but it would be included as a later phase depending on how long the outage will be.

Difference Between Test and Drill

I'll grant the differences between Drill and Test are nuanced, and not necessarily industry standard, but understanding the purpose, logistics, and evaluation is important.

Generically, a test basically asks the question will it work. Testing typically also has a more relaxed atmosphere. In contrast, drills are more about practice and training so you can replicate the execution of a failover and bring up.

A DR drill, typically, is also a more comprehensive exercise that simulates an entire datacenter completely going offline for a period of time. The DR drill can still be non-disruptive, meaning all the drill operations are isolated from production in the origin environment. But as far as the drill exercise goes it assumes the origin production is completely gone or at least unavailable.

Of course there are gradations, you can do a DR drill for a specific set of applications and not the entire datacenter.

Applying Application Categorization and Prioritization to Testing and Drill

Feeding into your DR test and drill strategy and resulting plan and implementation is your application categorization and prioritization analysis (see part 3).

DR test and drill necessarily involves phases, with layers coming up and being available in proper sequence. It's not practical, and most importantly, not reproducibly consistent, to bring everything up all at once. Booting servers is a highly intensive operation for all resources in a datacenter, storage, networking, and compute. Without strategically defined phases in your bring up, it's likely your simple, less important, applications will become available first as they typically are smaller and consume fewer resources during boot and bring up.

A well designed plan will have an Infrastructure phase as layer 1. For large environments there may even be second and even third layers/phases. For example, you may have a licensing server in layer/phase 2 that requires Active Directory to be available from layer/phase 1.

Tiered applications will have 2 or 3 tiers. For applications to boot with the least intervention usually require the database server to be available when it boots. Sure, if the app boots first, you can intervene and make it work, but that breaks the automation and requires diagnostic, debug, and manual intervention.

Be Realistic About Recovery Times

I sometimes hear people declare they need a Recovery Time Objective (RTO) of 5 minutes. That may be your High Availability objective, but it's not a practical DR objective.

Consider the layers/phases discussed in the previous section. It may take 10 to 15 minutes, minimum, for your infrastructure layers to be not only booted but verified. You should have some level tests prepared or automated for each layer, applications, and functions. It may be fast, but it will still take some number of minutes.

For tiered applications, large databases take 10 to 15 minutes (or more) to boot properly and initialize before accepting connectivity to applications. After that, the app tier becomes available with its own boot time and initialization. Of course, the app tier will be faster than the DB tier, but the minutes add up. And many tiered applications will have a third web tier with its own bring up metrics.

There may be 5 or more layers/phases to be booted, initialized and available for work before that app can accept production transactions.

And consider application prioritization. Your mission critical applications will be first followed by any number of lower categorizations.

Final Thoughts

A lot more could be written about DR testing and drills though I try to address important criteria that are often overlooked and avoid the topics already discussed ad nauseum on the Internet.

In addition to understanding your plan for DR testing and drills it's equally important to understand how your DR testing and drills needs affect your overall DR plan and selection of site and product.

Product selection should include the ease of use of testing in a non-disruptive manner but also minimize costs for each iteration of a test. Making it easy to test and drill frequently and without a great deal of extra cost will ensure a well exercised plan and an ability to quickly and effectively resume operations.