A successful migration project is not just about moving data from one location to another, but about making sure that everything goes as expected. This can be challenging for even the most sophisticated IT pro. In fact, an independent study found that more than 80% of data migration projects exceed their timelines, go over budget, or fail to meet their goals.

Through long experience and thousands of migration projects, NetApp Global Services (NGS) has established a variety of standard methodologies to ensure projects involving data migration are completed on budget and on time. (For a complete description, see the NGS Data Migration Methodology.)

Rather than rehash information that is available elsewhere, this article uses the example of a large-scale migration project in which the customer brought in NGS to help migrate 57TB of engineering data as part of a massive storage technology refresh. To complicate matters, the customer allowed planned downtime only one weekend a year, and the data migration was just one element of a move to a new data center. (See sidebar for details.)

The result? The six-month project was completed on time and on budget and has been deemed a complete success by the customer.

There were many factors that contributed to the success of this effort, including the customer's decision to involve NGS in the initial planning phases of the project. In this article, I'll share three tips to help ensure that your next migration project runs smoothly:

• Make sure you choose the right tools for the job
• Test, test, test
• Be prepared to reevaluate timelines as new information becomes available

Tip 1: Choose the Right Tool for the Job

Automated tools can make a huge difference in the amount of time, resources, and pain associated with a migration, but nothing takes the place of careful, upfront planning and a thorough discovery process. That said, you should choose your tools carefully and always consider capabilities versus costs.

There are two main migration phases in which tools can be useful:

• Discovery. This process can be manual (using command-line options to gather information to complete a checklist) or automated (via a discovery tool). A standard NGS best practice is to use automated discovery tools whenever possible because they help minimize the chance of error.
  
  
• Migration. Information collected during discovery helps guide the selection of tools to physically move data from the source (existing location) to the target (new location).

Unfortunately, no single tool (despite any marketing claims to the contrary) does everything well. Although you'll find that some commercial tools have both discovery and migration capabilities, in our experience, it is extremely unlikely that any one tool will meet all your needs for either task (let alone both!). To complete a complex migration, you will almost certainly use a variety of "tools," including checklists, spreadsheets, operating system commands, tools provided as part of your business applications, and storage system utilities, in addition to—or instead of—commercial tools.

You should also be aware that new commercial migration tools are emerging on a regular basis, and old ones are being enhanced. When NGS first started planning the migration project mentioned above, for example, we had to perform all discovery manually because none of the tools available at the time met our requirements.

If we were starting this project today, there are several automated tools which do a particularly good job finding and classifying data. I might also use ISI Snapshot to inventory the hosts and hardware, which would have saved two to three days of effort spent on manual data gathering.

Because the migration required organizing data below the qtree level, we relied on NetApp NDMPcopy as the primary migration tool. Typically we use SnapMirror® for migration between NetApp systems because it allows very frequent mirroring, has a minimal impact on the infrastructure, and can be throttled to overcome bandwidth or CPU limitations. However, SnapMirror isn't an option below the qtree level. An added advantage of NDMPcopy is that it allows scripting, which saved time managing the customer's 811 mountpoints.

If you are moving more diverse data types, be prepared to use different tools to handle the migration of the different types (for example, NFS/UNIX®, CIFS/Windows®, databases and business applications, and block-oriented SAN data).

In addition, your choice of tools will be driven by your goals and requirements. For example, if you have a situation in which you have minimal or no allowable downtime to cut over to the new storage, NeoPath File Director can be helpful. For migrations from UNIX to NetApp, NGS consultants often use simple, host-based tools such as rsync or rdist. If part of a CIFS deployment includes implementing a global namespace, you may be able to use VFM™ for the implementation and then seamlessly do the migration under the covers.

Path lengths may also be a consideration in migration tool selection. Many Windows tools (for example, Explorer, CACLS) have a path length limitation of 256 characters. Since paths on NetApp systems can easily exceed that, if you want to use a Windows tool, you might look at the Windows 2003 version of Robocopy (part of the Windows Resource Kit) which does not suffer from the 256 character path limitation.

Depending on your situation, this is probably either more than you want to know or not nearly enough. Because of the plethora of tools available and trade-offs associated with each, if you are planning a complex migration that involves NetApp storage, I recommend having NGS help you identify the best tools for your situation.

Tip 2: Test, Test, Test

Even if you've carefully chosen your toolset, nothing takes the place of testing to increase confidence and ensure a smooth migration. Often when something goes wrong during a migration, it could have been identified with upfront testing.

The standard NGS testing methodology includes:

• Verifying the infrastructure. You don't want to hunt down and fix network problems, such as duplex and jumbo frame mismatches or IP conflicts, after the migration begins. Test your infrastructure before you do any other testing. Make sure you are able to achieve the read and write performance you would expect.
  
  
• Testing the migration tool(s) with real data. Based on discovery, NGS typically identifies a representative sample of the data and tests the migration tool(s) on it to ensure that everything works as expected. In addition, the test will provide a good estimate of how long the migration will take. Test data selection criteria include data type (NFS, CIFS, block, and so on), complexity of permissions and ACLs, depth of directory structure, and so on.
  
  
• Executing a complete dry run up to the cutover point. A dry run is the best way to identify all potential problems and find out exactly how long a data migration will take. This is highly recommended but optional; a dry run is not always feasible due to a lack of timing or resources.
  
  
• Validating the completed migration. Before the migration project can be considered a success, someone must validate the post-migration environment and confirm that all expectations have been met. At a minimum, network access, file permissions, directory structure, and database/applications need to be tested. Only the owner of the data can truly validate that the migrated data is correct and usable, so ideally, all owners should participate in the validation.

In the project mentioned above, data collection began a full six months prior to the scheduled cutover date. The new storage systems were then installed, and testing began. We almost immediately identified network bandwidth as a showstopper issue and had to install a separate network to accommodate migration traffic without affecting day-to-day operations. Three months before cutover, we did a complete dry run, which told us the live data migration would take 13 days to complete. Doing the dry run so far in advance gave us enough time to do the test again if things didn't go as expected. After the migration was completed, validation was performed by a dedicated customer team.

Even if you're not facing a fixed time window, insufficient testing can be costly. NGS was recently brought in to troubleshoot an issue with a large aerospace company that inadvertently idled 7,000 users for 36 hours when the chosen migration tool encountered a previously unknown bug. This problem would have been identified with proper testing.

Tip 3: Reevaluate Timelines as New Information Becomes Available

Customers who have experience with small migrations are often surprised that multiterabyte migrations designed to minimally impact ongoing operations take many days or even weeks to complete. The planning process alone can easily span months. Even if every other aspect of your migration goes perfectly, the migration won't be considered successful if the timeline isn't accurate or people don't know and buy into it. The results of pre-migration testing should be constantly reviewed and used to finalize your project timeline.

In the project referenced above, timelines were so tight that we only had a single weekend to complete the data migration and cutover to the new storage. By doing exhaustive upfront testing and monitoring data changes after our tests completed, we identified a four-hour start time window in which we had to commence the actual data movement. If we had started sooner, the data movement would have completed too soon (before work activity ceased). If we started later, the data movement would not have completed in time to do all the things we needed to accomplish during the 48-hour downtime window. (Storage systems were being moved to a new data center, so they had to be packaged, moved, and reinstalled.)

Conclusion

Complex data migrations are rarely executed without a certain amount of stress, but it is possible to accomplish them on time and with minimal business disruption. Careful attention to tool selection can eliminate a lot of potential problems. Testing validates your tools and methods, helps you identify potential problems before they affect the migration, and allows you to better understand your migration timeline.

If you need help, NetApp Global Services has years of experience with data migrations at businesses of all sizes. We are structured to provide help where and when you need it. We can do the whole migration, help with the upfront planning to ensure you make the right planning decisions, or just complement your in-house expertise.

---