NetApp Storage Arrays:
A Viable Option for Exchange Server?
Avanade is a leading systems integrator that focuses on the Microsoft enterprise platform. Recently, Avanade undertook a comprehensive series of tests to validate the capabilities of NetApp storage systems for use in Microsoft Exchange environments. Tech OnTap caught up with Avanade Chief Architect Patrick Cimprich to learn more.
Q: Before we get into specifics about test setup and results, could you explain at a high level why storage is such a critical element of an Exchange infrastructure?
The way Exchange Server treats storage is atypical of any other application that's out there. Due to the small size of Exchange I/O transactions and the large number of them, Exchange is a poor storage citizen and back-end storage has to work very hard to keep up. The sizing and scaling limiter of any Exchange implementation has historically been storage, making it the key element of an Exchange deployment.
Q: What were the main motivations that drove Avanade to test NetApp storage with Exchange?
Over the years, Avanade has encountered NetApp a number of times in various projects. Recently, we’ve begun seeing NetApp storage more and more frequently, and our customers have been asking for our guidance on NetApp solutions.
We thought it was time to take another look at NetApp technologies, which have evolved significantly over the last few years, so that we would be better equipped to respond to user requests.
In our Avanade Test Center we developed a rigorous test plan and worked with NetApp to evaluate its products to test how they perform with Microsoft Exchange Server 2003.
Q: We understand you were skeptical about NetApp at the outset. Why was that?
To be frank, in the past, there were many within Avanade who kept NetApp at arm’s length. This was largely due to the fact that our early experiences with NetApp occurred before NetApp offered Fibre Channel SAN and iSCSI. Trying to support Windows applications that require block I/O on a NAS platform was difficult at best.
When we started seeing more and more demand for NetApp from customers, we still thought of NetApp mostly as a NAS company. We didn’t know if its SAN products were really capable of meeting the punishing I/O demands of Exchange Server 2003.
And, to be completely honest, my own attitude toward NetApp was lukewarm at best. I have over a decade of experience with various storage technologies and a very comfortable understanding of EMC technology in particular. Given how different NetApp is from other storage technologies, I wasn’t sure how the tests would turn out. I definitely came in with a healthy dose of skepticism, and I didn’t expect this to be a complete success.
Q: And are you still skeptical?
No. Definitely not.
From a performance perspective, most of the results we saw were in line with similar configurations from other vendors. Latency was in an acceptable range, with disk reads and writes well below Exchange’s 20-millisecond threshold in all of the configurations we tested.
In addition, NetApp has a great value-add in its ability to create and store multiple Snapshot™ copies without any performance hit. This was definitely one of the coolest things I’ve learned in this experience; NetApp Snapshot copies are great. Overall, I found NetApp storage much easier to learn and manage than any other storage environment I’ve worked with in the past 10 years.
Q: Let’s talk more about the testing process and how a former skeptic became a believer in NetApp SAN solutions. Can you give us an overview of the tests you conducted?
One of the guiding principles that I established when we went into these tests is that whatever we test must be supported, “normal” configurations. We had to choose configurations that we could pick up the phone and call the vendors involved and get support for. That meant no beta or alpha code, only general availability hardware and software, and all configurations had to be on the Microsoft hardware compatibility list. In other words, we did exactly what most customers would do. We picked configurations that are supported and certified and that reflect what we’re designing for customers out in the real world.
Avanade Testing Center Eval: NetApp Storage Arrays
This 10-page white paper details the benchmarks and results of Avanade's in-depth lab study.
At a macro level, the testing that we did was to establish how NetApp performed as a storage target for Exchange Server 2003. To do that we ran through a whole host of tests, starting with a basic load test using iometer. This gave us a baseline and helped us ensure that we knew what we were doing since we were relatively new to NetApp.
Then we moved into a series of Exchange Server 2003 tests using Microsoft supplied tools, each targeted at different levels of the Exchange Server 2003 stack.
We looked at disk performance and how user response times correlated to disk latency in terms of reads and writes, how the data was read across NetApp volumes, and how the entire system reacted to load.
Our testing was executed over the course of nine months. Overall, there were on the order of 115 million e-mails sent and 15 to 20 billion IOPs. We did very thorough testing and collected performance data throughout, as well as all the analysis required to understand what the data meant.
We tested across a number of different server platforms—blades and traditional servers—and we also tested a variety of iSCSI transports (iSCSI cards, TOEs, built-in NICs, HBAs, and so on) and Fibre Channel SAN configurations.
Q: Could you talk about the first Exchange-specific test using Jetstress?
The first Exchange-specific test we ran used the Microsoft Jetstress utility. This test directly simulates Exchange patterned disk I/O. It’s specifically designed to ensure that any storage subsystem you are planning to put in place for Exchange Server 2003 will meet the expected I/O demand, which in this case was 6,000+ IOPs.
We configured the NetApp storage system and our Exchange servers exactly as we would use them in a customer deployment. We used 64 drives for Exchange data and logs and configured four Exchange storage groups with five databases per group. Logs and databases were in separate aggregates. A separate LUN of 384GB was created for each storage group plus a corresponding 115GB LUN for logs.
Once again, the results of this test were in line with similar configurations from other vendors. The key number in this test is latency. Twenty milliseconds is considered the ceiling. If latency exceeds 20 milliseconds, Exchange begins to fail. All FAS3050 configurations exceeded 6,000 IOPs and passed with an average response of 15.5 milliseconds, a very good result. [Ed note: This refers to reads.]
Q: What about the next Exchange-specific test using LoadSim?
Walking up the stack from an Exchange point of view, this test exercises the next layer of functionality in an Exchange system and tests the whole configuration from end to end. This test was executed using the same configuration as the Jetstress test with Exchange loaded on the servers. A simulation tool from Microsoft called LoadSim was used to actually transport mail messages within the Exchange system. In this test we’re looking at the corresponding disk load on the tail end while keeping in mind the user response time.
LoadSim creates a score after weighing a whole series of factors such as how long it takes to send a message, how long it takes to read from disk, and how long it takes to write to disk. The scores are measured in milliseconds. Microsoft considers a score of 1,000 milliseconds (1 second) a passing score.
All NetApp scores were below 500 (half a second). These represent outstanding performance and once again are consistent with other vendors. Disk reads and writes were well below the 20-millisecond threshold in all cases.
Q: Did you test NetApp capabilities for backup and recovery?
Yes. We specifically looked at storage-based backup and recovery for Exchange Server 2003—for NetApp that means Snapshot copies. We performed three backups a day under load: one at 7 a.m., one at noon, and one at 7 p.m.—a realistic scenario. We ran this test over a period of five days. When we started there were no saved backups. Over the course of the test, we added backups until we got to the point where there were seven backups. Once we got to Snapshot number eight, we rolled off the earliest backup and maintained a set of seven rolling backups for the remainder of the test.
If you’re interested in more specifics, check out the white paper.
Throughout the entire duration of the test we measured end user response time, storage latency, disk latency, and so on. We saw no performance impact as additional Snapshot copies were added. Database latency stays flat at four milliseconds (20 milliseconds is the ceiling), and client latency averaged 167 milliseconds (1,000 milliseconds is considered the ceiling).
These results are outstanding and I believe are a significant differentiator for NetApp. Other storage vendors can typically only achieve similar results using full copies or clones of the disk, not snapshots. Full copies are obviously much more expensive in terms of disk space and the cost of maintaining multiple backups. If customers are new to NetApp, I believe you should definitely take the time to study and understand how NetApp Snapshot technology works versus snapshots from other vendors.
Q: What about SnapManager® for Exchange and Single Mailbox Recovery?
We leveraged SnapManager for Exchange as the execution tool to coordinate and configure all the Snapshot copies that we did in our backup and recovery tests. I think that was another highlight of the testing. The ease of installation, the ease of use, and the fact that it just worked and wasn’t complex were all clear advantages. The tool is targeted specifically at Snapshot backups for Exchange. Quite frankly, it was a pleasure to work with relative to other tools I’ve used.
We did not use Single Mailbox Recovery as part of this test, but I do have experience with it. I think this is another highlight for NetApp in the context of Exchange. A lot of vendors provide similar functionality, but the integration that NetApp provides again makes the tools simple, direct, straightforward, and easy to use versus single mailbox recovery tools from some other vendors.
Q: Disaster recovery can also be a critical element of Exchange infrastructures. Did you look at NetApp DR capabilities?
Not in these tests. That was beyond the scope of this evaluation.
However, I recently did a frame swap for the NetApp system I have in the lab. I used SnapMirror® to do the whole migration. I was able to migrate 10TB from one system to another in about 16 hours of execution. Three-quarters of that time was spent waiting for data to replicate. This was the first time I’d used SnapMirror, and I’d never done this type of migration on NetApp. I accomplished the whole thing with very little planning. I have years of experience with other storage platforms, and there’s no way I would even have attempted this with any other storage array.
Q: How does NetApp storage compare with other storage systems you’ve configured for Exchange?
NetApp performed on par with other storage vendors. When it comes to Exchange, the most critical factor is the number of spindles, so I wouldn’t expect any storage system to differentiate itself with significantly greater performance. Cache is always swamped; it does not matter. Barring a serious bottleneck to disk I/O in an implementation, it boils down to a wash from a performance perspective. So, the important question becomes “how easy is the system to install and manage?”
Implementation is where you really see the NetApp difference. 
It’s in the way NetApp goes about combining the power of those disk drives in terms of virtualization and administration. The NetApp approach to Snapshot copies and Snapshot integration with applications is better than other storage vendors I’ve used. Speed of configuration, ease of configuration, day-to-day operations, and management give NetApp a leg up on the competition.
It took roughly 10 steps to configure NetApp for our 6,000-user Exchange configuration. It was so straightforward, I was able to train one of our interns to assist in the execution of these tests. He’s a bright guy, but I would have been very nervous asking him to manage other storage platforms.
Q: One question that comes up over and over is whether customers should choose Fibre Channel or iSCSI as the storage infrastructure with Exchange and other applications that require block-oriented storage. Did the Avanade Test Center review shed light on that question?
Apart from the NetApp performance, this in my opinion was one of the most interesting results we got from the test. I was an iSCSI skeptic going in, thinking that it wouldn’t be able to perform as well as Fibre Channel. After billions of IOPs and millions of e-mails there was zero difference, and sometimes iSCSI actually beat Fibre Channel. Granted, when iSCSI “beat” Fibre Channel in these tests, it was really rounding error, but the point is for these tests with our Exchange load, there was no substantive difference between Fibre Channel and iSCSI. Based on this assessment, I am very comfortable encouraging customers to consider iSCSI as a viable solution for enterprise-class configurations.
Again, there’s a lot more detail about these tests and the results in the white paper. Additionally, I recently spoke with ComputerWorld about 10 Gigabit Ethernet and iSCSI. There’s a lot of interest out there, and I think we’ll all be hearing a lot more about this.
Q: Is there anything else you learned from this testing that you think would interest Tech OnTap readers?
Overall, I think it’s really telling how NetApp evolution as a company translates into less complex hardware and software. NetApp started out providing solutions for smaller requirements—departmental needs and smaller divisions—and has grown up into the data center. The way NetApp approaches hardware and software is very much geared toward less complexity and less operational overhead. In the Avanade Test Center we found it to be easier to configure, easier to set up, and there’s a single operating system across the entire line.
Some competitors offer management products that have a broader scope. They cover not just Exchange, but SQL Server, file service, and so on in a single product. This strategy does have advantages, but it’s more difficult to install and maintain. NetApp gives you a set of products targeted to particular applications that are easier to configure and maintain and give you the functionality you care about.
Want to learn more about Avanade’s testing? View the full test report.
