Sentara serves approximately two million patients across Virginia and North Carolina. In most cases, we are focusing on eliminating paper records and keeping everything from medical histories to patient exams in and outside the hospital online. This dramatically improves care efficiency, but has brought about a paradigm shift in how the hospital–and everyone in our organization–views IT.

At Sentara, the cost of downtime is no longer measured in dollars per minute but in potential lives lost.

Even though downtime procedures are a part of the clinical workflow, as patient care providers become more dependent on technology the requirements for nondisruptive technology are imperative. Our internal threshold used to be that no patient care system could be down for more than four hours. In reality, we are finding that threshold really needs to be minutes (or much less!) because every bit of patient information is electronic. The customer mindset has changed from some downtime being acceptable to any downtime, either planned or unplanned, being completely unacceptable.

Overview: eCare Initiative Expansion Drives Storage Evolution

In 2005, we launched the Sentara eCare™ Health Network, a comprehensive electronic medical record (EMR) system linking clinical information with scheduling, billing, and registration data over a secure network. This technology allows the secure sharing of patient information between hospitals, physician offices, diagnostic centers, and patients' homes. When fully implemented, the system will enable patients to book their own appointments from home and hospitals to transfer full medical records to other states or countries as needed. It has also facilitated smart prescription centers that automatically check medication allergies and conflicts and can even make recommendations about potential treatments.

As part of the eCare initiative, our team originally implemented a pair of HP StorageWorks XP12000 disk arrays to support UNIX®-based hosts. However, as the implementation expanded we needed to deliver the same class of availability to Windows® hosts while simplifying provisioning and other storage-management processes.

After a thorough review of options, our team chose NetApp storage virtualization technology. This ended up being only the first in a three stage deployment that has helped revolutionize our storage environment:

• Stage #1: Virtualize existing storage arrays for multi-OS and multi-protocol access
• Stage #2: Maximize data availability
• Stage #3: Consolidate servers

As a result, we've not only streamlined backups and significantly expanded our environment without increasing costs but–most important–we have achieved our 99.99% uptime goal.

Stage #1: Virtualize Existing Storage with NetApp V-Series

Every dollar spent on IT is a dollar that can't be spent on new types of medical training or technology. Any IT investment is therefore under extreme scrutiny and must provide a maximum return. We chose XP to support mission-critical solutions on HP Superdome Servers, but required a more flexible solution for the Windows environments. In 2005, we were in a situation in which our XP arrays were supporting our UNIX environment, but we had no budget to build a parallel infrastructure and no simple path to expand this environment to support Windows systems.

The Sentara SAN team initiated an evaluation of the NetApp V-Series system. (see sidebar) Basically V-Series provides virtualization for pre-existing storage arrays by taking functions that would happen either on the host or the array and pushing those functions to the virtualization layer.

We were blown away by the early results, and extended our original 90-day proof-of-concept operation to a 120-day in-depth evaluation of the NetApp storage virtualization solution.

Ultimately, there were three key reasons we chose the NetApp solution:

1. Ability to present (via FCP and iSCSI) the existing XP12000 arrays as storage to our Windows hosts while simplifying provisioning and enhancing data protection
2. Ability to streamline backups and provide rapid restores via integrated tools
3. Flexible ways to replicate data via existing IP infrastructure

We deployed clustered V3050 controllers and connected them to our existing HP StorageWorks XP12000 disk arrays. The benefits of this deployment have been considerable.

First, storage virtualization gives us the full range of NetApp capabilities–including Snapshot™, FlexVol®, and FlexClone®–in conjunction with our existing storage arrays. With a relatively modest additional investment, we were able to leverage our existing infrastructure to not only support our Windows environment but also gain substantial flexibility.

Multi-protocol support has been a substantial advantage. While our primary Microsoft® SQL Server™ applications are on a Fibre Channel SAN, the same NetApp V-Series clustered system also provides NAS connectivity using the CIFS protocol to a repository of some 30 million TIFF images in our OnBase document management system from Hyland Software.

Additionally, if we connect a host running Microsoft SQL Server directly to a storage array, we would typically have to take a database offline to do backup. With our V-Series solution, we use NetApp SnapManager® for SQL Server, which coordinates activities between the host and storage to create consistent, point-in-time Snapshot copies of a database that can then be backed up without significantly interfering with ongoing database activity.

We also utilize a capability called NetApp FlexClone to clone existing databases for test and development and a variety of other purposes. FlexClone clones can be created in seconds and only consume additional storage space as changes are made to the cloned volumes, so they are both time and space efficient compared to traditional cloning methods.

The FlexVol volumes that the V-Series presents to hosts are spread across all the disks in the back-end storage pool for greater performance. On one very large database we have to do a re-index on a particular table monthly or quarterly. This took 24 hours on the old storage infrastructure. When we moved it to NetApp we were surprised (and pleased) to find that the same procedure took a much more manageable four hours to complete–a 6X improvement.

While we haven't analyzed all possible reasons for the speed-up, it is likely that it results from spreading out the activity across more disks in the back-end storage arrays. Our use of another NetApp product, NetApp SyncMirror®, to ensure availability probably also contributes to this performance improvement. Which leads us to…

Stage #2: Ensure Availability with SyncMirror

Sentara's availability goal is 99.99% uptime. By combining a virtualized storage infrastructure with NetApp SyncMirror synchronous mirroring software we successfully achieved this goal for the past 12 months. With SyncMirror, our clustered V3050 controllers maintain two fully consistent copies of all critical data, one on each XP12000 array. This protects us against all types of hardware outages including multiple disk failures or the failure of an entire array.

SyncMirror improves the performance of random disk read operations using simultaneous round-robin reads from both mirrored copies of data. This results in up to 80% improved random read performance, which is particularly beneficial for database environments. The substantial re-indexing speed-up discussed above probably resulted in part from SyncMirror's improved read performance.

Stage #3: Consolidate Servers with VMware VI3 and FAS Storage

About four months after the initial implementation we purchased a NetApp FAS3020c to provide back-end storage for our server consolidation efforts using VMware. Our initial experiences with NetApp had been extremely positive, plus we knew that we wanted to use iSCSI for connectivity within this environment.

iSCSI has been a perfect fit–it offers relatively simple management and low-cost Ethernet networking. (We used Fibre Channel rather than iSCSI with the V-Series deployment because we were inserting it as a virtualization layer within an existing FC SAN.)

Because the FAS3020c is clustered, we can perform rolling upgrades on each head to avoid downtime during upgrades. Through the combination of NetApp FAS storage systems, VMotion, and VMware High Availability (VMware HA) we have been able to maintain service levels above four nines (>99.99% availability) for the past 12 months.

So far, we've consolidated about 30% of our server environment. We've replaced about 190 physical servers with direct attached storage (DAS), with virtual servers using NetApp storage. That's resulted in about $700K in savings over the past four months. That number includes savings from a wide variety of factors including reductions in air conditioning, power, administration costs, etc. It does not include cost reductions due to reduced downtime, which is a significant benefit we get from NetApp. As a result of this effort, we've cut the deployment time for new servers from four weeks down to two weeks.

Availability and Flexibility on the Critical Path to Success

The digital revolution in the healthcare industry has put IT infrastructure on the critical path for patient care. Application and data availability have now become critical to healthcare delivery. Those of us tasked with creating the IT infrastructure to support leading-edge healthcare facilities are focused on delivering the highest possible availability with the flexibility to adapt to rapid change.

For Sentara, NetApp has become a key partner in achieving these goals. It may have been possible to do what we've done without using NetApp technology, but NetApp has made it simple and relatively painless. Within two days of installing the V-Series, we were presenting storage for use by hosts, and the availability we've achieved speaks for itself.

The NetApp V-Series has allowed us to better utilize the storage we already have, protecting that investment. Both the V-Series and our NetApp FAS systems give us the ability to support more platforms using more protocols while providing new capabilities such as FlexVol, FlexClone, SnapManager, and SyncMirror. That flexibility will be critical going forward.

Sentara's core mission is to improve health every day and lead the community by providing the best healthcare possible. This requires technology that is always available–obviously no small feat, but a goal NetApp is helping us achieve.

---