This is the first in a three-part series about Oracle® storage architectures. Next month's article explores how NetApp has optimized storage for Oracle in FC SAN environments, and November will feature a technical case study involving Oracle on NetApp using NAS, iSCSI, and FC SAN.

There is no simple "one size fits all" solution when it comes to storage system design for Oracle. Performance, availability, and of course budget requirements vary dramatically. Choosing a storage protocol may be one of the most critical decisions most IT teams make—although many may not think of it as a decision at all.

IT teams have traditionally deployed block-based protocols to support Oracle databases and applications, but a large number of companies today run Oracle on the Network File System (NFS). Many of these organizations—ranging from small IT shops to major Fortune 100 companies, including Oracle itself—use Network Appliance^TM storage. NetApp has been a leader in the NFS market since 1992 and has pioneered and legitimized the use of NFS with Oracle.

This article describes how NetApp engineering has worked to make NFS a robust storage option for Oracle. Key investments include:

• Enhancing NFS implementations for database
• Partnering to ensure high availability
• Driving industry standards including NFSv4
• Guiding emerging technologies like Oracle Direct NFS
• Developing integrated, Oracle-specific tools
• Defining Oracle on NFS best practices

We'll begin by examining why it makes sense to design an Oracle storage architecture based on NFS in the first place.

Oracle on NetApp NFS provides users with an easy to manage and flexible infrastructure capable of running a large number of database instances. Ethernet-based NFS facilitates simple provisioning, management, and sharing plus the flexibility to rapidly move and re-purpose database and storage resources. NFS implementations are also typically cost-effective as they do not require special Fibre Channel SAN expertise or training and allow IT teams to leverage standard Ethernet knowledge.

Oracle on NetApp NFS can be deployed for use in virtually any type of Oracle environment ranging from test/dev to 24x7 productions OLTP (including ERP and custom applications) environments. NetApp customers have found that—when properly configured and tuned—Oracle on NetApp NFS delivers 80-90% of the transactional performance of Fibre Channel using identical hardware and test conditions, and NFS performance is typically comparable to iSCSI. NFS can achieve the same absolute performance levels as FCP through the addition of hardware resources.

The observed differential with FCP results primarily from two factors:

• FCP protocol processing (or iSCSI when an iSCSI HBA is in use) are offloaded from system CPUs to the HBA. Since all NFS processing is done on system CPUs, NFS requires slightly more CPU resources to do the same job.
• Fibre Channel networks currently have a per network bandwidth advantage (4Gbits/sec is common) versus IP networks (1Gbit/sec is common). As 10Gbits/sec Ethernet becomes more prevalent, this advantage will go away.

NetApp has done a careful evaluation of NFS performance versus FCP and other protocols on a variety of environments, including Oracle 10g^TM on Linux®, Solaris^TM, HP-UX, and AIX,as well as Oracle 10g RAC on Linux. You can refer to these reports for more information relative to your particular requirements.

Protocol Comparison NFS FC SAN Simpler provisioning/management Less expensive hardware support Easy sharing to support RAC/grid Highest performance per CPU Utilizes host file system or ASM Leverages FC investments

NFS version 3 (NFSv3, the version that is in wide use today) consists of server software—for instance, the software that runs on NetApp storage—and client software running on database servers or other hosts that require access to network storage. Achieving good performance and ensuring proper behavior require that both sides of the connection, client and server, are correctly implemented.

To ensure robustness on the server side, NetApp pairs its time-tested server code with a hardware platform that delivers proven availability and reliability through features such as active-active controller configuration and double-parity RAID.

The NFS client can be implemented as part of any of a number of operating systems, including Solaris, HP-UX, AIX, and Linux. NetApp engineering has worked closely with all of the major implementers of NFS to verify that client implementations are functionally correct and perform well for database workloads. NetApp has a team of interoperability engineers who have carefully studied the requirements for a storage interconnect to recover correctly from error conditions and have worked to make sure that NFS implementations consistently meet those requirements.

NetApp engineering's work with the engineering teams of various vendors has led to the wide implementation of the following NFS enhancements:

• Direct I/O is the ability to do NFS I/O directly from an application over a network to storage, bypassing the NFS client buffer cache. Direct I/O avoids redundant caching on the server, gives Oracle control over client I/O behavior, and supports data-sharing semantics required by Oracle RAC.
• Asynchronous I/O allows a database to manage many concurrent I/O transactions without processes blocking on individual transactions. This results in greater throughput by reducing process counts and overhead.
• Elimination of kernel bottlenecks that throttle client I/O enables support for large numbers of concurrent I/O operations.
• Additional features support Oracle RAC on NFS.
• Quorum and I/O fencing are supported, in conjunction with high-availability partners.

Clustering is important to ensure high availability for critical database servers. NetApp has partnered with Oracle, Symantec, and Sun to ensure that NFS is supported for use with Oracle RAC, Veritas^TM Cluster Server (VCS), and Sun^TM Cluster, respectively.

NetApp and Sun have worked together to develop a package for Sun Cluster that allows clustered servers to access shared storage via NFS rather than a shared Fibre Channel or SCSI connection to disk. The NetApp software includes support for I/O fencing and quorum to ensure proper behavior in the event of a server failure. The NetApp software package for Sun Cluster supports both Sun SPARC and x86 platform running Sun Solaris.

The VCS agent for NFS supports Oracle running on a broad range of operating systems (Linux, AIX, Solaris, etc.) and is a good option for customers who need a high-availability solution for single-instance failover.

Both the Sun package and the VCS agent for NetApp NFS are available on the NOW^TM (NetApp on the Web) site.

NetApp chief scientist Steve Kleiman was one of the inventors of NFS, and over the years NetApp technical experts have taken leadership roles in helping define industry-wide standards. For example:

• Brian Pawlowski (NetApp chief technical officer) coauthored the NFSv3 specification and is cochair of the NFS version 4 (NFSv4) Working Group of the Internet Engineering Task Force (IETF).
• Garth Goodson (NetApp engineer) is co-author and editor of the pNFS portion of NFSv4.1.
• Mike Eisler (NetApp senior technical director) is the primary author of RFC 2203, which enhances the security of NFS, and recently coauthored RFC 3530, the NFSv4 specification. He is currently an editor for the NFSv4.1 specification. Prior to joining NetApp, Mike led the SEAM project at Sun that produced the first NFS implementation that used Kerberos V5 authentication.
• Tom Talpey (NetApp technical director) coauthored several of the NFS over RDMA specifications at IETF.
• Dave Noveck (NetApp senior engineer) is also a coeditor of the NFSv4.1 specification and a coauthor of RFC 3530.

NFSv4 is designed as an even more robust implementation of NFS and includes features that are beneficial in Oracle environments. For instance, the NFSv4 redirection feature allows a storage grid and a compute grid to mutually optimize I/O paths. A storage system can tell a compute server which storage system can best service particular requests to facilitate grid-based scale-out.

NFSv4 is currently gaining acceptance, and we expect to see a broader support from database applications for NFSv4 in the future. Current information about NFSv4 developments is available on Eisler's NFS Blog.

Direct NFS is an NFS client created by Oracle that is shipping as part of Oracle 11g^TM (announced in July, 2007). Instead of relying on the operating system, Oracle's Direct NFS client generates NFS requests directly from the database.

In a recent blog post, NetApp founder Dave Hitz explained that Oracle optimized its code specifically for NFS based on the company's experience using NFS to run applications on tens of thousands of Linux servers accessing many petabytes of NetApp storage. NetApp worked closely with Oracle on Direct NFS development, providing technical consulting as well as functional and performance testing resources.

This Oracle client implementation delivers several unique advantages:

• Oracle can use the same NFS client, regardless of the platform it is running on, for more consistent and predictable performance.
• Because the client spans platforms, bugs can be identified and fixed faster.
• This will be the first NFS client validated for Oracle on Windows®.
• No mount option tuning is necessary. Existing clients require specific mount options for optimal Oracle performance, and the options differ for each operating system.
• It is easy to distribute NFS workloads across up to 16 network links. This has been difficult with traditional NFS clients.

Oracle DBAs like running Oracle on NFS because it essentially gives them the ability to provision their own storage. Because Oracle can create files or make them bigger, the DBA can simply create or grow tablespaces and other files as necessary, without having to contact a storage administrator to add LUNs or grow underlying LUNs every time. NFS also makes it easy to move storage or applications from one server to another; you don't have to do any complicated rezoning of switches, etc. Because NFS is a shared file system, by design it works with Oracle RAC.

Snapshot and FlexClone

Several of the capabilities of NetApp storage contribute to the ease of Oracle data management in NFS environments. NetApp Snapshot^TM copies are space-efficient, point-in-time copies that can be created in a matter of seconds. To create an Oracle backup, you only need to put the database in hot backup mode for a few seconds while a Snapshot copy is being created. The impact to operations is usually negligible, so most DBAs schedule multiple Snapshot copies throughout the day as a safety net. You can retain up to 255 Snapshot copies of each storage volume and restore individual files from previous Snapshot copies or revert to a previous copy if a problem occurs.

Typically, IT shops using Oracle keep a number of Snapshot copies on primary storage to meet immediate needs. Snapshot copies can also be copied to secondary storage using NetApp SnapVault®, mirrored to a remote disaster recovery site using SnapMirror®, backed up to tape, or any combination of these options—whatever is necessary to meet data protection requirements. Because these operations occur on the storage system, there is no impact to running Oracle applications.

NetApp FlexClone® is another option that has proven to be highly useful in Oracle and other database environments. FlexClone allows you to create a writeable clone of any data volume. New disk space is consumed only as the original volume and the clone deviate from each other. This means that you can easily clone production data for test and development, data mining, or other purposes without the typical 2X storage requirement. As with Snapshot, you can create up to 255 FlexClone volumes of a single volume. (A recent Tech OnTap case study described the use of FlexClone to streamline Oracle application development and test.)

NetApp has also created a number of value-added software tools specifically for Oracle environments.

SnapManager for Oracle

SnapManager® for Oracle is a management tool that simplifies the management of Oracle backup and recovery. It works across all storage protocols and integrates closely with Oracle ASM and RMAN. SnapManager for Oracle makes it easy to schedule the creation of consistent Snapshot copies and FlexClone volumes, enabling operations that previously required complicated scripts to be accomplished with a few clicks or scheduled for regular execution. (See sidebar for details.)

SnapValidator for Oracle

In rare fault conditions, data can get corrupted on the data path between server and storage. Oracle keeps a checksum on each block so that this corruption is detected, but in many cases it may be months before data is actually reread and checked, resulting in the need to recover from a very old backup. To address this issue, SnapValidator® for Oracle complies with the Oracle hardware assisted resilient data (HARD) initiative to verify the checksum and block offset every time a block is written. When a problem is detected the write is failed, forcing the server to repeat the write so that no restore is required.

This feature works particularly well with NFS. Traditional host file systems cannot support HARD validation due primarily to the mixture of database blocks and file system metadata in writes to the storage system. Under Oracle 9i^TM, HARD validation is practical only for raw disk and NFS; Oracle 10g extended that capability to Oracle ASM.

SnapLock Integration with Oracle

Oracle includes partitioning and transportable tablespace capabilities that make it possible to relocate subsets of your data to different storage. SnapLock® integration makes it possible to relocate content and messages from Oracle to write once, read many (WORM) volumes. This ensures that the data cannot be modified, helping to meet the requirements of Sarbanes-Oxley (SOX), HIPAA, and other regulatory guidelines. This feature is particularly advantageous with NFS because of the finer level of granularity that NFS provides; you can choose to use SnapLock on a particular file or sets of files. In a SAN environment, you would have to use SnapLock on an entire LUN.

Although deploying Oracle on NFS is fairly straightforward, NetApp has formalized a variety of best practices to optimize performance. The NetApp Technical Report library includes a number of implementation guides and best practices, including a 34-page guide to Best Practices for Oracle (PDF). NetApp engineers have also teamed with peers at leading OS vendors to document best practices for specific environments. A recent joint paper by Sun and NetApp (PDF), for example, provides proven guidelines for optimizing Oracle on NFS in a Solaris environment, including Oracle tuning options, appropriate mount options for the Solaris NFS client, optimal database layout on NetApp storage, and NFS tuning on NFS storage.

If your primary objective is to create a simple, easily managed infrastructure that is capable of flexibly supporting a large number of Oracle database instances, NetApp NFS may be the right choice for you. NetApp has a proven track record of Oracle deployments on NFS, and the combination of NetApp storage features such as Snapshot and FlexClone with Oracle-specific software like SnapManager for Oracle, SnapValidator, and SnapLock can make the lives of DBAs and storage administrators much simpler.

Ultimately, of course, your choice of protocol depends on a wide variety of factors that include existing infrastructure, processes, and skill sets, in addition to the relative capabilities of each technology. NetApp offers excellent implementation options and tools for deploying and managing Oracle environments for NFS, Fibre Channel, and iSCSI, alone or in combination, so that customers deploying Oracle databases on NetApp storage realize best-of-breed capabilities on the protocols that fit their requirements.

For more information, visit NetApp at Oracle OpenWorld in San Francisco, November 11 -15, 2007. Stop by booth 1602 or attend one of four NetApp special sessions, including the following:

---

Comment on this article

---

The information in this article is useful to me. 1 2 3 4 5 Strongly Disagree Strongly Agree What other types of information would you like to see included in this article? Additional comments:   Thanks!