Virtualizing Oracle Databases on vSphere Best Practices

Hi all ...

In our next post, we will talk about Oracle Databases as our next candidate of Virtualization. Oracle DBs are known to be heavy duty, Tier 0 or Tier 1 servers that losing it can lead to business disruption significantly.
Oracle is a Strategic Partner of VMware. This provides a great support and compatibility of Oracle DBs above or below vSphere Platform. As we'll see in following sections, Oracle DB is supported as a virtual application above vSphere Platform or as a store for vSphere data used by vCenter Server for example.

As Microsoft SQL Servers, Oracle Applications and Database has its own features that are aligned with vSphere features to push Oracle Databases performance, availability and scalability to another level. Oracle Real Application Clusters (RAC) feature is completely supported on vSphere 5.x and can be used with vSphere HA to push databases availability towards the magic five 9's. Best practices mentioned here is collected from different sources that are mentioned in References section below. I'll follow the same schema of my previous posts and relate these best practices to our five Design Qualifiers (AMPRS - Availability, Manageability, Performance, Recoverability and Security) in addition to Scalability.

Availability:

1-) Try to use vSphere HA in addition to Oracle RAC Cluster to provide the highest level of availability. Adapt a protection policy of N+1, as N is the number of Oracle RAC Cluster members VMs in a vSphere Cluster of N+1 hosts. In case of an ESXi failure, in the background, vSphere HA powers-on the failed virtual machine on another host, restoring the failed Oracle RAC Clusters member and bringing the newly passive (failed old active node) member up and ready to take over in case of a failure, or to be manually reactivated as the primary active instance.

2-) Try to separate your Oracle RAC Cluster VMs on different Racks, Blade Chassis and Storage Arrays if available using VMs Affinity/Anti-affinity rules and Storage Affinity/Anti-affinity rules for most availability.

3-) For Oracle RAC Cluster VMs, use VMs anti-affinity rules to separate them over different hosts. When HA restart a VM, it’ll not respect the anti-affintiy rule, but on the following DRS invocation, the VM will be migrated to respect the rule. In vSphere 5.5, configure the vSphere Cluster with “das.respectVmVmAntiAffinityRules” set to 1 to respect all VMs Anti-affinity rules during HA restart.

4-) For zero down time, deploy Orace RAC Multi-node Cluster with leveraging vSphere HA, but it’ll lead to high costs of licenses and physical hosts to host many Oracle VMs. For near-zero downtime, you can deploy Oracle RAC Single-Node cluster with leveraging vSphere HA which costs much lower than Multi-Node cluster, but it suffers from some downtime while moving Oracle Instance from failed node to another node using OMotion. Eventually, it’s your choice according to your technical requirements and SLA.

5-) Try to leverage VM Monitoring to mitigate the risk of Guest OS failure. VMware Tools inside Oracle VMs will send heartbeats to HA driver on the host. If it’s stopped because Guest OS failure, the host will monitor IO and network activity of the VM for certain period. If there’s also no activity, the host will restart the VM. This add additional layer of availability for Oracle VMs. For more information, check this: vSphere HA VM Monitoring - Back to Basics | VMware vSphere Blog - VMware Blogs

6-) Try to leverage Symantec Application HA Agent for Oracle with vSphere HA for max. availability. Using Application HA, the monitoring agent will monitor Oracle instance and its services, sending heartbeats to HA driver on ESXi host. In case of application failure, it may restart services and any dependent resources. If Application HA Agent can’t recover the application from that failure, it’ll stop sending heartbeats and the host will initiate a VM restart as a HA action. This adds another layer of availability to your Oracle nodes. For more informations, check this pdf from VMware.

7-) vMotion:

a- vMotion is supported on Oracle single-node DB and Oracle RACs using shared disks with multi-writer flags only not with RDM shared disks using SCSI-Bus Sharing. Check vMotion Test on Workload Characterization document in References section below.

b- Leverage Multi-NIC vMotion with your Oracle Nodes for faster vMotion operations of large Oracle nodes.

c- Try to enable Jumbo Frames on vMotion Network to reduce overhead and increase throughput.

Performance:

1-) Configure the following BIOS Settings on each ESXi Host:

2-) Remove unnecessary services from the Guest OS, wither it’s Windows or Linux. For example, on Windows: Indexing Service, System Restore and Remote Desktop. For example on Linux: IPTables, Autofs and cups.

3-) Set VM settings to “Automatically Choose Best CPU/MMU Virtualization Mode”.

4-) CPU Sizing:

a- Don’t over-commit CPUs. It’s better to keep it nearly 1:1. In some cases like small environments, over-commitment is allowed after establishing a performance baseline of normal-state utilization.

b- Enable Hyperthreading when available. It won’t double the processing power –in opposite to what shown on ESXi host as double number of logical cores- but it’ll give a CPU processing boost up to 20-25% in some cases.

c- ESXi Hypervisor is NUMA aware and it leverages the NUMA topology to gain a significant performance boost. Try to size your Oracle VMs to fit inside single NUMA node to gain the performance boost of NUMA node locality.

d- In case of large Oracle VMs that won’t fit inside single NUMA node, keep in mind that Oracle 11g and above are supporting NUMA, i.e. Virtual NUMA-aware, but it’s disabled by default. It’s recommended to test if enabling NUMA Support and exposing vNUMA to Oracle would increase performance or not before applying it in production. To enable NUMA awareness on Oracle 11g, check Oracle Documentation Doc ID 864633.1.

5-) Memory Sizing:

a- Don’t over-commit memory, as Oracle is a memory-intensive application. If needed, reserve the configured memory to provide the required performance level. Keep in mind that memory reservation affects many aspects, like: HA Slot Size, vMotion chances and time. In addition, reservation of memory removes VM swapfiles from datastores and hence, its space is usable for adding more VMs. For some cases, where a lot of underutilized testing Oracle Servers there, over-commitment is preferred to get higher consolidation ratios. Performance monitoring is mandatory in this case to maintain a baseline of normal-state utilization.

b- Don’t disable Balloon Driver installed with VMware Tools. Ballooning is the last line of defense of ESXi Host before compression and swapping to disk when its memory becomes too low. When Ballooning is needed, Balloon Driver will force Guest OS to swap the idle memory pages to disk to return the free memory pages to the Host, i.e. swapping is done according to Guest OS techniques. Swapping to disk is done by ESXi Host itself. Host’d swap memory pages from physical memory to VM swap file on disk without knowing what these pages contain or if these pages are required (active) or idle. Ballooning effect on performance is somehow much lower than Swapping to disk. Generally speaking as mentioned in the previous point, don’t over-commit memory for business-critical Oracle VMs and if you’ll do some over-commitment, don’t push it to these extreme limits.

c- Use Large Memory Pages for Tier-1 Oracle VMs. Oracle Server supports the concept of Large Memory Pages when allocating memory since version 9i R2 for Linux and 10g R2 for Windows.

6-) Storage Sizing:

a- Always consider any storage space overhead while calculating VMs space size required. Overhead can be: swapfiles, VMs logs or snapshots. It’s recommended to add 20-30% of space as an overhead.

b- Separate different Oracle VMs’ disks on different –dedicated if needed- datastores to avoid IO contention, as Oracle is an IO-intensive application.

c- Provide at least 4 paths, through two HBAs, between each ESXi host and the Storage Array for max. availability.

d- For IP-bases Storage, enable Jumbo Frames on its network end-to-end. Jumbo Frames reduces network overhead and increases Throughput.

e- RDM can be used in many cases, like: Oracle P2V migration or to leverage 3^rd Party array-based backup tool. Choosing RDM disks or VMFS-based disks are based on your technical requirements. No performance difference between these two types of disks. Keep in mind that, Oracle Real Application Cluster (RAC) supports vMotion using shared VMFS datastores hosting its disks.

f- Use Paravirtual SCSI Driver in all of your Oracle VMs, specially disks used for DB and Logs, for max. performance, least latency and least CPU overhead.

g- Distribute any Oracle VM disks on the four allowed SCSI drivers for max. performance paralleling and higher IOps. It’s recommended to use Eager-zeroed Thick disks for DB and Logs disks.

h- The following table shows the Read/Write behavior of each of Oracle DB components:

i- Partition Alignment gives a performance boost to your backend storage, as spindles will not make two reads or writes to process single request. VMFS5 datastores created using vSphere (Web) Client will be aligned automatically as well as any disks formatted using newer versions of Windows. Any upgraded VMFS datastores or upgraded versions of Windows Guests will require a partitions alignment process. For upgraded VMFS, it’s done by migrating VMs disks to another datastore using Storage vMotion, then format and recreate the datastore on VMFS5.

j- Don’t use Oracle Clustered File System (OCFS). It is the predecessor of Oracle Automatic Storage Management (ASM) and doesn’t allow for single instance Oracle Server. Oracle ASM performs better and allows for single instance and clustered Oracle RACs.

k- Don’t use Oracle Automatic Storage Management (ASM) Failure Groups. It costs additional CPU overhead and may behave unexpectedly after failure of one disk in virtual environments. For disk redundancy, use external RAID –or similar technology- that’s done on storage-array level, transparently to Oracle Stack.

l- Create Oracle ASM Disk groups on similar disks and storage arrays, as a disk group perform as fast as the slowest disk inside the group.

m- For Oracle Multi-Node RAC Cluster, you can use either VMDK disks on VMFS datastores or RDM LUNs for shared CSR/Voting Disks. In case you use VMDK disks on VMFS datastores as CSR/Voting Disks, you have to disable simultaneous write protection provided by VMFS using the multi-writer flag (VMware KB: Enabling or disabling simultaneous write protection provided by VMFS using the multi-writer flag) to share these disks between nodes as well as using VMDK disks as “Independent Persistent” disks. For RDM disks as CSR/Voting Disks, you should use SCSI Bus Sharing in Physical Mode to share these disks between nodes.

7-) Network:

a- Use VMXNet3 vNIC in all Oracle VMs for max. performance and throughput and least CPU overhead.

b- Oracle VMs port group should have at least 2 physical NICs for redundancy and NIC teaming capabilities. Connect each physical NIC to a different physical switch for max. redundancy.

c- Consider network separation between different types of networks, like: vMotion, Management, Oracle production, Oracle Replication, Fault Tolerance, etc. Network separation is either physical or logical using VLANs.

d- Clustered Oracle VMs should have two vNICs, one for public network and the other one for heartbeat and replication network. It’s better to dedicate a physical NIC on ESXi hosts for replication network between Clustered Oracle RAC VMs.

8-) Monitoring:

Try to establish a performance baseline for your Oracle VMs and VI by monitoring the following:

- ESXi Hosts and VMs counters:

In-guest monitoring is important as well, as CPU and Memory usage is more accurately obtained using in-guest counters.

Manageability:

1-) Oracle Support statement for VMware:Oracle Support Statement ID 249212.1

2-) VMware Expanded support for Oracle DB:VMware Oracle Support Policy

3-) Try to leverage a Golden Template to provide a base to your Oracle VM. It’ll reduce the time required for deploying or scaling your Oracle environment as well as preserve consistency of configuration throughout your environment.

4-) Time Synchronization is one of the most important things in Oracle environments. It’s recommended to do the following:

a- Let all your Oracle VMs sync their time according to the following best practices:

Windows:VMware KB: Timekeeping best practices for Windows, including NTP

Linux:VMware KB: Timekeeping best practices for Linux guests

b- Disable time-sync between Oracle VMs and Hosts using VMware Tools totally (Even after uncheck the box from VM settings page, VM can sync with the Host using VMware Tools in case of startup, resume, snapshotting, etc.) according to the following KB:VMware KB: Disabling Time Synchronization

c- Sync all ESXi Hosts in the VI to the same Startum 1 NTP Server which is the same time source of your forest/domain.

5-) Leverage CPU/Memory Hot add with SQL VMs to scale them as needed.

Recoverability:

1-) You can use VMware Site Recovery Manager (SRM) with array-based replication capabilities to protect your Oracle RAC nodes. Oracle RAC nodes can't be protected with vSphere Replication as it doesn't support shared disk using Multi-writer flag or RDM.

2-) For single-instance Oracle DB's, vSphere Replication and vSphere SRM based on vSphere Replication can be used without any problem.

3-) Any backup solution can be used with single-instance Oracle DB's. It should integrate with Oracle DB for properly quiescing the DB before backing it up to prevent data corruption.

Security:

1-) All security procedures done for securing physical Oracle DBs should be done in virtual environment, like: Role-based Access Policy.

2-) Follow VMware Hardening Guide (v5.1/v5.5) for more security procedures to secure both of your VMs and vCenter Server.

Scalability:

1-) vSphere supports sizing Oracle VMs using Scale-out or Scale-up approaches. Scale-up Approach of Oracle VMs requires a large ESXi Hosts with many sockets and RAM. It reduces the number of VMs required to serve certain number of transactions and hence, a single failed VM will affect a large portion of users. That’s why Scale-up Approach needs a careful attention to availability of Oracle VMs. In the same time it reduces the cost of software licenses and physical hosts. Scale-out Approach requires smaller ESXi Hosts and gives a more flexibility in designing of an Oracle VM, but requires high number of ESXi hosts to provide the required level of availability and performance and more software licenses and hence, more cost. A single VM failure has a less effect using Scale-out Approach and it requires less time for migration using vMotion and hence, DRS will be more effective. There’s no best approach here. It all depends on your environment and your requirements.

2-) Leverage CPU/Memory Hot add with Oracle VMs to scale them as needed. Oracle -64bit versions mainly- is Hot-add aware and there’s no need to reboot a VM afer hot-add operation. It’ll use the added resource immediately.

I hope that I could summarize clearly the considerations of virtualizing Oracle DBs and RACs on vSphere 5.x. It's not so much complicated like Microsoft Clustering Services (MSCS), but it has its own considerations too.
Share the knowledge ...

References:

-- Oracle Databases on VMware - Best Practices Guide.

-- Oracle Databases on VMware - High Availability Guidelines.

-- Oracle Databases High Availability on VMware vSphere.

-- Oracle Databases on VMware - Workload Characterization.

-- Oracle Databases on VMware - RAC Deployment Guide.

-- http://www.amazon.com/VMware-vSphere-Design-Forbes-Guthrie/dp/1118407911/ref=pd_sim_b_7?ie=UTF8&refRID=1CDACQ85E8R8BWYB7GVSvSphere Design Sybex 2nd Edition by Scott Lowe, Kendrick Coleman and Forbes Guthrie