How to Enable Oracle VM 2.1.2 High Availability (HA) on an Existing Server Pool using NFS Storage

In this Document
Goal
Solution
References

---

Applies to:

Oracle VM - Version: 2.1.2 to 2.1.2
Information in this document applies to any platform.

Goal

How to implement a High Availability (HA) fail over for an existing server pool in Oracle VM 2.1.2 using NFS. This configuration also uses ocfs2 clustering technologies (nfs + dmnfs + ocfs2 hb).

Solution

This NFS HA is using two Oracle 2.1.2 VM servers already part of a server pool (vm_pool_master.us.oracle.com, vm_pool.us.oracle.com) and one NFS server (nfs-lin.us.oracle.com).

Note: This setup has already pre-configured the ocfs2 cluster.conf and is not running iptables on both nodes.

Disabled iptables:
#/etc/init.d/iptables off ; iptables -F ; chkconfig --level 345 iptables off

Pre-configured ocfs2 cluster.conf:
http://oss.oracle.com/projects/ocfs2/dist/documentation/v1.2/ocfs2_faq.html#CONFIGURE

The servers where already part of a server pool, and confirmed live migrations where a success within VM Manager.

There is already a shared cluster root at /OVS using NFS, and not local.

[root@vm_pool_master /]# uname -a
Linux vm_pool_master 2.6.18-8.1.15.1.16.el5xen #1 SMP Fri Aug 1 18:27:30 EDT 2008 i686 i686 i386 GNU/Linux
[root@vm_pool_master /]# df -h
Filesystem Size Used Avail Use% Mounted on
/dev/sda2 3.0G 2.1G 696M 76% /
/dev/sda1 99M 40M 55M 43% /boot
tmpfs 257M 0 257M 0% /dev/shm
nfs-lin.us.oracle.com:/OVS
228G 179G 37G 83% /OVS

[root@vm_pool_master /]#
[root@vm_pool_master /]# ls -al /OVS
total 20
drwxr-xr-x 5 root root 4096 Sep 30 16:22 .
drwxr-xr-x 30 root root 4096 Sep 30 15:36 ..
drwxr-xr-x 2 root root 4096 Sep 30 16:12 iso_pool
drwxr-xr-x 2 root root 4096 Sep 30 16:12 running_pool
drwxr-xr-x 2 root root 4096 Sep 30 16:12 seed_pool
[root@vm_pool_master /]#



[root@vm_pool /]# uname -a
Linux vm_pool 2.6.18-8.1.15.1.16.el5xen #1 SMP Fri Aug 1 18:27:30 EDT 2008 i686 i686 i386 GNU/Linux
[root@vm_pool /]# df -h
Filesystem Size Used Avail Use% Mounted on
/dev/sda2 3.0G 825M 2.0G 30% /
/dev/sda1 99M 44M 51M 47% /boot
tmpfs 272M 0 272M 0% /dev/shm
nfs-lin.us.oracle.com:/OVS
228G 179G 37G 83% /OVS

[root@vm_pool /]#
[root@vm_pool /]# ls -al /OVS
total 20
drwxr-xr-x 5 root root 4096 Sep 30 16:22 .
drwxr-xr-x 30 root root 4096 Sep 30 15:36 ..
drwxr-xr-x 2 root root 4096 Sep 30 16:12 iso_pool
drwxr-xr-x 2 root root 4096 Sep 30 16:12 running_pool
drwxr-xr-x 2 root root 4096 Sep 30 16:12 seed_pool
[root@vm_pool /]#


1. From VM Server pool master to setup ocfs2, this script

# /usr/lib/ovs/ovs-cluster-configure

generates the /etc/ocfs2/cluster.conf file if it does not already exist and configures and starts the o2cb service. Here we pre-configure ocfs2 cluster.conf:

[root@vm_pool_master /]#
[root@vm_pool_master /]# cat /etc/ocfs2/cluster.conf
node:
ip_port = 7777
ip_address = 10.1.1.10
number = 0
name = vm_pool_master
cluster = ocfs2

node:
ip_port = 7777
ip_address = 10.1.1.11
number = 1
name = vm_pool
cluster = ocfs2

cluster:
node_count = 2
name = ocfs2
[root@vm_pool_master /]#
[root@vm_pool_master /]# /etc/init.d/o2cb configure
Configuring the O2CB driver.

This will configure the on-boot properties of the O2CB driver.
The following questions will determine whether the driver is loaded on
boot. The current values will be shown in brackets ('[]'). Hitting
without typing an answer will keep that current value. Ctrl-C
will abort.

Load O2CB driver on boot (y/n) [n]: y
Cluster to start on boot (Enter "none" to clear) [ocfs2]:
Specify heartbeat dead threshold (>=7) [31]:
Specify network idle timeout in ms (>=5000) [30000]:
Specify network keepalive delay in ms (>=1000) [2000]:
Specify network reconnect delay in ms (>=2000) [2000]:
Writing O2CB configuration: OK
Loading module "configfs": OK
Mounting configfs filesystem at /config: OK
Loading module "ocfs2_nodemanager": OK
Loading module "ocfs2_dlm": OK
Loading module "ocfs2_dlmfs": OK
Mounting ocfs2_dlmfs filesystem at /dlm: OK
Starting O2CB cluster ocfs2: OK
[root@vm_pool_master /]#
[root@vm_pool_master /]#
[root@vm_pool_master /]# /etc/init.d/o2cb status
Module "configfs": Loaded
Filesystem "configfs": Mounted
Module "ocfs2_nodemanager": Loaded
Module "ocfs2_dlm": Loaded
Module "ocfs2_dlmfs": Loaded
Filesystem "ocfs2_dlmfs": Mounted
Checking O2CB cluster ocfs2: Online
Heartbeat dead threshold: 31
Network idle timeout: 30000
Network keepalive delay: 2000
Network reconnect delay: 2000
Checking O2CB heartbeat: Not active
[root@vm_pool_master /]#
[root@vm_pool_master /]#
[root@vm_pool_master /]# /usr/lib/ovs/ovs-cluster-configure
O2CB cluster ocfs2 already online
[root@vm_pool_master /]#


2. On the VM Server Pool Master, configure the shared storage with

# /usr/lib/ovs/ovs-makerepo source C description. 

This creates a .ovsrepo file in the NFS mounted /OVS, and updates /etc/ovs/repositories file :

[root@vm_pool_master /]# /usr/lib/ovs/ovs-makerepo nfs-lin.us.oracle.com:/OVS C "ha for prod_pool"
Initializing NEW repository nfs-lin.us.oracle.com:/OVS
Updating local repository list.
ovs-makerepo complete
[root@vm_pool_master /]#
[root@vm_pool_master /]#
[root@vm_pool_master /]# more /OVS/.ovsrepo
OVS_REPO_UUID=05B0CE2EFE9C4842A2EFA39926657236
OVS_REPO_SHARED=1
OVS_REPO_DESCRIPTION=ha for prod_pool
OVS_REPO_VERSION=1
[root@vm_pool_master /]#
[root@vm_pool_master /]# cat /etc/ovs/repositories
# This configuration file was generated by ovs-makerepo
# DO NOT EDIT
[root@vm_pool_master /]#


3. On the VM Server Pool Master, check the cluster has been configured correctly and is running:

# /usr/lib/ovs/ovs-cluster-check --master --alter-fstab

where

--master parameter should only be used on the Server Pool Master.
--alter-fstab parameter modifies the /etc/fstab file.

A backup of the original /etc/fstab file is created in /tmp.

This also creates a .server_pool_hb file on your /OVS mount along with removing any mounts in your /etc/fstab file for /OVS as now all the mounting of this will be done from the /etc/ovs/repositories file.:

[root@vm_pool_master /]# /usr/lib/ovs/ovs-cluster-check --master --alter-fstab
O2CB cluster ocfs2 already online
300+0 records in
300+0 records out
314572800 bytes (315 MB) copied, 169.856 seconds, 1.9 MB/s
mkfs.ocfs2 1.2.7
Filesystem label=server_pool_hb
Block size=2048 (bits=11)
Cluster size=4096 (bits=12)
Volume size=314572800 (76800 clusters) (153600 blocks)
5 cluster groups (tail covers 13312 clusters, rest cover 15872 clusters)
Journal size=8388608
Initial number of node slots: 32
Creating bitmaps: done
Initializing superblock: done
Writing system files: done
Writing superblock: done
Writing backup superblock: 0 block(s)
Formatting Journals: done
Writing lost+found: done
mkfs.ocfs2 successful

Cluster setup complete.
[root@vm_pool_master /]#
[root@vm_pool_master /]#
[root@vm_pool_master /]# ls -al /OVS
total 307528
drwxr-xr-x 5 root root 4096 Sep 30 16:22 .
drwxr-xr-x 30 root root 4096 Sep 30 15:36 ..
drwxr-xr-x 2 root root 4096 Sep 30 16:12 iso_pool
-rw-r--r-- 1 root root 122 Sep 30 16:12 .ovsrepo
drwxr-xr-x 2 root root 4096 Sep 30 16:12 running_pool
drwxr-xr-x 2 root root 4096 Sep 30 16:12 seed_pool
-rw-r--r-- 1 root root 314572800 Sep 30 16:54 .server_pool_hb
[root@vm_pool_master /]#



4. Log into you VM Manager to check and enable the HA feature.

Under Server Pools Tab -> Edit your Server Pool you will see:
"High Availability Infrastructure: Check"
Click on Check button, you should see:

All the servers should share the same storage.
High Availability configuration should be correctly setup.

Now you can enable HA.
Checking the box "Enable High Availability: " and then Apply. Which should return:

The Server pool updated successfully.

In the VM Manager you should now see under the "High Availability Status" column Enabled for the Pool in the "Server Pools" tab.


5. The second VM server will show /etc/ovs/repositories and /etc/ocfs2/cluster.conf matching that of the VM Server pool master. The /OVS mount, must be maintained in the /etc/ovs/repositories file.
Confirm no /OVS mount is in /etc/fstab, a backup should be in the /tmp directory:

[root@vm_pool /]#
[root@vm_pool /]#
[root@vm_pool /]# cat /etc/ocfs2/cluster.conf
node:
ip_port = 7777
ip_address = 10.1.1.10
number = 0
name = vm_pool_master
cluster = ocfs2

node:
ip_port = 7777
ip_address = 10.1.1.11
number = 1
name = vm_pool
cluster = ocfs2

cluster:
node_count = 2
name = ocfs2
[root@vm_pool /]#
[root@vm_pool /]# more /etc/ovs/repositories
# This configuration file was generated by ovs-makerepo
# DO NOT EDIT
[root@vm_pool /]#
[root@vm_pool /]# more /etc/fstab
LABEL=/ / ext3 defaults 1 1
LABEL=/boot /boot ext3 defaults 1 2
devpts /dev/pts devpts gid=5,mode=620 0 0
tmpfs /dev/shm tmpfs defaults 0 0
proc /proc proc defaults 0 0
sysfs /sys sysfs defaults 0 0
LABEL=SWAP-sda5 swap swap defaults 0 0
#
[root@vm_pool /]#
[root@vm_pool /]# diff /etc/fstab /tmp/fstab.vzJsGq3831
9a10
> nfs-lin.us.oracle.com:/OVS /OVS nfs rw,bg,intr,hard,timeo=600,wsize=32768,rsize=32768,nfsvers=3,tcp 0 0
[root@vm_pool /]#


6. Confirm ocfs2 is loaded and set to start on boot along with the repositories on both VM servers.:

[root@vm_pool /]# /etc/init.d/o2cb status
Module "configfs": Loaded
Filesystem "configfs": Mounted
Module "ocfs2_nodemanager": Loaded
Module "ocfs2_dlm": Loaded
Module "ocfs2_dlmfs": Loaded
Filesystem "ocfs2_dlmfs": Mounted
Checking O2CB cluster ocfs2: Online
Heartbeat dead threshold: 31
Network idle timeout: 30000
Network keepalive delay: 2000
Network reconnect delay: 2000
Checking O2CB heartbeat: Active
[root@vm_pool /]#
[root@vm_pool /]# chkconfig --list | egrep "ocfs2|ovs"
ocfs2 0:off 1:off 2:on 3:on 4:on 5:on 6:off
ovs-agent 0:off 1:off 2:on 3:on 4:on 5:on 6:off
ovsrepositories 0:off 1:off 2:on 3:on 4:on 5:on 6:off
[root@vm_pool /]#
[root@vm_pool /]# ls -al /OVS
total 307528
drwxr-xr-x 5 root root 4096 Sep 30 16:22 .
drwxr-xr-x 30 root root 4096 Sep 30 15:36 ..
drwxr-xr-x 2 root root 4096 Sep 30 16:12 iso_pool
-rw-r--r-- 1 root root 122 Sep 30 16:12 .ovsrepo
drwxr-xr-x 2 root root 4096 Sep 30 16:12 running_pool
drwxr-xr-x 2 root root 4096 Sep 30 16:12 seed_pool
-rw-r--r-- 1 root root 314572800 Sep 30 16:54 .server_pool_hb
[root@vm_pool /]#



Note, you can monitor for errors in the /var/log/ovs-agent/ovs_operation.log on the VM Server pool master during setup.

Success Log was:

"2008-09-26 20:03:24" INFO=> ha_sync_shared_storage_mount: among servers(['vm_pool_master.us.oracle.com', 'vm_pool.us.oracle.com'])...
"2008-09-26 20:03:24" INFO=> ha_mount_shared_storage: success
"2008-09-26 20:03:24" INFO=> ha_sync_shared_storage_mount: success.
"2008-09-26 20:03:25" INFO=> ha_setup_ocfs2_cluster: trying to reconfigure the cluster...
"2008-09-26 20:03:26" INFO=> ha_setup_ocfs2_cluster: success.
"2008-09-26 20:03:26" INFO=> ha_start_cluster_service: success.
"2008-09-26 20:03:27" INFO=> ha_start_agent_guard: success.
"2008-09-26 20:03:27" INFO=> ha_start_agent_guard_all: success.
"2008-09-26 20:03:27" INFO=> ha_setup_cluster_infrastructure: success.
"2008-09-26 20:03:27" INFO=> set_pool_ha_enable: pool_ha_enable=True


A failure Log (which was from a mis-configured ocfs2) was:

"2008-09-26 18:57:37" INFO=> ha_chec _oracle_vm_version: among servers(['vm_pool_master.us.oracle.com', 'vm_pool.us.oracle.com'])...
"2008-09- 6 18:57:38" INFO=> ha_check_oracle_vm_version: success:oracle_vm_version=['2.1.2']
"2008-09- 6 18:57:38" INFO=> ha_check_agent_version: among servers(['vm_pool_master.us.oracle.com', vm_pool.us.oracle.com'])...
"2008-09-26 18:57:38" INFO=> ha_check_agent_version: success:age t_version=2.2-51
"2008-09-26 18:57:38" INFO=> ha_precheck_storage_mount: success.
"2 08-09-26 18:57:38" INFO=> ha_check_cluster_prerequisite: success.
"2008-09-26 18:57:38& uot; INFO=> ha_check_shared_storage: among servers(['vm_pool_master.us.oracle.com', 'vm_pool.us.oracle.com'])...
"2008-09-26 18:57:38" INFO=> ha_check_shared_storage: success. /OVS/['']
"2008-09-26 18:57:38" INFO=> ha_check_cluster_status: among servers(['vm_pool_master.us.oracle.com', 'vm_pool.us.oracle.com'])...

"2008-09-26 18:57:39" ERROR=> ha_check_cluster_status: failed. =>
StackTrace:
File "/opt/ovs-agent-2.2/OVSSiteHA.py", line 308, in ha_check_cluster_status
raise Exception(rs)
>
StackTrace:
File "/opt/ovs-agent-2.2/OVSSiteHA.py", line 311, in ha_check_cluster_status
raise Exception("srv('%s'). failed! => %s" % (hostname, errmsg))
"2008-09-26 18:57:39" ERROR=> ha_check_cluster_infrastructure: error. =>
StackTrace:
File "/opt/ovs-agent-2.2/OVSSiteHA.py", line 308, in ha_check_cluster_status
raise Exception(rs)
>
StackTrace:
File "/opt/ovs-agent-2.2/OVSSiteHA.py", line 311, in ha_check_cluster_status
raise Exception("srv('%s'). failed! => %s" % (hostname, errmsg))
>
StackTrace:
File "/opt/ovs-agent-2.2/OVSSiteHA.py", line 373, in ha_check_cluster_infrastructure
raise Exception(msg)



References

http://download.oracle.com/docs/cd/E11081_01/doc/doc.21/e10898/ha.htm#insertedID1
http://download.oracle.com/docs/cd/E11081_01/doc/doc.21/e10898/repository.htm#insertedID1
http://oss.oracle.com/projects/ocfs2/dist/documentation/v1.2/ocfs2_faq.html#CONFIGURE

Keywords

OCFS2 ; ORACLE~VM ; NFS ; HIGH~AVAILABILITY ; OCFS2_DLM_EVICTION_CB ; ORACLE~VM~SERVER ;
Thanks MetaLink