Monday, October 15, 2012

11.2.0.1 Grid infrastructure (Non Rolling) upgrade to 11.2.0.3 Grid infrastructure




The steps followed to upgrade 11.2.0.1 grid infrastructure to 11.2.0.3 GI


Current Environment:-

GI home - /grid/11.2 Version 11.2.0.1
New GI home - /grid/11.2.0.3 Version 11.2.0.3

Nodes - gridtest1 & gridtest2


1) Check whether all the prerequsite are met before doing the upgrade, there are two type of upgrade
one is rolling upgrade and other is non rolling upgrade.

Check the Metalink note for the patches to be applied if we are going with rolling upgrade.

Things to Consider Before Upgrading to 11.2.0.2 Grid Infrastructure/ASM [ID 1312225.1]

In this we are proceeding with NON rolling upgrade.


Run the cluvfy from the staging area of 11.2.0.3 software.

$ cd /software/11.2.0.3/grid

$ ./runcluvfy.sh stage -pre crsinst -upgrade -n gridtest1,gridtest2 -src_crshome /grid/11.2 -dest_crshome /grid/11.2.0.3 -dest_version 11.2.0.3.0 -fixup -fixupdir /home/oracle -verbose


Performing pre-checks for cluster services setup

Checking node reachability...

Check: Node reachability from node "gridtest1"
  Destination Node                      Reachable?
  ------------------------------------  ------------------------
  gridtest1                             yes
  gridtest2                             yes
Result: Node reachability check passed from node "gridtest1"


Checking user equivalence...

Check: User equivalence for user "oracle"
  Node Name                             Status
  ------------------------------------  ------------------------
  gridtest2                             passed
  gridtest1                             passed
Result: User equivalence check passed for user "oracle"

Checking CRS user consistency
Result: CRS user consistency check successful

Checking node connectivity...

Checking hosts config file...
  Node Name                             Status
  ------------------------------------  ------------------------
  gridtest2                             passed
  gridtest1                             passed

Verification of the hosts config file successful


Interface information for node "gridtest2"
 Name   IP Address      Subnet          Gateway         Def. Gateway    HW Address        MTU
 ------ --------------- --------------- --------------- --------------- ----------------- ------
 eth0   192.168.0.31    192.168.0.0     0.0.0.0         192.168.0.1     00:0C:29:6B:A6:76 1500
 eth0   192.168.0.36    192.168.0.0     0.0.0.0         192.168.0.1     00:0C:29:6B:A6:76 1500
 eth1   10.10.10.31     10.10.10.0      0.0.0.0         192.168.0.1     00:0C:29:6B:A6:80 1500


Interface information for node "gridtest1"
 Name   IP Address      Subnet          Gateway         Def. Gateway    HW Address        MTU
 ------ --------------- --------------- --------------- --------------- ----------------- ------
 eth0   192.168.0.30    192.168.0.0     0.0.0.0         192.168.0.1     00:0C:29:B0:4B:8F 1500
 eth0   192.168.0.38    192.168.0.0     0.0.0.0         192.168.0.1     00:0C:29:B0:4B:8F 1500
 eth0   192.168.0.35    192.168.0.0     0.0.0.0         192.168.0.1     00:0C:29:B0:4B:8F 1500
 eth1   10.10.10.30     10.10.10.0      0.0.0.0         192.168.0.1     00:0C:29:B0:4B:99 1500


Check: Node connectivity for interface "eth0"
  Source                          Destination                     Connected?
  ------------------------------  ------------------------------  ----------------
  gridtest2[192.168.0.31]         gridtest2[192.168.0.36]         yes
  gridtest2[192.168.0.31]         gridtest1[192.168.0.30]         yes
  gridtest2[192.168.0.31]         gridtest1[192.168.0.38]         yes
  gridtest2[192.168.0.31]         gridtest1[192.168.0.35]         yes
  gridtest2[192.168.0.36]         gridtest1[192.168.0.30]         yes
  gridtest2[192.168.0.36]         gridtest1[192.168.0.38]         yes
  gridtest2[192.168.0.36]         gridtest1[192.168.0.35]         yes
  gridtest1[192.168.0.30]         gridtest1[192.168.0.38]         yes
  gridtest1[192.168.0.30]         gridtest1[192.168.0.35]         yes
  gridtest1[192.168.0.38]         gridtest1[192.168.0.35]         yes
Result: Node connectivity passed for interface "eth0"


Check: TCP connectivity of subnet "192.168.0.0"
  Source                          Destination                     Connected?
  ------------------------------  ------------------------------  ----------------
  gridtest1:192.168.0.30          gridtest2:192.168.0.31          passed
  gridtest1:192.168.0.30          gridtest2:192.168.0.36          passed
  gridtest1:192.168.0.30          gridtest1:192.168.0.38          passed
  gridtest1:192.168.0.30          gridtest1:192.168.0.35          passed
Result: TCP connectivity check passed for subnet "192.168.0.0"


Check: Node connectivity for interface "eth1"
  Source                          Destination                     Connected?
  ------------------------------  ------------------------------  ----------------
  gridtest2[10.10.10.31]          gridtest1[10.10.10.30]          yes
Result: Node connectivity passed for interface "eth1"


Check: TCP connectivity of subnet "10.10.10.0"
  Source                          Destination                     Connected?
  ------------------------------  ------------------------------  ----------------
  gridtest1:10.10.10.30           gridtest2:10.10.10.31           passed
Result: TCP connectivity check passed for subnet "10.10.10.0"

Checking subnet mask consistency...
Subnet mask consistency check passed for subnet "192.168.0.0".
Subnet mask consistency check passed for subnet "10.10.10.0".
Subnet mask consistency check passed.

Result: Node connectivity check passed

Checking multicast communication...

Checking subnet "192.168.0.0" for multicast communication with multicast group "230.0.1.0"...
Check of subnet "192.168.0.0" for multicast communication with multicast group "230.0.1.0" passed.

Checking subnet "10.10.10.0" for multicast communication with multicast group "230.0.1.0"...
Check of subnet "10.10.10.0" for multicast communication with multicast group "230.0.1.0" passed.

Check of multicast communication passed.

Checking OCR integrity...

OCR integrity check passed

Checking ASMLib configuration.
  Node Name                             Status
  ------------------------------------  ------------------------
  gridtest2                             passed
  gridtest1                             passed
Result: Check for ASMLib configuration passed.

Check: Total memory
  Node Name     Available                 Required                  Status
  ------------  ------------------------  ------------------------  ----------
  gridtest2     1.5932GB (1670600.0KB)    1.5GB (1572864.0KB)       passed
  gridtest1     1.5932GB (1670600.0KB)    1.5GB (1572864.0KB)       passed
Result: Total memory check passed

Check: Available memory
  Node Name     Available                 Required                  Status
  ------------  ------------------------  ------------------------  ----------
  gridtest2     431.457MB (441812.0KB)    50MB (51200.0KB)          passed
  gridtest1     380.0625MB (389184.0KB)   50MB (51200.0KB)          passed
Result: Available memory check passed

Check: Swap space
  Node Name     Available                 Required                  Status
  ------------  ------------------------  ------------------------  ----------
  gridtest2     3.1867GB (3341512.0KB)    2.3898GB (2505900.0KB)    passed
  gridtest1     3.1867GB (3341512.0KB)    2.3898GB (2505900.0KB)    passed
Result: Swap space check passed

Check: Free disk space for "gridtest2:/grid/11.2.0.3,gridtest2:/tmp"
  Path              Node Name     Mount point   Available     Required      Status
  ----------------  ------------  ------------  ------------  ------------  ------------
  /grid/11.2.0.3    gridtest2     /             12.3008GB     7.5GB         passed
  /tmp              gridtest2     /             12.3008GB     7.5GB         passed
Result: Free disk space check passed for "gridtest2:/grid/11.2.0.3,gridtest2:/tmp"

Check: Free disk space for "gridtest1:/grid/11.2.0.3,gridtest1:/tmp"
  Path              Node Name     Mount point   Available     Required      Status
  ----------------  ------------  ------------  ------------  ------------  ------------
  /grid/11.2.0.3    gridtest1     /             12.137GB      7.5GB         passed
  /tmp              gridtest1     /             12.137GB      7.5GB         passed
Result: Free disk space check passed for "gridtest1:/grid/11.2.0.3,gridtest1:/tmp"

Check: User existence for "oracle"
  Node Name     Status                    Comment
  ------------  ------------------------  ------------------------
  gridtest2     passed                    exists(501)
  gridtest1     passed                    exists(501)

Checking for multiple users with UID value 501
Result: Check for multiple users with UID value 501 passed
Result: User existence check passed for "oracle"

Check: Group existence for "oinstall"
  Node Name     Status                    Comment
  ------------  ------------------------  ------------------------
  gridtest2     passed                    exists
  gridtest1     passed                    exists
Result: Group existence check passed for "oinstall"

Check: Membership of user "oracle" in group "oinstall" [as Primary]
  Node Name         User Exists   Group Exists  User in Group  Primary       Status
  ----------------  ------------  ------------  ------------  ------------  ------------
  gridtest2         yes           yes           yes           yes           passed
  gridtest1         yes           yes           yes           yes           passed
Result: Membership check for user "oracle" in group "oinstall" [as Primary] passed

Check: Run level
  Node Name     run level                 Required                  Status
  ------------  ------------------------  ------------------------  ----------
  gridtest2     5                         3,5                       passed
  gridtest1     5                         3,5                       passed
Result: Run level check passed

Check: Hard limits for "maximum open file descriptors"
  Node Name         Type          Available     Required      Status
  ----------------  ------------  ------------  ------------  ----------------
  gridtest2         hard          65536         65536         passed
  gridtest1         hard          65536         65536         passed
Result: Hard limits check passed for "maximum open file descriptors"

Check: Soft limits for "maximum open file descriptors"
  Node Name         Type          Available     Required      Status
  ----------------  ------------  ------------  ------------  ----------------
  gridtest2         soft          1024          1024          passed
  gridtest1         soft          1024          1024          passed
Result: Soft limits check passed for "maximum open file descriptors"

Check: Hard limits for "maximum user processes"
  Node Name         Type          Available     Required      Status
  ----------------  ------------  ------------  ------------  ----------------
  gridtest2         hard          16384         16384         passed
  gridtest1         hard          16384         16384         passed
Result: Hard limits check passed for "maximum user processes"

Check: Soft limits for "maximum user processes"
  Node Name         Type          Available     Required      Status
  ----------------  ------------  ------------  ------------  ----------------
  gridtest2         soft          2047          2047          passed
  gridtest1         soft          2047          2047          passed
Result: Soft limits check passed for "maximum user processes"

Checking for Oracle patch "9413827" in home "/grid/11.2".
  Node Name     Applied                   Required                  Comment
  ------------  ------------------------  ------------------------  ----------
  gridtest2     missing                   9413827                   failed
  gridtest1     missing                   9413827                   failed
Result: Check for Oracle patch "9413827" in home "/grid/11.2" failed

There are no oracle patches required for home "/grid/11.2.0.3".

Check: System architecture
  Node Name     Available                 Required                  Status
  ------------  ------------------------  ------------------------  ----------
  gridtest2     x86_64                    x86_64                    passed
  gridtest1     x86_64                    x86_64                    passed
Result: System architecture check passed

Check: Kernel version
  Node Name     Available                 Required                  Status
  ------------  ------------------------  ------------------------  ----------
  gridtest2     2.6.18-194.el5            2.6.18                    passed
  gridtest1     2.6.18-194.el5            2.6.18                    passed
Result: Kernel version check passed

Check: Kernel parameter for "semmsl"
  Node Name         Current       Configured    Required      Status        Comment
  ----------------  ------------  ------------  ------------  ------------  ------------
  gridtest2         250           250           250           passed
  gridtest1         250           250           250           passed
Result: Kernel parameter check passed for "semmsl"

Check: Kernel parameter for "semmns"
  Node Name         Current       Configured    Required      Status        Comment
  ----------------  ------------  ------------  ------------  ------------  ------------
  gridtest2         32000         32000         32000         passed
  gridtest1         32000         32000         32000         passed
Result: Kernel parameter check passed for "semmns"

Check: Kernel parameter for "semopm"
  Node Name         Current       Configured    Required      Status        Comment
  ----------------  ------------  ------------  ------------  ------------  ------------
  gridtest2         100           100           100           passed
  gridtest1         100           100           100           passed
Result: Kernel parameter check passed for "semopm"

Check: Kernel parameter for "semmni"
  Node Name         Current       Configured    Required      Status        Comment
  ----------------  ------------  ------------  ------------  ------------  ------------
  gridtest2         128           128           128           passed
  gridtest1         128           128           128           passed
Result: Kernel parameter check passed for "semmni"

Check: Kernel parameter for "shmmax"
  Node Name         Current       Configured    Required      Status        Comment
  ----------------  ------------  ------------  ------------  ------------  ------------
  gridtest2         4294967295    4294967295    855347200     passed
  gridtest1         4294967295    4294967295    855347200     passed
Result: Kernel parameter check passed for "shmmax"

Check: Kernel parameter for "shmmni"
  Node Name         Current       Configured    Required      Status        Comment
  ----------------  ------------  ------------  ------------  ------------  ------------
  gridtest2         4096          unknown       4096          failed        Configured value too low.
  gridtest1         4096          unknown       4096          failed        Configured value too low.
Result: Kernel parameter check failed for "shmmni"

Check: Kernel parameter for "shmall"
  Node Name         Current       Configured    Required      Status        Comment
  ----------------  ------------  ------------  ------------  ------------  ------------
  gridtest2         4294967296    4294967296    2097152       passed
  gridtest1         4294967296    4294967296    2097152       passed
Result: Kernel parameter check passed for "shmall"

Check: Kernel parameter for "file-max"
  Node Name         Current       Configured    Required      Status        Comment
  ----------------  ------------  ------------  ------------  ------------  ------------
  gridtest2         6815744       6815744       6815744       passed
  gridtest1         6815744       6815744       6815744       passed
Result: Kernel parameter check passed for "file-max"

Check: Kernel parameter for "ip_local_port_range"
  Node Name         Current       Configured    Required      Status        Comment
  ----------------  ------------  ------------  ------------  ------------  ------------
  gridtest2         between 9000.0 & 65500.0  between 9000.0 & 65500.0  between 9000.0 & 65500.0  passed
  gridtest1         between 9000.0 & 65500.0  between 9000.0 & 65500.0  between 9000.0 & 65500.0  passed
Result: Kernel parameter check passed for "ip_local_port_range"

Check: Kernel parameter for "rmem_default"
  Node Name         Current       Configured    Required      Status        Comment
  ----------------  ------------  ------------  ------------  ------------  ------------
  gridtest2         262144        262144        262144        passed
  gridtest1         262144        262144        262144        passed
Result: Kernel parameter check passed for "rmem_default"

Check: Kernel parameter for "rmem_max"
  Node Name         Current       Configured    Required      Status        Comment
  ----------------  ------------  ------------  ------------  ------------  ------------
  gridtest2         4194304       4194304       4194304       passed
  gridtest1         4194304       4194304       4194304       passed
Result: Kernel parameter check passed for "rmem_max"

Check: Kernel parameter for "wmem_default"
  Node Name         Current       Configured    Required      Status        Comment
  ----------------  ------------  ------------  ------------  ------------  ------------
  gridtest2         262144        262144        262144        passed
  gridtest1         262144        262144        262144        passed
Result: Kernel parameter check passed for "wmem_default"

Check: Kernel parameter for "wmem_max"
  Node Name         Current       Configured    Required      Status        Comment
  ----------------  ------------  ------------  ------------  ------------  ------------
  gridtest2         1048586       1048586       1048576       passed
  gridtest1         1048586       1048586       1048576       passed
Result: Kernel parameter check passed for "wmem_max"

Check: Kernel parameter for "aio-max-nr"
  Node Name         Current       Configured    Required      Status        Comment
  ----------------  ------------  ------------  ------------  ------------  ------------
  gridtest2         1048576       1048576       1048576       passed
  gridtest1         1048576       1048576       1048576       passed
Result: Kernel parameter check passed for "aio-max-nr"

Check: Package existence for "make"
  Node Name     Available                 Required                  Status
  ------------  ------------------------  ------------------------  ----------
  gridtest2     make-3.81-3.el5           make-3.81                 passed
  gridtest1     make-3.81-3.el5           make-3.81                 passed
Result: Package existence check passed for "make"

Check: Package existence for "binutils"
  Node Name     Available                 Required                  Status
  ------------  ------------------------  ------------------------  ----------
  gridtest2     binutils-2.17.50.0.6-14.el5  binutils-2.17.50.0.6      passed
  gridtest1     binutils-2.17.50.0.6-14.el5  binutils-2.17.50.0.6      passed
Result: Package existence check passed for "binutils"

Check: Package existence for "gcc(x86_64)"
  Node Name     Available                 Required                  Status
  ------------  ------------------------  ------------------------  ----------
  gridtest2     gcc(x86_64)-4.1.2-48.el5  gcc(x86_64)-4.1.2         passed
  gridtest1     gcc(x86_64)-4.1.2-48.el5  gcc(x86_64)-4.1.2         passed
Result: Package existence check passed for "gcc(x86_64)"

Check: Package existence for "libaio(x86_64)"
  Node Name     Available                 Required                  Status
  ------------  ------------------------  ------------------------  ----------
  gridtest2     libaio(x86_64)-0.3.106-5  libaio(x86_64)-0.3.106    passed
  gridtest1     libaio(x86_64)-0.3.106-5  libaio(x86_64)-0.3.106    passed
Result: Package existence check passed for "libaio(x86_64)"

Check: Package existence for "glibc(x86_64)"
  Node Name     Available                 Required                  Status
  ------------  ------------------------  ------------------------  ----------
  gridtest2     glibc(x86_64)-2.5-49      glibc(x86_64)-2.5-24      passed
  gridtest1     glibc(x86_64)-2.5-49      glibc(x86_64)-2.5-24      passed
Result: Package existence check passed for "glibc(x86_64)"

Check: Package existence for "compat-libstdc++-33(x86_64)"
  Node Name     Available                 Required                  Status
  ------------  ------------------------  ------------------------  ----------
  gridtest2     compat-libstdc++-33(x86_64)-3.2.3-61  compat-libstdc++-33(x86_64)-3.2.3  passed
  gridtest1     compat-libstdc++-33(x86_64)-3.2.3-61  compat-libstdc++-33(x86_64)-3.2.3  passed
Result: Package existence check passed for "compat-libstdc++-33(x86_64)"

Check: Package existence for "elfutils-libelf(x86_64)"
  Node Name     Available                 Required                  Status
  ------------  ------------------------  ------------------------  ----------
  gridtest2     elfutils-libelf(x86_64)-0.137-3.el5  elfutils-libelf(x86_64)-0.125  passed
  gridtest1     elfutils-libelf(x86_64)-0.137-3.el5  elfutils-libelf(x86_64)-0.125  passed
Result: Package existence check passed for "elfutils-libelf(x86_64)"

Check: Package existence for "elfutils-libelf-devel"
  Node Name     Available                 Required                  Status
  ------------  ------------------------  ------------------------  ----------
  gridtest2     elfutils-libelf-devel-0.125-3.el5  elfutils-libelf-devel-0.125  passed
  gridtest1     elfutils-libelf-devel-0.125-3.el5  elfutils-libelf-devel-0.125  passed
Result: Package existence check passed for "elfutils-libelf-devel"

Check: Package existence for "glibc-common"
  Node Name     Available                 Required                  Status
  ------------  ------------------------  ------------------------  ----------
  gridtest2     glibc-common-2.5-49       glibc-common-2.5          passed
  gridtest1     glibc-common-2.5-49       glibc-common-2.5          passed
Result: Package existence check passed for "glibc-common"

Check: Package existence for "glibc-devel(x86_64)"
  Node Name     Available                 Required                  Status
  ------------  ------------------------  ------------------------  ----------
  gridtest2     glibc-devel(x86_64)-2.5-49  glibc-devel(x86_64)-2.5   passed
  gridtest1     glibc-devel(x86_64)-2.5-49  glibc-devel(x86_64)-2.5   passed
Result: Package existence check passed for "glibc-devel(x86_64)"

Check: Package existence for "glibc-headers"
  Node Name     Available                 Required                  Status
  ------------  ------------------------  ------------------------  ----------
  gridtest2     glibc-headers-2.5-49      glibc-headers-2.5         passed
  gridtest1     glibc-headers-2.5-49      glibc-headers-2.5         passed
Result: Package existence check passed for "glibc-headers"

Check: Package existence for "gcc-c++(x86_64)"
  Node Name     Available                 Required                  Status
  ------------  ------------------------  ------------------------  ----------
  gridtest2     gcc-c++(x86_64)-4.1.2-48.el5  gcc-c++(x86_64)-4.1.2     passed
  gridtest1     gcc-c++(x86_64)-4.1.2-48.el5  gcc-c++(x86_64)-4.1.2     passed
Result: Package existence check passed for "gcc-c++(x86_64)"

Check: Package existence for "libaio-devel(x86_64)"
  Node Name     Available                 Required                  Status
  ------------  ------------------------  ------------------------  ----------
  gridtest2     libaio-devel(x86_64)-0.3.106-3.2  libaio-devel(x86_64)-0.3.106  passed
  gridtest1     libaio-devel(x86_64)-0.3.106-3.2  libaio-devel(x86_64)-0.3.106  passed
Result: Package existence check passed for "libaio-devel(x86_64)"

Check: Package existence for "libgcc(x86_64)"
  Node Name     Available                 Required                  Status
  ------------  ------------------------  ------------------------  ----------
  gridtest2     libgcc(x86_64)-4.1.2-48.el5  libgcc(x86_64)-4.1.2      passed
  gridtest1     libgcc(x86_64)-4.1.2-48.el5  libgcc(x86_64)-4.1.2      passed
Result: Package existence check passed for "libgcc(x86_64)"

Check: Package existence for "libstdc++(x86_64)"
  Node Name     Available                 Required                  Status
  ------------  ------------------------  ------------------------  ----------
  gridtest2     libstdc++(x86_64)-4.1.2-48.el5  libstdc++(x86_64)-4.1.2   passed
  gridtest1     libstdc++(x86_64)-4.1.2-48.el5  libstdc++(x86_64)-4.1.2   passed
Result: Package existence check passed for "libstdc++(x86_64)"

Check: Package existence for "libstdc++-devel(x86_64)"
  Node Name     Available                 Required                  Status
  ------------  ------------------------  ------------------------  ----------
  gridtest2     libstdc++-devel(x86_64)-4.1.2-48.el5  libstdc++-devel(x86_64)-4.1.2  passed
  gridtest1     libstdc++-devel(x86_64)-4.1.2-48.el5  libstdc++-devel(x86_64)-4.1.2  passed
Result: Package existence check passed for "libstdc++-devel(x86_64)"

Check: Package existence for "sysstat"
  Node Name     Available                 Required                  Status
  ------------  ------------------------  ------------------------  ----------
  gridtest2     sysstat-7.0.2-1.el5       sysstat-7.0.2             passed
  gridtest1     sysstat-7.0.2-1.el5       sysstat-7.0.2             passed

WARNING:
PRVF-7584 : Multiple versions of package "sysstat" found on node gridtest2: sysstat(x86_64)-7.0.2-1.el5,sysstat(i386)-7.0.2-1.el5

WARNING:
PRVF-7584 : Multiple versions of package "sysstat" found on node gridtest1: sysstat(x86_64)-7.0.2-1.el5,sysstat(i386)-7.0.2-1.el5
Result: Package existence check passed for "sysstat"

Check: Package existence for "ksh"
  Node Name     Available                 Required                  Status
  ------------  ------------------------  ------------------------  ----------
  gridtest2     ksh-20100202-1.el5        ksh-20060214              passed
  gridtest1     ksh-20100202-1.el5        ksh-20060214              passed
Result: Package existence check passed for "ksh"

Checking for multiple users with UID value 0
Result: Check for multiple users with UID value 0 passed

Check: Current group ID
Result: Current group ID check passed

Starting check for consistency of primary group of root user
  Node Name                             Status
  ------------------------------------  ------------------------
  gridtest2                             passed
  gridtest1                             passed

Check for consistency of root user's primary group passed

Starting Clock synchronization checks using Network Time Protocol(NTP)...

NTP Configuration file check started...
The NTP configuration file "/etc/ntp.conf" is available on all nodes
NTP Configuration file check passed

Checking daemon liveness...

Check: Liveness for "ntpd"
  Node Name                             Running?
  ------------------------------------  ------------------------
  gridtest2                             yes
  gridtest1                             yes
Result: Liveness check passed for "ntpd"
Check for NTP daemon or service alive passed on all nodes

Checking NTP daemon command line for slewing option "-x"
Check: NTP daemon command line
  Node Name                             Slewing Option Set?
  ------------------------------------  ------------------------
  gridtest2                             yes
  gridtest1                             yes
Result:
NTP daemon slewing option check passed

Checking NTP daemon's boot time configuration, in file "/etc/sysconfig/ntpd", for slewing option "-x"

Check: NTP daemon's boot time configuration
  Node Name                             Slewing Option Set?
  ------------------------------------  ------------------------
  gridtest2                             yes
  gridtest1                             yes
Result:
NTP daemon's boot time configuration check for slewing option passed

Checking whether NTP daemon or service is using UDP port 123 on all nodes

Check for NTP daemon or service using UDP port 123
  Node Name                             Port Open?
  ------------------------------------  ------------------------
  gridtest2                             yes
  gridtest1                             yes

NTP common Time Server Check started...
PRVF-5408 : NTP Time Server "124.82.204.2" is common only to the following nodes "gridtest1"
NTP Time Server ".LOCL." is common to all nodes on which the NTP daemon is running
Check of common NTP Time Server passed

Clock time offset check from NTP Time Server started...
Checking on nodes "[gridtest2, gridtest1]"...
Check: Clock time offset from NTP Time Server

Time Server: .LOCL.
Time Offset Limit: 1000.0 msecs
  Node Name     Time Offset               Status
  ------------  ------------------------  ------------------------
  gridtest2     0.0                       passed
  gridtest1     0.0                       passed
Time Server ".LOCL." has time offsets that are within permissible limits for nodes "[gridtest2, gridtest1]".
Clock time offset check passed

Result: Clock synchronization check using Network Time Protocol(NTP) passed

Checking Core file name pattern consistency...
Core file name pattern consistency check passed.

Checking to make sure user "oracle" is not in "root" group
  Node Name     Status                    Comment
  ------------  ------------------------  ------------------------
  gridtest2     passed                    does not exist
  gridtest1     passed                    does not exist
Result: User "oracle" is not part of "root" group. Check passed

Check default user file creation mask
  Node Name     Available                 Required                  Comment
  ------------  ------------------------  ------------------------  ----------
  gridtest2     0022                      0022                      passed
  gridtest1     0022                      0022                      passed
Result: Default user file creation mask check passed
Checking consistency of file "/etc/resolv.conf" across nodes

Checking the file "/etc/resolv.conf" to make sure only one of domain and search entries is defined
File "/etc/resolv.conf" does not have both domain and search entries defined
Checking if domain entry in file "/etc/resolv.conf" is consistent across the nodes...
domain entry in file "/etc/resolv.conf" is consistent across nodes
Checking if search entry in file "/etc/resolv.conf" is consistent across the nodes...
PRVF-5622 : search entry does not exist in file "/etc/resolv.conf" on nodes: "gridtest1"
Checking file "/etc/resolv.conf" to make sure that only one search entry is defined
All nodes have one search entry defined in file "/etc/resolv.conf"
Checking DNS response time for an unreachable node
  Node Name                             Status
  ------------------------------------  ------------------------
  gridtest2                             failed
  gridtest1                             passed
PRVF-5636 : The DNS response time for an unreachable node exceeded "15000" ms on following nodes: gridtest2

File "/etc/resolv.conf" is not consistent across nodes


UDev attributes check for OCR locations started...
Result: UDev attributes check passed for OCR locations


UDev attributes check for Voting Disk locations started...
Result: UDev attributes check passed for Voting Disk locations

Check: Time zone consistency
Result: Time zone consistency check passed
Checking VIP configuration.
Checking VIP Subnet configuration.
Check for VIP Subnet configuration passed.
Checking VIP reachability
Check for VIP reachability passed.

Checking Oracle Cluster Voting Disk configuration...

ASM Running check passed. ASM is running on all specified nodes

Oracle Cluster Voting Disk configuration check passed

Clusterware version consistency passed
Fixup information has been generated for following node(s):
gridtest1,gridtest2
Please run the following script on each node as "root" user to execute the fixups:
'/tmp/CVU_11.2.0.3.0_oracle/runfixup.sh'

Pre-check for cluster services setup was unsuccessful on all the nodes.





2) The pre-check has been failed with the below errors.

a)

Check: Kernel parameter for "shmmni"
  Node Name         Current       Configured    Required      Status        Comment
  ----------------  ------------  ------------  ------------  ------------  ------------
  gridtest2         4096          unknown       4096          failed        Configured value too low.
  gridtest1         4096          unknown       4096          failed        Configured value too low.


b) Checking for Oracle patch "9413827" in home "/grid/11.2".
  Node Name     Applied                   Required                  Comment
  ------------  ------------------------  ------------------------  ----------
  gridtest2     missing                   9413827                   failed
  gridtest1     missing                   9413827                   failed
Result: Check for Oracle patch "9413827" in home "/grid/11.2" failed

c) Checking the file "/etc/resolv.conf" to make sure only one of domain and search entries is defined
File "/etc/resolv.conf" does not have both domain and search entries defined
Checking if domain entry in file "/etc/resolv.conf" is consistent across the nodes...
domain entry in file "/etc/resolv.conf" is consistent across nodes
Checking if search entry in file "/etc/resolv.conf" is consistent across the nodes...
PRVF-5622 : search entry does not exist in file "/etc/resolv.conf" on nodes: "gridtest1"
Checking file "/etc/resolv.conf" to make sure that only one search entry is defined
All nodes have one search entry defined in file "/etc/resolv.conf"
Checking DNS response time for an unreachable node
  Node Name                             Status
  ------------------------------------  ------------------------
  gridtest2                             failed
  gridtest1                             passed
PRVF-5636 : The DNS response time for an unreachable node exceeded "15000" ms on following nodes: gridtest2

File "/etc/resolv.conf" is not consistent across nodes


3) Fix all the above said errors.

1) Checked and updated the kernel.shmmni correctly.
2) Apply the patch 9413827 in grid home /grid/11.2
3) Check the consistency of /etc/resolve.conf file.

4) Run the cluvfy utility again to check still you face any errors.

       [oracle@gridtest1 grid]$ ./runcluvfy.sh stage -pre crsinst -upgrade -n gridtest1,gridtest2 -src_crshome /grid/11.2 -dest_crshome /grid/11.2.0.3 -dest_version 11.2.0.3.0 -fixup -fixupdir /home/oracle -verbose

Performing pre-checks for cluster services setup

Checking node reachability...

Check: Node reachability from node "gridtest1"
  Destination Node                      Reachable?
  ------------------------------------  ------------------------
  gridtest1                             yes
  gridtest2                             yes
Result: Node reachability check passed from node "gridtest1"


Checking user equivalence...

Check: User equivalence for user "oracle"
  Node Name                             Status
  ------------------------------------  ------------------------
  gridtest2                             passed
  gridtest1                             passed
Result: User equivalence check passed for user "oracle"

Checking CRS user consistency
Result: CRS user consistency check successful

Checking node connectivity...

Checking hosts config file...
  Node Name                             Status
  ------------------------------------  ------------------------
  gridtest2                             passed
  gridtest1                             passed

Verification of the hosts config file successful


Interface information for node "gridtest2"
 Name   IP Address      Subnet          Gateway         Def. Gateway    HW Address        MTU
 ------ --------------- --------------- --------------- --------------- ----------------- ------
 eth0   192.168.0.31    192.168.0.0     0.0.0.0         192.168.0.1     00:0C:29:6B:A6:76 1500
 eth0   192.168.0.36    192.168.0.0     0.0.0.0         192.168.0.1     00:0C:29:6B:A6:76 1500
 eth1   10.10.10.31     10.10.10.0      0.0.0.0         192.168.0.1     00:0C:29:6B:A6:80 1500


Interface information for node "gridtest1"
 Name   IP Address      Subnet          Gateway         Def. Gateway    HW Address        MTU
 ------ --------------- --------------- --------------- --------------- ----------------- ------
 eth0   192.168.0.30    192.168.0.0     0.0.0.0         192.168.0.1     00:0C:29:B0:4B:8F 1500
 eth0   192.168.0.35    192.168.0.0     0.0.0.0         192.168.0.1     00:0C:29:B0:4B:8F 1500
 eth0   192.168.0.38    192.168.0.0     0.0.0.0         192.168.0.1     00:0C:29:B0:4B:8F 1500
 eth1   10.10.10.30     10.10.10.0      0.0.0.0         192.168.0.1     00:0C:29:B0:4B:99 1500


Check: Node connectivity for interface "eth0"
  Source                          Destination                     Connected?
  ------------------------------  ------------------------------  ----------------
  gridtest2[192.168.0.31]         gridtest2[192.168.0.36]         yes
  gridtest2[192.168.0.31]         gridtest1[192.168.0.30]         yes
  gridtest2[192.168.0.31]         gridtest1[192.168.0.35]         yes
  gridtest2[192.168.0.31]         gridtest1[192.168.0.38]         yes
  gridtest2[192.168.0.36]         gridtest1[192.168.0.30]         yes
  gridtest2[192.168.0.36]         gridtest1[192.168.0.35]         yes
  gridtest2[192.168.0.36]         gridtest1[192.168.0.38]         yes
  gridtest1[192.168.0.30]         gridtest1[192.168.0.35]         yes
  gridtest1[192.168.0.30]         gridtest1[192.168.0.38]         yes
  gridtest1[192.168.0.35]         gridtest1[192.168.0.38]         yes
Result: Node connectivity passed for interface "eth0"


Check: TCP connectivity of subnet "192.168.0.0"
  Source                          Destination                     Connected?
  ------------------------------  ------------------------------  ----------------
  gridtest1:192.168.0.30          gridtest2:192.168.0.31          passed
  gridtest1:192.168.0.30          gridtest2:192.168.0.36          passed
  gridtest1:192.168.0.30          gridtest1:192.168.0.35          passed
  gridtest1:192.168.0.30          gridtest1:192.168.0.38          passed
Result: TCP connectivity check passed for subnet "192.168.0.0"


Check: Node connectivity for interface "eth1"
  Source                          Destination                     Connected?
  ------------------------------  ------------------------------  ----------------
  gridtest2[10.10.10.31]          gridtest1[10.10.10.30]          yes
Result: Node connectivity passed for interface "eth1"


Check: TCP connectivity of subnet "10.10.10.0"
  Source                          Destination                     Connected?
  ------------------------------  ------------------------------  ----------------
  gridtest1:10.10.10.30           gridtest2:10.10.10.31           passed
Result: TCP connectivity check passed for subnet "10.10.10.0"

Checking subnet mask consistency...
Subnet mask consistency check passed for subnet "192.168.0.0".
Subnet mask consistency check passed for subnet "10.10.10.0".
Subnet mask consistency check passed.

Result: Node connectivity check passed

Checking multicast communication...

Checking subnet "192.168.0.0" for multicast communication with multicast group "230.0.1.0"...
Check of subnet "192.168.0.0" for multicast communication with multicast group "230.0.1.0" passed.

Checking subnet "10.10.10.0" for multicast communication with multicast group "230.0.1.0"...
Check of subnet "10.10.10.0" for multicast communication with multicast group "230.0.1.0" passed.

Check of multicast communication passed.

Checking OCR integrity...

OCR integrity check passed

Checking ASMLib configuration.
  Node Name                             Status
  ------------------------------------  ------------------------
  gridtest2                             passed
  gridtest1                             passed
Result: Check for ASMLib configuration passed.

Check: Total memory
  Node Name     Available                 Required                  Status
  ------------  ------------------------  ------------------------  ----------
  gridtest2     1.5932GB (1670600.0KB)    1.5GB (1572864.0KB)       passed
  gridtest1     1.5932GB (1670600.0KB)    1.5GB (1572864.0KB)       passed
Result: Total memory check passed

Check: Available memory
  Node Name     Available                 Required                  Status
  ------------  ------------------------  ------------------------  ----------
  gridtest2     703.5742MB (720460.0KB)   50MB (51200.0KB)          passed
  gridtest1     640.5391MB (655912.0KB)   50MB (51200.0KB)          passed
Result: Available memory check passed

Check: Swap space
  Node Name     Available                 Required                  Status
  ------------  ------------------------  ------------------------  ----------
  gridtest2     3.1867GB (3341512.0KB)    2.3898GB (2505900.0KB)    passed
  gridtest1     3.1867GB (3341512.0KB)    2.3898GB (2505900.0KB)    passed
Result: Swap space check passed

Check: Free disk space for "gridtest2:/grid/11.2.0.3,gridtest2:/tmp"
  Path              Node Name     Mount point   Available     Required      Status
  ----------------  ------------  ------------  ------------  ------------  ------------
  /grid/11.2.0.3    gridtest2     /             10.4902GB     7.5GB         passed
  /tmp              gridtest2     /             10.4902GB     7.5GB         passed
Result: Free disk space check passed for "gridtest2:/grid/11.2.0.3,gridtest2:/tmp"

Check: Free disk space for "gridtest1:/grid/11.2.0.3,gridtest1:/tmp"
  Path              Node Name     Mount point   Available     Required      Status
  ----------------  ------------  ------------  ------------  ------------  ------------
  /grid/11.2.0.3    gridtest1     /             8.4503GB      7.5GB         passed
  /tmp              gridtest1     /             8.4503GB      7.5GB         passed
Result: Free disk space check passed for "gridtest1:/grid/11.2.0.3,gridtest1:/tmp"

Check: User existence for "oracle"
  Node Name     Status                    Comment
  ------------  ------------------------  ------------------------
  gridtest2     passed                    exists(501)
  gridtest1     passed                    exists(501)

Checking for multiple users with UID value 501
Result: Check for multiple users with UID value 501 passed
Result: User existence check passed for "oracle"

Check: Group existence for "oinstall"
  Node Name     Status                    Comment
  ------------  ------------------------  ------------------------
  gridtest2     passed                    exists
  gridtest1     passed                    exists
Result: Group existence check passed for "oinstall"

Check: Membership of user "oracle" in group "oinstall" [as Primary]
  Node Name         User Exists   Group Exists  User in Group  Primary       Status
  ----------------  ------------  ------------  ------------  ------------  ------------
  gridtest2         yes           yes           yes           yes           passed
  gridtest1         yes           yes           yes           yes           passed
Result: Membership check for user "oracle" in group "oinstall" [as Primary] passed

Check: Run level
  Node Name     run level                 Required                  Status
  ------------  ------------------------  ------------------------  ----------
  gridtest2     5                         3,5                       passed
  gridtest1     5                         3,5                       passed
Result: Run level check passed

Check: Hard limits for "maximum open file descriptors"
  Node Name         Type          Available     Required      Status
  ----------------  ------------  ------------  ------------  ----------------
  gridtest2         hard          65536         65536         passed
  gridtest1         hard          65536         65536         passed
Result: Hard limits check passed for "maximum open file descriptors"

Check: Soft limits for "maximum open file descriptors"
  Node Name         Type          Available     Required      Status
  ----------------  ------------  ------------  ------------  ----------------
  gridtest2         soft          1024          1024          passed
  gridtest1         soft          1024          1024          passed
Result: Soft limits check passed for "maximum open file descriptors"

Check: Hard limits for "maximum user processes"
  Node Name         Type          Available     Required      Status
  ----------------  ------------  ------------  ------------  ----------------
  gridtest2         hard          16384         16384         passed
  gridtest1         hard          16384         16384         passed
Result: Hard limits check passed for "maximum user processes"

Check: Soft limits for "maximum user processes"
  Node Name         Type          Available     Required      Status
  ----------------  ------------  ------------  ------------  ----------------
  gridtest2         soft          2047          2047          passed
  gridtest1         soft          2047          2047          passed
Result: Soft limits check passed for "maximum user processes"

Checking for Oracle patch "9413827" in home "/grid/11.2".
  Node Name     Applied                   Required                  Comment
  ------------  ------------------------  ------------------------  ----------
  gridtest2     9413827                   9413827                   passed
  gridtest1     9413827                   9413827                   passed
Result: Check for Oracle patch "9413827" in home "/grid/11.2" passed

There are no oracle patches required for home "/grid/11.2.0.3".

Check: System architecture
  Node Name     Available                 Required                  Status
  ------------  ------------------------  ------------------------  ----------
  gridtest2     x86_64                    x86_64                    passed
  gridtest1     x86_64                    x86_64                    passed
Result: System architecture check passed

Check: Kernel version
  Node Name     Available                 Required                  Status
  ------------  ------------------------  ------------------------  ----------
  gridtest2     2.6.18-194.el5            2.6.18                    passed
  gridtest1     2.6.18-194.el5            2.6.18                    passed
Result: Kernel version check passed

Check: Kernel parameter for "semmsl"
  Node Name         Current       Configured    Required      Status        Comment
  ----------------  ------------  ------------  ------------  ------------  ------------
  gridtest2         250           250           250           passed
  gridtest1         250           250           250           passed
Result: Kernel parameter check passed for "semmsl"

Check: Kernel parameter for "semmns"
  Node Name         Current       Configured    Required      Status        Comment
  ----------------  ------------  ------------  ------------  ------------  ------------
  gridtest2         32000         32000         32000         passed
  gridtest1         32000         32000         32000         passed
Result: Kernel parameter check passed for "semmns"

Check: Kernel parameter for "semopm"
  Node Name         Current       Configured    Required      Status        Comment
  ----------------  ------------  ------------  ------------  ------------  ------------
  gridtest2         100           100           100           passed
  gridtest1         100           100           100           passed
Result: Kernel parameter check passed for "semopm"

Check: Kernel parameter for "semmni"
  Node Name         Current       Configured    Required      Status        Comment
  ----------------  ------------  ------------  ------------  ------------  ------------
  gridtest2         128           128           128           passed
  gridtest1         128           128           128           passed
Result: Kernel parameter check passed for "semmni"

Check: Kernel parameter for "shmmax"
  Node Name         Current       Configured    Required      Status        Comment
  ----------------  ------------  ------------  ------------  ------------  ------------
  gridtest2         4294967295    4294967295    855347200     passed
  gridtest1         4294967295    4294967295    855347200     passed
Result: Kernel parameter check passed for "shmmax"

Check: Kernel parameter for "shmmni"
  Node Name         Current       Configured    Required      Status        Comment
  ----------------  ------------  ------------  ------------  ------------  ------------
  gridtest2         4096          4096          4096          passed
  gridtest1         4096          4096          4096          passed
Result: Kernel parameter check passed for "shmmni"

Check: Kernel parameter for "shmall"
  Node Name         Current       Configured    Required      Status        Comment
  ----------------  ------------  ------------  ------------  ------------  ------------
  gridtest2         4294967296    4294967296    2097152       passed
  gridtest1         4294967296    4294967296    2097152       passed
Result: Kernel parameter check passed for "shmall"

Check: Kernel parameter for "file-max"
  Node Name         Current       Configured    Required      Status        Comment
  ----------------  ------------  ------------  ------------  ------------  ------------
  gridtest2         6815744       6815744       6815744       passed
  gridtest1         6815744       6815744       6815744       passed
Result: Kernel parameter check passed for "file-max"

Check: Kernel parameter for "ip_local_port_range"
  Node Name         Current       Configured    Required      Status        Comment
  ----------------  ------------  ------------  ------------  ------------  ------------
  gridtest2         between 9000.0 & 65500.0  between 9000.0 & 65500.0  between 9000.0 & 65500.0  passed
  gridtest1         between 9000.0 & 65500.0  between 9000.0 & 65500.0  between 9000.0 & 65500.0  passed
Result: Kernel parameter check passed for "ip_local_port_range"

Check: Kernel parameter for "rmem_default"
  Node Name         Current       Configured    Required      Status        Comment
  ----------------  ------------  ------------  ------------  ------------  ------------
  gridtest2         262144        262144        262144        passed
  gridtest1         262144        262144        262144        passed
Result: Kernel parameter check passed for "rmem_default"

Check: Kernel parameter for "rmem_max"
  Node Name         Current       Configured    Required      Status        Comment
  ----------------  ------------  ------------  ------------  ------------  ------------
  gridtest2         4194304       4194304       4194304       passed
  gridtest1         4194304       4194304       4194304       passed
Result: Kernel parameter check passed for "rmem_max"

Check: Kernel parameter for "wmem_default"
  Node Name         Current       Configured    Required      Status        Comment
  ----------------  ------------  ------------  ------------  ------------  ------------
  gridtest2         262144        262144        262144        passed
  gridtest1         262144        262144        262144        passed
Result: Kernel parameter check passed for "wmem_default"

Check: Kernel parameter for "wmem_max"
  Node Name         Current       Configured    Required      Status        Comment
  ----------------  ------------  ------------  ------------  ------------  ------------
  gridtest2         1048586       1048586       1048576       passed
  gridtest1         1048586       1048586       1048576       passed
Result: Kernel parameter check passed for "wmem_max"

Check: Kernel parameter for "aio-max-nr"
  Node Name         Current       Configured    Required      Status        Comment
  ----------------  ------------  ------------  ------------  ------------  ------------
  gridtest2         1048576       1048576       1048576       passed
  gridtest1         1048576       1048576       1048576       passed
Result: Kernel parameter check passed for "aio-max-nr"

Check: Package existence for "make"
  Node Name     Available                 Required                  Status
  ------------  ------------------------  ------------------------  ----------
  gridtest2     make-3.81-3.el5           make-3.81                 passed
  gridtest1     make-3.81-3.el5           make-3.81                 passed
Result: Package existence check passed for "make"

Check: Package existence for "binutils"
  Node Name     Available                 Required                  Status
  ------------  ------------------------  ------------------------  ----------
  gridtest2     binutils-2.17.50.0.6-14.el5  binutils-2.17.50.0.6      passed
  gridtest1     binutils-2.17.50.0.6-14.el5  binutils-2.17.50.0.6      passed
Result: Package existence check passed for "binutils"

Check: Package existence for "gcc(x86_64)"
  Node Name     Available                 Required                  Status
  ------------  ------------------------  ------------------------  ----------
  gridtest2     gcc(x86_64)-4.1.2-48.el5  gcc(x86_64)-4.1.2         passed
  gridtest1     gcc(x86_64)-4.1.2-48.el5  gcc(x86_64)-4.1.2         passed
Result: Package existence check passed for "gcc(x86_64)"

Check: Package existence for "libaio(x86_64)"
  Node Name     Available                 Required                  Status
  ------------  ------------------------  ------------------------  ----------
  gridtest2     libaio(x86_64)-0.3.106-5  libaio(x86_64)-0.3.106    passed
  gridtest1     libaio(x86_64)-0.3.106-5  libaio(x86_64)-0.3.106    passed
Result: Package existence check passed for "libaio(x86_64)"

Check: Package existence for "glibc(x86_64)"
  Node Name     Available                 Required                  Status
  ------------  ------------------------  ------------------------  ----------
  gridtest2     glibc(x86_64)-2.5-49      glibc(x86_64)-2.5-24      passed
  gridtest1     glibc(x86_64)-2.5-49      glibc(x86_64)-2.5-24      passed
Result: Package existence check passed for "glibc(x86_64)"

Check: Package existence for "compat-libstdc++-33(x86_64)"
  Node Name     Available                 Required                  Status
  ------------  ------------------------  ------------------------  ----------
  gridtest2     compat-libstdc++-33(x86_64)-3.2.3-61  compat-libstdc++-33(x86_64)-3.2.3  passed
  gridtest1     compat-libstdc++-33(x86_64)-3.2.3-61  compat-libstdc++-33(x86_64)-3.2.3  passed
Result: Package existence check passed for "compat-libstdc++-33(x86_64)"

Check: Package existence for "elfutils-libelf(x86_64)"
  Node Name     Available                 Required                  Status
  ------------  ------------------------  ------------------------  ----------
  gridtest2     elfutils-libelf(x86_64)-0.137-3.el5  elfutils-libelf(x86_64)-0.125  passed
  gridtest1     elfutils-libelf(x86_64)-0.137-3.el5  elfutils-libelf(x86_64)-0.125  passed
Result: Package existence check passed for "elfutils-libelf(x86_64)"

Check: Package existence for "elfutils-libelf-devel"
  Node Name     Available                 Required                  Status
  ------------  ------------------------  ------------------------  ----------
  gridtest2     elfutils-libelf-devel-0.125-3.el5  elfutils-libelf-devel-0.125  passed
  gridtest1     elfutils-libelf-devel-0.125-3.el5  elfutils-libelf-devel-0.125  passed
Result: Package existence check passed for "elfutils-libelf-devel"

Check: Package existence for "glibc-common"
  Node Name     Available                 Required                  Status
  ------------  ------------------------  ------------------------  ----------
  gridtest2     glibc-common-2.5-49       glibc-common-2.5          passed
  gridtest1     glibc-common-2.5-49       glibc-common-2.5          passed
Result: Package existence check passed for "glibc-common"

Check: Package existence for "glibc-devel(x86_64)"
  Node Name     Available                 Required                  Status
  ------------  ------------------------  ------------------------  ----------
  gridtest2     glibc-devel(x86_64)-2.5-49  glibc-devel(x86_64)-2.5   passed
  gridtest1     glibc-devel(x86_64)-2.5-49  glibc-devel(x86_64)-2.5   passed
Result: Package existence check passed for "glibc-devel(x86_64)"

Check: Package existence for "glibc-headers"
  Node Name     Available                 Required                  Status
  ------------  ------------------------  ------------------------  ----------
  gridtest2     glibc-headers-2.5-49      glibc-headers-2.5         passed
  gridtest1     glibc-headers-2.5-49      glibc-headers-2.5         passed
Result: Package existence check passed for "glibc-headers"

Check: Package existence for "gcc-c++(x86_64)"
  Node Name     Available                 Required                  Status
  ------------  ------------------------  ------------------------  ----------
  gridtest2     gcc-c++(x86_64)-4.1.2-48.el5  gcc-c++(x86_64)-4.1.2     passed
  gridtest1     gcc-c++(x86_64)-4.1.2-48.el5  gcc-c++(x86_64)-4.1.2     passed
Result: Package existence check passed for "gcc-c++(x86_64)"

Check: Package existence for "libaio-devel(x86_64)"
  Node Name     Available                 Required                  Status
  ------------  ------------------------  ------------------------  ----------
  gridtest2     libaio-devel(x86_64)-0.3.106-3.2  libaio-devel(x86_64)-0.3.106  passed
  gridtest1     libaio-devel(x86_64)-0.3.106-3.2  libaio-devel(x86_64)-0.3.106  passed
Result: Package existence check passed for "libaio-devel(x86_64)"

Check: Package existence for "libgcc(x86_64)"
  Node Name     Available                 Required                  Status
  ------------  ------------------------  ------------------------  ----------
  gridtest2     libgcc(x86_64)-4.1.2-48.el5  libgcc(x86_64)-4.1.2      passed
  gridtest1     libgcc(x86_64)-4.1.2-48.el5  libgcc(x86_64)-4.1.2      passed
Result: Package existence check passed for "libgcc(x86_64)"

Check: Package existence for "libstdc++(x86_64)"
  Node Name     Available                 Required                  Status
  ------------  ------------------------  ------------------------  ----------
  gridtest2     libstdc++(x86_64)-4.1.2-48.el5  libstdc++(x86_64)-4.1.2   passed
  gridtest1     libstdc++(x86_64)-4.1.2-48.el5  libstdc++(x86_64)-4.1.2   passed
Result: Package existence check passed for "libstdc++(x86_64)"

Check: Package existence for "libstdc++-devel(x86_64)"
  Node Name     Available                 Required                  Status
  ------------  ------------------------  ------------------------  ----------
  gridtest2     libstdc++-devel(x86_64)-4.1.2-48.el5  libstdc++-devel(x86_64)-4.1.2  passed
  gridtest1     libstdc++-devel(x86_64)-4.1.2-48.el5  libstdc++-devel(x86_64)-4.1.2  passed
Result: Package existence check passed for "libstdc++-devel(x86_64)"

Check: Package existence for "sysstat"
  Node Name     Available                 Required                  Status
  ------------  ------------------------  ------------------------  ----------
  gridtest2     sysstat-7.0.2-1.el5       sysstat-7.0.2             passed
  gridtest1     sysstat-7.0.2-1.el5       sysstat-7.0.2             passed

WARNING:
PRVF-7584 : Multiple versions of package "sysstat" found on node gridtest2: sysstat(x86_64)-7.0.2-1.el5,sysstat(i386)-7.0.2-1.el5

WARNING:
PRVF-7584 : Multiple versions of package "sysstat" found on node gridtest1: sysstat(x86_64)-7.0.2-1.el5,sysstat(i386)-7.0.2-1.el5
Result: Package existence check passed for "sysstat"

Check: Package existence for "ksh"
  Node Name     Available                 Required                  Status
  ------------  ------------------------  ------------------------  ----------
  gridtest2     ksh-20100202-1.el5        ksh-20060214              passed
  gridtest1     ksh-20100202-1.el5        ksh-20060214              passed
Result: Package existence check passed for "ksh"

Checking for multiple users with UID value 0
Result: Check for multiple users with UID value 0 passed

Check: Current group ID
Result: Current group ID check passed

Starting check for consistency of primary group of root user
  Node Name                             Status
  ------------------------------------  ------------------------
  gridtest2                             passed
  gridtest1                             passed

Check for consistency of root user's primary group passed

Starting Clock synchronization checks using Network Time Protocol(NTP)...

NTP Configuration file check started...
The NTP configuration file "/etc/ntp.conf" is available on all nodes
NTP Configuration file check passed

Checking daemon liveness...

Check: Liveness for "ntpd"
  Node Name                             Running?
  ------------------------------------  ------------------------
  gridtest2                             yes
  gridtest1                             yes
Result: Liveness check passed for "ntpd"
Check for NTP daemon or service alive passed on all nodes

Checking NTP daemon command line for slewing option "-x"
Check: NTP daemon command line
  Node Name                             Slewing Option Set?
  ------------------------------------  ------------------------
  gridtest2                             yes
  gridtest1                             yes
Result:
NTP daemon slewing option check passed

Checking NTP daemon's boot time configuration, in file "/etc/sysconfig/ntpd", for slewing option "-x"

Check: NTP daemon's boot time configuration
  Node Name                             Slewing Option Set?
  ------------------------------------  ------------------------
  gridtest2                             yes
  gridtest1                             yes
Result:
NTP daemon's boot time configuration check for slewing option passed

Checking whether NTP daemon or service is using UDP port 123 on all nodes

Check for NTP daemon or service using UDP port 123
  Node Name                             Port Open?
  ------------------------------------  ------------------------
  gridtest2                             yes
  gridtest1                             yes

NTP common Time Server Check started...
PRVF-5408 : NTP Time Server "124.82.204.2" is common only to the following nodes "gridtest1"
NTP Time Server ".LOCL." is common to all nodes on which the NTP daemon is running
Check of common NTP Time Server passed

Clock time offset check from NTP Time Server started...
Checking on nodes "[gridtest2, gridtest1]"...
Check: Clock time offset from NTP Time Server

Time Server: .LOCL.
Time Offset Limit: 1000.0 msecs
  Node Name     Time Offset               Status
  ------------  ------------------------  ------------------------
  gridtest2     0.0                       passed
  gridtest1     0.0                       passed
Time Server ".LOCL." has time offsets that are within permissible limits for nodes "[gridtest2, gridtest1]".
Clock time offset check passed

Result: Clock synchronization check using Network Time Protocol(NTP) passed

Checking Core file name pattern consistency...
Core file name pattern consistency check passed.

Checking to make sure user "oracle" is not in "root" group
  Node Name     Status                    Comment
  ------------  ------------------------  ------------------------
  gridtest2     passed                    does not exist
  gridtest1     passed                    does not exist
Result: User "oracle" is not part of "root" group. Check passed

Check default user file creation mask
  Node Name     Available                 Required                  Comment
  ------------  ------------------------  ------------------------  ----------
  gridtest2     0022                      0022                      passed
  gridtest1     0022                      0022                      passed
Result: Default user file creation mask check passed
Checking consistency of file "/etc/resolv.conf" across nodes

Checking the file "/etc/resolv.conf" to make sure only one of domain and search entries is defined
File "/etc/resolv.conf" does not have both domain and search entries defined
Checking if domain entry in file "/etc/resolv.conf" is consistent across the nodes...
domain entry in file "/etc/resolv.conf" is consistent across nodes
Checking if search entry in file "/etc/resolv.conf" is consistent across the nodes...
search entry in file "/etc/resolv.conf" is consistent across nodes
Checking DNS response time for an unreachable node
  Node Name                             Status
  ------------------------------------  ------------------------
  gridtest2                             passed
  gridtest1                             passed
The DNS response time for an unreachable node is within acceptable limit on all nodes

File "/etc/resolv.conf" is consistent across nodes


UDev attributes check for OCR locations started...
Result: UDev attributes check passed for OCR locations


UDev attributes check for Voting Disk locations started...
Result: UDev attributes check passed for Voting Disk locations

Check: Time zone consistency
Result: Time zone consistency check passed
Checking VIP configuration.
Checking VIP Subnet configuration.
Check for VIP Subnet configuration passed.
Checking VIP reachability
Check for VIP reachability passed.

Checking Oracle Cluster Voting Disk configuration...

ASM Running check passed. ASM is running on all specified nodes

Oracle Cluster Voting Disk configuration check passed

Clusterware version consistency passed

Pre-check for cluster services setup was successful.


5) Now all the pre-check has been passed. Inorder to perform NON rolling upgrade shutdown the crs on
all the nodes.
Node 1

# crsctl stop crs
Node 2
# crsctl stop crs


Now run the ./runInstaller and select the upgrade existing clusterware option and complete the installation.

when prompted run the ./rootupgrade.sh in node 1, once the execution is completed proceed with executing it on the 2nd node.

Node 1:-

[root@gridtest1 bin]# cd /grid/11.2.0.3/
[root@gridtest1 11.2.0.3]# ./rootupgrade.sh
Performing root user operation for Oracle 11g

The following environment variables are set as:
    ORACLE_OWNER= oracle
    ORACLE_HOME=  /grid/11.2.0.3

Enter the full pathname of the local bin directory: [/usr/local/bin]:
The contents of "dbhome" have not changed. No need to overwrite.
The file "oraenv" already exists in /usr/local/bin.  Overwrite it? (y/n)
[n]:
The file "coraenv" already exists in /usr/local/bin.  Overwrite it? (y/n)
[n]:

Entries will be added to the /etc/oratab file as needed by
Database Configuration Assistant when a database is created
Finished running generic part of root script.
Now product-specific root actions will be performed.
Using configuration parameter file: /grid/11.2.0.3/crs/install/crsconfig_params
Creating trace directory

ASM upgrade has initialized on first node.


OLR initialization - successful
Replacing Clusterware entries in inittab
clscfg: EXISTING configuration version 5 detected.
clscfg: version 5 is 11g Release 2.
Successfully accumulated necessary OCR keys.
Creating OCR keys for user 'root', privgrp 'root'..
Operation successful.

ASM upgrade has started on first node.

Preparing packages for installation...
cvuqdisk-1.0.9-1
Configure Oracle Grid Infrastructure for a Cluster ... succeeded


Node 2:-
[root@gridtest2 bin]# cd /grid/11.2.0.3/
[root@gridtest2 11.2.0.3]# ./rootupgrade.sh
Performing root user operation for Oracle 11g

The following environment variables are set as:
    ORACLE_OWNER= oracle
    ORACLE_HOME=  /grid/11.2.0.3

Enter the full pathname of the local bin directory: [/usr/local/bin]:
The contents of "dbhome" have not changed. No need to overwrite.
The file "oraenv" already exists in /usr/local/bin.  Overwrite it? (y/n)
[n]:
The file "coraenv" already exists in /usr/local/bin.  Overwrite it? (y/n)
[n]:

Entries will be added to the /etc/oratab file as needed by
Database Configuration Assistant when a database is created
Finished running generic part of root script.
Now product-specific root actions will be performed.

Using configuration parameter file: /grid/11.2.0.3/crs/install/crsconfig_params
Creating trace directory
OLR initialization - successful
Replacing Clusterware entries in inittab
clscfg: EXISTING configuration version 5 detected.
clscfg: version 5 is 11g Release 2.
Successfully accumulated necessary OCR keys.
Creating OCR keys for user 'root', privgrp 'root'..
Operation successful.
Started to upgrade the Oracle Clusterware. This operation may take a few minutes.
Started to upgrade the CSS.
Started to upgrade the CRS.
The CRS was successfully upgraded.
Oracle Clusterware operating version was successfully set to 11.2.0.3.0

ASM upgrade has finished on last node.

Preparing packages for installation...
cvuqdisk-1.0.9-1
Configure Oracle Grid Infrastructure for a Cluster ... succeeded


6) Now the clusterware upgrade has been completed. Check the resource status

[oracle@gridtest2 oracle]$ crsctl check cluster -all
**************************************************************
gridtest1:
CRS-4537: Cluster Ready Services is online
CRS-4529: Cluster Synchronization Services is online
CRS-4533: Event Manager is online
**************************************************************
gridtest2:
CRS-4537: Cluster Ready Services is online
CRS-4529: Cluster Synchronization Services is online
CRS-4533: Event Manager is online
**************************************************************

[oracle@gridtest2 oracle]$ crsctl stat res -t
--------------------------------------------------------------------------------
NAME           TARGET  STATE        SERVER                   STATE_DETAILS
--------------------------------------------------------------------------------
Local Resources
--------------------------------------------------------------------------------
ora.GRID_OCR_VOTING.dg
               ONLINE  ONLINE       gridtest1
               ONLINE  ONLINE       gridtest2
ora.LISTENER.lsnr
               ONLINE  ONLINE       gridtest1
               ONLINE  ONLINE       gridtest2
ora.asm
               ONLINE  ONLINE       gridtest1                Started
               ONLINE  ONLINE       gridtest2                Started
ora.gsd
               OFFLINE OFFLINE      gridtest1
               OFFLINE OFFLINE      gridtest2
ora.net1.network
               ONLINE  ONLINE       gridtest1
               ONLINE  ONLINE       gridtest2
ora.ons
               ONLINE  ONLINE       gridtest1
               ONLINE  ONLINE       gridtest2
ora.registry.acfs
               ONLINE  ONLINE       gridtest1
               ONLINE  ONLINE       gridtest2
--------------------------------------------------------------------------------
Cluster Resources
--------------------------------------------------------------------------------
ora.LISTENER_SCAN1.lsnr
      1        ONLINE  ONLINE       gridtest1
ora.cvu
      1        ONLINE  ONLINE       gridtest2
ora.gridtest1.vip
      1        ONLINE  ONLINE       gridtest1
ora.gridtest2.vip
      1        ONLINE  ONLINE       gridtest2
ora.oc4j
      1        ONLINE  ONLINE       gridtest2
ora.scan1.vip
      1        ONLINE  ONLINE       gridtest1


[oracle@gridtest1 oracle]$ crsctl query crs softwareversion
Oracle Clusterware version on node [gridtest1] is [11.2.0.3.0]

[oracle@gridtest2 oracle]$ crsctl query crs softwareversion
Oracle Clusterware version on node [gridtest2] is [11.2.0.3.0]