Sunday, December 2, 2012

Recovering OCR Voting in in 11.2.0.3

Recovering OCR/VOTING disk in 11.2.0.3. which stored in ASM


Test Case:-


We have two node Gird (gridtest1,gridtest2) up and running fine, the
OCR/Voting disks are placed in the ASM diskgroup. Now we are going
to remove the disk in which the OCR_VOTING disk group has been
created after which we can recover the same.


1. Stopped the crs on both the nodes and removed the asm disk which
has been used to create the OCR_VOTING diskgroup.

2. Now attempted to start the crs.


[root@gridtest1 bin]# ./crsctl check crs
CRS-4638: Oracle High Availability Services is online
CRS-4535: Cannot communicate with Cluster Ready Services
CRS-4530: Communications failure contacting Cluster Synchronization Services daemon
CRS-4534: Cannot communicate with Event Manager

The OHA services are onile but the other services are down. The below errors are
reported in the grid alert log.

[cssd(14456)]CRS-1714:Unable to discover any voting files, retrying discovery in 15 seconds; Details at (:CSSNM00070:) in /grid/11.2.0.3/log/gridtest1/cssd/ocssd.log
2012-12-02 04:13:11.470
[cssd(14456)]CRS-1714:Unable to discover any voting files, retrying discovery in 15 seconds; Details at (:CSSNM00070:) in /grid/11.2.0.3/log/gridtest1/cssd/ocssd.log
2012-12-02 04:13:26.504

Now we need to recover the GI.

1. Stop the CRS.

[root@gridtest1 bin]# ./crsctl stop crs
CRS-2796: The command may not proceed when Cluster Ready Services is not running
CRS-4687: Shutdown command has completed with errors.
CRS-4000: Command Stop failed, or completed with errors.


--Stop the crs using force option.

[root@gridtest1 bin]# ./crsctl stop crs -f
CRS-2791: Starting shutdown of Oracle High Availability Services-managed resources on 'gridtest1'
CRS-2673: Attempting to stop 'ora.mdnsd' on 'gridtest1'
CRS-2673: Attempting to stop 'ora.crf' on 'gridtest1'
CRS-2677: Stop of 'ora.mdnsd' on 'gridtest1' succeeded
CRS-2677: Stop of 'ora.crf' on 'gridtest1' succeeded
CRS-2673: Attempting to stop 'ora.gipcd' on 'gridtest1'
CRS-2677: Stop of 'ora.gipcd' on 'gridtest1' succeeded
CRS-2673: Attempting to stop 'ora.gpnpd' on 'gridtest1'
CRS-2677: Stop of 'ora.gpnpd' on 'gridtest1' succeeded
CRS-2793: Shutdown of Oracle High Availability Services-managed resources on 'gridtest1' has completed
CRS-4133: Oracle High Availability Services has been stopped.


2. Now create the asm disks.


[root@gridtest1 bin]# /etc/init.d/oracleasm createdisk GRID_OCR1 /dev/sdg1
Marking disk "GRID_OCR1" as an ASM disk:                   [  OK  ]

[root@gridtest1 bin]# /etc/init.d/oracleasm createdisk GRID_OCR2 /dev/sdh1
Marking disk "GRID_OCR2" as an ASM disk:                   [  OK  ]

[root@gridtest1 bin]# /etc/init.d/oracleasm createdisk GRID_OCR3 /dev/sdi1
Marking disk "GRID_OCR3" as an ASM disk:                   [  OK  ]


3. We need the ASM instance to recover the OCR/Voting disk, hence these files are placed on the
asm diskgroups.Current version of Oracle Grid Infrastructure (GI) can be started in exclusive mode
on one node to allow checks and restore.

[root@gridtest1 bin]# ./crsctl start crs -excl -nocrs
CRS-4123: Oracle High Availability Services has been started.
CRS-2672: Attempting to start 'ora.mdnsd' on 'gridtest1'
CRS-2676: Start of 'ora.mdnsd' on 'gridtest1' succeeded
CRS-2672: Attempting to start 'ora.gpnpd' on 'gridtest1'
CRS-2676: Start of 'ora.gpnpd' on 'gridtest1' succeeded
CRS-2672: Attempting to start 'ora.cssdmonitor' on 'gridtest1'
CRS-2672: Attempting to start 'ora.gipcd' on 'gridtest1'
CRS-2676: Start of 'ora.cssdmonitor' on 'gridtest1' succeeded
CRS-2676: Start of 'ora.gipcd' on 'gridtest1' succeeded
CRS-2672: Attempting to start 'ora.cssd' on 'gridtest1'
CRS-2672: Attempting to start 'ora.diskmon' on 'gridtest1'
CRS-2676: Start of 'ora.diskmon' on 'gridtest1' succeeded
CRS-2676: Start of 'ora.cssd' on 'gridtest1' succeeded
CRS-2672: Attempting to start 'ora.drivers.acfs' on 'gridtest1'
CRS-2679: Attempting to clean 'ora.cluster_interconnect.haip' on 'gridtest1'
CRS-2672: Attempting to start 'ora.ctssd' on 'gridtest1'
CRS-2681: Clean of 'ora.cluster_interconnect.haip' on 'gridtest1' succeeded
CRS-2672: Attempting to start 'ora.cluster_interconnect.haip' on 'gridtest1'
CRS-2676: Start of 'ora.drivers.acfs' on 'gridtest1' succeeded
CRS-2676: Start of 'ora.ctssd' on 'gridtest1' succeeded
CRS-2676: Start of 'ora.cluster_interconnect.haip' on 'gridtest1' succeeded
CRS-2672: Attempting to start 'ora.asm' on 'gridtest1'
CRS-2676: Start of 'ora.asm' on 'gridtest1' succeeded


[root@gridtest1 bin]# ps -ef | grep d.bin
root     19258     1  0 13:08 ?        00:00:02 /grid/11.2.0.3/bin/ohasd.bin exclusive
oracle   19392     1  0 13:09 ?        00:00:00 /grid/11.2.0.3/bin/mdnsd.bin
oracle   19402     1  0 13:09 ?        00:00:00 /grid/11.2.0.3/bin/gpnpd.bin
oracle   19415     1  0 13:09 ?        00:00:01 /grid/11.2.0.3/bin/gipcd.bin
oracle   19459     1  0 13:09 ?        00:00:01 /grid/11.2.0.3/bin/ocssd.bin -X
root     19505     1  0 13:09 ?        00:00:00 /grid/11.2.0.3/bin/octssd.bin
root     20218 19037  0 13:16 pts/1    00:00:00 grep d.bin


[root@gridtest1 bin]# ps -ef | grep pmon
oracle   15395     1  0 12:25 ?        00:00:00 asm_pmon_+ASM1
root     18526 12981  0 12:58 pts/1    00:00:00 grep pmon


[root@gridtest1 bin]# ./crsctl stop resource ora.crsd -init
[root@gridtest1 bin]# pwd
/grid/11.2.0.3/bin
[root@gridtest1 bin]# ./crsctl stat res -t -init
--------------------------------------------------------------------------------
NAME           TARGET  STATE        SERVER                   STATE_DETAILS
--------------------------------------------------------------------------------
Cluster Resources
--------------------------------------------------------------------------------
ora.asm
      1        ONLINE  INTERMEDIATE gridtest1                OCR not started
ora.cluster_interconnect.haip
      1        ONLINE  ONLINE       gridtest1
ora.crf
      1        OFFLINE OFFLINE
ora.crsd
      1        OFFLINE OFFLINE
ora.cssd
      1        ONLINE  ONLINE       gridtest1
ora.cssdmonitor
      1        ONLINE  ONLINE       gridtest1
ora.ctssd
      1        ONLINE  ONLINE       gridtest1                OBSERVER
ora.diskmon
      1        OFFLINE OFFLINE
ora.drivers.acfs
      1        ONLINE  ONLINE       gridtest1
ora.evmd
      1        OFFLINE OFFLINE
ora.gipcd
      1        ONLINE  ONLINE       gridtest1
ora.gpnpd
      1        ONLINE  ONLINE       gridtest1
ora.mdnsd
      1        ONLINE  ONLINE       gridtest1




Now we could see that the asm instance is up and running.
ASM is running. Now we can connect and create a disk group for OCR and vote disk. Be sure that your
new group will have proper COMPATIBLE.ASM value. You can put OCR and vote
on groups with COMPATIBLE.ASM greater than 11.2.

ORACLE_SID=+ASM1
ORACLE_HOME=/grid/11.2.0.3
[oracle@gridtest1 app]$ sqlplus / as sysasm

SQL*Plus: Release 11.2.0.3.0 Production on Sun Dec 2 13:20:23 2012

Copyright (c) 1982, 2011, Oracle.  All rights reserved.


Connected to:
Oracle Database 11g Enterprise Edition Release 11.2.0.3.0 - 64bit Production
With the Real Application Clusters and Automatic Storage Management options

SQL> set lines 200 pages 200;
SQL> col path form a40;
SQL> select path, header_status from V$asm_disk;

PATH                                     HEADER_STATU
---------------------------------------- ------------
ORCL:DATA_DISK01                         MEMBER
ORCL:DATA_DISK02                         MEMBER
ORCL:GRID_DATA1                          MEMBER
ORCL:GRID_DATA2                          MEMBER
ORCL:OCR_VOTING_02                       MEMBER
ORCL:GRID_OCR2                           PROVISIONED
ORCL:GRID_OCR3                           PROVISIONED
ORCL:OCR_VOTING_00                       MEMBER
ORCL:OCR_VOTING_01                       MEMBER
ORCL:GRID_OCR1                           PROVISIONED

10 rows selected.

SQL> create diskgroup GRID_OCR_VOTING normal redundancy
     disk 'ORCL:GRID_OCR1','ORCL:GRID_OCR2','ORCL:GRID_OCR3' attribute 'COMPATIBLE.ASM' = '11.2';

Diskgroup created.

SQL> show parameter spfile;

NAME                                 TYPE        VALUE
------------------------------------ ----------- ------------------------------
spfile                               string


The asm instance is not using the spfile, now we need to create a spfile .

vi /u01/app/init.ora

*.asm_power_limit=1
*.diagnostic_dest='/u01/app/oracle'
*.instance_type='asm'
*.large_pool_size=12M
*.asm_diskstring = 'ORCL:*'


[oracle@gridtest1 app]$ sqlplus / as sysasm

SQL*Plus: Release 11.2.0.3.0 Production on Sun Dec 2 13:20:23 2012

Copyright (c) 1982, 2011, Oracle.  All rights reserved.


Connected to:
Oracle Database 11g Enterprise Edition Release 11.2.0.3.0 - 64bit Production
With the Real Application Clusters and Automatic Storage Management options

SQL> create spfile = '+GRID_OCR_VOTING' from pfile='/u01/app/init.ora';

File created.


Now we need to restore the ocr from the backup.


[root@gridtest1 gridtest]# /grid/11.2.0.3/bin/ocrconfig -restore backup00.ocr
[root@gridtest1 gridtest]# /grid/11.2.0.3/bin/ocrcheck
Status of Oracle Cluster Registry is as follows :
         Version                  :          3
         Total space (kbytes)     :     262120
         Used space (kbytes)      :       3308
         Available space (kbytes) :     258812
         ID                       :  428205433
         Device/File Name         : +GRID_OCR_VOTING
                                    Device/File integrity check succeeded

                                    Device/File not configured

                                    Device/File not configured

                                    Device/File not configured

                                    Device/File not configured

         Cluster registry integrity check succeeded

         Logical corruption check succeeded


Now we need to restore the voting disk.

[root@gridtest1 gridtest]# /grid/11.2.0.3/bin/crsctl replace votedisk +GRID_OCR_VOTING
Successful addition of voting disk a4301c1d8cf14fe1bf566a26029627e0.
Successful addition of voting disk 1e5ac321102f4fbdbfe264620cfa794a.
Successful addition of voting disk 937512d798d84fc1bfcd876c9a223702.
Successfully replaced voting disk group with +GRID_OCR_VOTING.
CRS-4266: Voting file(s) successfully replaced

[root@gridtest1 gridtest]# /grid/11.2.0.3/bin/crsctl query css votedisk
##  STATE    File Universal Id                File Name Disk group
--  -----    -----------------                --------- ---------
 1. ONLINE   a4301c1d8cf14fe1bf566a26029627e0 (ORCL:GRID_OCR1) [GRID_OCR_VOTING]
 2. ONLINE   1e5ac321102f4fbdbfe264620cfa794a (ORCL:GRID_OCR2) [GRID_OCR_VOTING]
 3. ONLINE   937512d798d84fc1bfcd876c9a223702 (ORCL:GRID_OCR3) [GRID_OCR_VOTING]
Located 3 voting disk(s).


Now stop the crs on the current node and then restart the crs on all nodes.

[root@gridtest1 gridtest]# /grid/11.2.0.3/bin/crsctl stop crs -f
CRS-2791: Starting shutdown of Oracle High Availability Services-managed resources on 'gridtest1'
CRS-2673: Attempting to stop 'ora.mdnsd' on 'gridtest1'
CRS-2673: Attempting to stop 'ora.ctssd' on 'gridtest1'
CRS-2673: Attempting to stop 'ora.asm' on 'gridtest1'
CRS-2673: Attempting to stop 'ora.drivers.acfs' on 'gridtest1'
CRS-2677: Stop of 'ora.mdnsd' on 'gridtest1' succeeded
CRS-2677: Stop of 'ora.asm' on 'gridtest1' succeeded
CRS-2673: Attempting to stop 'ora.cluster_interconnect.haip' on 'gridtest1'
CRS-2677: Stop of 'ora.ctssd' on 'gridtest1' succeeded
CRS-2677: Stop of 'ora.cluster_interconnect.haip' on 'gridtest1' succeeded
CRS-2673: Attempting to stop 'ora.cssd' on 'gridtest1'
CRS-2677: Stop of 'ora.cssd' on 'gridtest1' succeeded
CRS-2673: Attempting to stop 'ora.gipcd' on 'gridtest1'
CRS-2677: Stop of 'ora.drivers.acfs' on 'gridtest1' succeeded
CRS-2677: Stop of 'ora.gipcd' on 'gridtest1' succeeded
CRS-2673: Attempting to stop 'ora.gpnpd' on 'gridtest1'
CRS-2677: Stop of 'ora.gpnpd' on 'gridtest1' succeeded
CRS-2793: Shutdown of Oracle High Availability Services-managed resources on 'gridtest1' has completed
CRS-4133: Oracle High Availability Services has been stopped.


[root@gridtest1 gridtest]# /grid/11.2.0.3/bin/crsctl start crs
CRS-4123: Oracle High Availability Services has been started.


[oracle@gridtest1 ~]$ crsctl check crs
CRS-4638: Oracle High Availability Services is online
CRS-4537: Cluster Ready Services is online
CRS-4529: Cluster Synchronization Services is online
CRS-4533: Event Manager is online

Start the crs in all the remaining nodes.


[oracle@gridtest1 ~]$ crsctl stat res -t
--------------------------------------------------------------------------------
NAME           TARGET  STATE        SERVER                   STATE_DETAILS
--------------------------------------------------------------------------------
Local Resources
--------------------------------------------------------------------------------
ora.GRID_OCR_VOTING.dg
               ONLINE  ONLINE       gridtest1
               ONLINE  ONLINE       gridtest2
ora.GRID_TEST_DB.dg
               ONLINE  ONLINE       gridtest1
               ONLINE  ONLINE       gridtest2
ora.LISTENER.lsnr
               ONLINE  ONLINE       gridtest1
               ONLINE  ONLINE       gridtest2
ora.asm
               ONLINE  ONLINE       gridtest1                Started
               ONLINE  ONLINE       gridtest2                Started
ora.gsd
               OFFLINE OFFLINE      gridtest1
               OFFLINE OFFLINE      gridtest2
ora.net1.network
               ONLINE  ONLINE       gridtest1
               ONLINE  ONLINE       gridtest2
ora.ons
               ONLINE  ONLINE       gridtest1
               ONLINE  ONLINE       gridtest2
ora.registry.acfs
               ONLINE  ONLINE       gridtest1
               ONLINE  ONLINE       gridtest2
--------------------------------------------------------------------------------
Cluster Resources
--------------------------------------------------------------------------------
ora.LISTENER_SCAN1.lsnr
      1        ONLINE  ONLINE       gridtest1
ora.cvu
      1        ONLINE  ONLINE       gridtest1
ora.gridtest1.vip
      1        ONLINE  ONLINE       gridtest1
ora.gridtest2.vip
      1        ONLINE  ONLINE       gridtest2
ora.layatest.db
      1        ONLINE  ONLINE       gridtest1                Open
ora.layatest.layaconn.svc
      1        ONLINE  ONLINE       gridtest1
ora.oc4j
      1        ONLINE  ONLINE       gridtest1
ora.scan1.vip
      1        ONLINE  ONLINE       gridtest1






No comments:

Post a Comment