Recovering OCR/VOTING disk in 11.2.0.3. which stored in ASM
Test Case:-
We have two node Gird (gridtest1,gridtest2) up and running fine, the
OCR/Voting disks are placed in the ASM diskgroup. Now we are going
to remove the disk in which the OCR_VOTING disk group has been
created after which we can recover the same.
1. Stopped the crs on both the nodes and removed the asm disk which
has been used to create the OCR_VOTING diskgroup.
2. Now attempted to start the crs.
[root@gridtest1 bin]# ./crsctl check crs
CRS-4638: Oracle High Availability Services is online
CRS-4535: Cannot communicate with Cluster Ready Services
CRS-4530: Communications failure contacting Cluster Synchronization Services daemon
CRS-4534: Cannot communicate with Event Manager
The OHA services are onile but the other services are down. The below errors are
reported in the grid alert log.
[cssd(14456)]CRS-1714:Unable to discover any voting files, retrying discovery in 15 seconds; Details at (:CSSNM00070:) in /grid/11.2.0.3/log/gridtest1/cssd/ocssd.log
2012-12-02 04:13:11.470
[cssd(14456)]CRS-1714:Unable to discover any voting files, retrying discovery in 15 seconds; Details at (:CSSNM00070:) in /grid/11.2.0.3/log/gridtest1/cssd/ocssd.log
2012-12-02 04:13:26.504
Now we need to recover the GI.
1. Stop the CRS.
[root@gridtest1 bin]# ./crsctl stop crs
CRS-2796: The command may not proceed when Cluster Ready Services is not running
CRS-4687: Shutdown command has completed with errors.
CRS-4000: Command Stop failed, or completed with errors.
--Stop the crs using force option.
[root@gridtest1 bin]# ./crsctl stop crs -f
CRS-2791: Starting shutdown of Oracle High Availability Services-managed resources on 'gridtest1'
CRS-2673: Attempting to stop 'ora.mdnsd' on 'gridtest1'
CRS-2673: Attempting to stop 'ora.crf' on 'gridtest1'
CRS-2677: Stop of 'ora.mdnsd' on 'gridtest1' succeeded
CRS-2677: Stop of 'ora.crf' on 'gridtest1' succeeded
CRS-2673: Attempting to stop 'ora.gipcd' on 'gridtest1'
CRS-2677: Stop of 'ora.gipcd' on 'gridtest1' succeeded
CRS-2673: Attempting to stop 'ora.gpnpd' on 'gridtest1'
CRS-2677: Stop of 'ora.gpnpd' on 'gridtest1' succeeded
CRS-2793: Shutdown of Oracle High Availability Services-managed resources on 'gridtest1' has completed
CRS-4133: Oracle High Availability Services has been stopped.
2. Now create the asm disks.
[root@gridtest1 bin]# /etc/init.d/oracleasm createdisk GRID_OCR1 /dev/sdg1
Marking disk "GRID_OCR1" as an ASM disk: [ OK ]
[root@gridtest1 bin]# /etc/init.d/oracleasm createdisk GRID_OCR2 /dev/sdh1
Marking disk "GRID_OCR2" as an ASM disk: [ OK ]
[root@gridtest1 bin]# /etc/init.d/oracleasm createdisk GRID_OCR3 /dev/sdi1
Marking disk "GRID_OCR3" as an ASM disk: [ OK ]
3. We need the ASM instance to recover the OCR/Voting disk, hence these files are placed on the
asm diskgroups.Current version of Oracle Grid Infrastructure (GI) can be started in exclusive mode
on one node to allow checks and restore.
[root@gridtest1 bin]# ./crsctl start crs -excl -nocrs
CRS-4123: Oracle High Availability Services has been started.
CRS-2672: Attempting to start 'ora.mdnsd' on 'gridtest1'
CRS-2676: Start of 'ora.mdnsd' on 'gridtest1' succeeded
CRS-2672: Attempting to start 'ora.gpnpd' on 'gridtest1'
CRS-2676: Start of 'ora.gpnpd' on 'gridtest1' succeeded
CRS-2672: Attempting to start 'ora.cssdmonitor' on 'gridtest1'
CRS-2672: Attempting to start 'ora.gipcd' on 'gridtest1'
CRS-2676: Start of 'ora.cssdmonitor' on 'gridtest1' succeeded
CRS-2676: Start of 'ora.gipcd' on 'gridtest1' succeeded
CRS-2672: Attempting to start 'ora.cssd' on 'gridtest1'
CRS-2672: Attempting to start 'ora.diskmon' on 'gridtest1'
CRS-2676: Start of 'ora.diskmon' on 'gridtest1' succeeded
CRS-2676: Start of 'ora.cssd' on 'gridtest1' succeeded
CRS-2672: Attempting to start 'ora.drivers.acfs' on 'gridtest1'
CRS-2679: Attempting to clean 'ora.cluster_interconnect.haip' on 'gridtest1'
CRS-2672: Attempting to start 'ora.ctssd' on 'gridtest1'
CRS-2681: Clean of 'ora.cluster_interconnect.haip' on 'gridtest1' succeeded
CRS-2672: Attempting to start 'ora.cluster_interconnect.haip' on 'gridtest1'
CRS-2676: Start of 'ora.drivers.acfs' on 'gridtest1' succeeded
CRS-2676: Start of 'ora.ctssd' on 'gridtest1' succeeded
CRS-2676: Start of 'ora.cluster_interconnect.haip' on 'gridtest1' succeeded
CRS-2672: Attempting to start 'ora.asm' on 'gridtest1'
CRS-2676: Start of 'ora.asm' on 'gridtest1' succeeded
[root@gridtest1 bin]# ps -ef | grep d.bin
root 19258 1 0 13:08 ? 00:00:02 /grid/11.2.0.3/bin/ohasd.bin exclusive
oracle 19392 1 0 13:09 ? 00:00:00 /grid/11.2.0.3/bin/mdnsd.bin
oracle 19402 1 0 13:09 ? 00:00:00 /grid/11.2.0.3/bin/gpnpd.bin
oracle 19415 1 0 13:09 ? 00:00:01 /grid/11.2.0.3/bin/gipcd.bin
oracle 19459 1 0 13:09 ? 00:00:01 /grid/11.2.0.3/bin/ocssd.bin -X
root 19505 1 0 13:09 ? 00:00:00 /grid/11.2.0.3/bin/octssd.bin
root 20218 19037 0 13:16 pts/1 00:00:00 grep d.bin
[root@gridtest1 bin]# ps -ef | grep pmon
oracle 15395 1 0 12:25 ? 00:00:00 asm_pmon_+ASM1
root 18526 12981 0 12:58 pts/1 00:00:00 grep pmon
[root@gridtest1 bin]# ./crsctl stop resource ora.crsd -init
[root@gridtest1 bin]# pwd
/grid/11.2.0.3/bin
[root@gridtest1 bin]# ./crsctl stat res -t -init
--------------------------------------------------------------------------------
NAME TARGET STATE SERVER STATE_DETAILS
--------------------------------------------------------------------------------
Cluster Resources
--------------------------------------------------------------------------------
ora.asm
1 ONLINE INTERMEDIATE gridtest1 OCR not started
ora.cluster_interconnect.haip
1 ONLINE ONLINE gridtest1
ora.crf
1 OFFLINE OFFLINE
ora.crsd
1 OFFLINE OFFLINE
ora.cssd
1 ONLINE ONLINE gridtest1
ora.cssdmonitor
1 ONLINE ONLINE gridtest1
ora.ctssd
1 ONLINE ONLINE gridtest1 OBSERVER
ora.diskmon
1 OFFLINE OFFLINE
ora.drivers.acfs
1 ONLINE ONLINE gridtest1
ora.evmd
1 OFFLINE OFFLINE
ora.gipcd
1 ONLINE ONLINE gridtest1
ora.gpnpd
1 ONLINE ONLINE gridtest1
ora.mdnsd
1 ONLINE ONLINE gridtest1
Now we could see that the asm instance is up and running.
ASM is running. Now we can connect and create a disk group for OCR and vote disk. Be sure that your
new group will have proper COMPATIBLE.ASM value. You can put OCR and vote
on groups with COMPATIBLE.ASM greater than 11.2.
ORACLE_SID=+ASM1
ORACLE_HOME=/grid/11.2.0.3
[oracle@gridtest1 app]$ sqlplus / as sysasm
SQL*Plus: Release 11.2.0.3.0 Production on Sun Dec 2 13:20:23 2012
Copyright (c) 1982, 2011, Oracle. All rights reserved.
Connected to:
Oracle Database 11g Enterprise Edition Release 11.2.0.3.0 - 64bit Production
With the Real Application Clusters and Automatic Storage Management options
SQL> set lines 200 pages 200;
SQL> col path form a40;
SQL> select path, header_status from V$asm_disk;
PATH HEADER_STATU
---------------------------------------- ------------
ORCL:DATA_DISK01 MEMBER
ORCL:DATA_DISK02 MEMBER
ORCL:GRID_DATA1 MEMBER
ORCL:GRID_DATA2 MEMBER
ORCL:OCR_VOTING_02 MEMBER
ORCL:GRID_OCR2 PROVISIONED
ORCL:GRID_OCR3 PROVISIONED
ORCL:OCR_VOTING_00 MEMBER
ORCL:OCR_VOTING_01 MEMBER
ORCL:GRID_OCR1 PROVISIONED
10 rows selected.
SQL> create diskgroup GRID_OCR_VOTING normal redundancy
disk 'ORCL:GRID_OCR1','ORCL:GRID_OCR2','ORCL:GRID_OCR3' attribute 'COMPATIBLE.ASM' = '11.2';
Diskgroup created.
SQL> show parameter spfile;
NAME TYPE VALUE
------------------------------------ ----------- ------------------------------
spfile string
The asm instance is not using the spfile, now we need to create a spfile .
vi /u01/app/init.ora
*.asm_power_limit=1
*.diagnostic_dest='/u01/app/oracle'
*.instance_type='asm'
*.large_pool_size=12M
*.asm_diskstring = 'ORCL:*'
[oracle@gridtest1 app]$ sqlplus / as sysasm
SQL*Plus: Release 11.2.0.3.0 Production on Sun Dec 2 13:20:23 2012
Copyright (c) 1982, 2011, Oracle. All rights reserved.
Connected to:
Oracle Database 11g Enterprise Edition Release 11.2.0.3.0 - 64bit Production
With the Real Application Clusters and Automatic Storage Management options
SQL> create spfile = '+GRID_OCR_VOTING' from pfile='/u01/app/init.ora';
File created.
Now we need to restore the ocr from the backup.
[root@gridtest1 gridtest]# /grid/11.2.0.3/bin/ocrconfig -restore backup00.ocr
[root@gridtest1 gridtest]# /grid/11.2.0.3/bin/ocrcheck
Status of Oracle Cluster Registry is as follows :
Version : 3
Total space (kbytes) : 262120
Used space (kbytes) : 3308
Available space (kbytes) : 258812
ID : 428205433
Device/File Name : +GRID_OCR_VOTING
Device/File integrity check succeeded
Device/File not configured
Device/File not configured
Device/File not configured
Device/File not configured
Cluster registry integrity check succeeded
Logical corruption check succeeded
Now we need to restore the voting disk.
[root@gridtest1 gridtest]# /grid/11.2.0.3/bin/crsctl replace votedisk +GRID_OCR_VOTING
Successful addition of voting disk a4301c1d8cf14fe1bf566a26029627e0.
Successful addition of voting disk 1e5ac321102f4fbdbfe264620cfa794a.
Successful addition of voting disk 937512d798d84fc1bfcd876c9a223702.
Successfully replaced voting disk group with +GRID_OCR_VOTING.
CRS-4266: Voting file(s) successfully replaced
[root@gridtest1 gridtest]# /grid/11.2.0.3/bin/crsctl query css votedisk
## STATE File Universal Id File Name Disk group
-- ----- ----------------- --------- ---------
1. ONLINE a4301c1d8cf14fe1bf566a26029627e0 (ORCL:GRID_OCR1) [GRID_OCR_VOTING]
2. ONLINE 1e5ac321102f4fbdbfe264620cfa794a (ORCL:GRID_OCR2) [GRID_OCR_VOTING]
3. ONLINE 937512d798d84fc1bfcd876c9a223702 (ORCL:GRID_OCR3) [GRID_OCR_VOTING]
Located 3 voting disk(s).
Now stop the crs on the current node and then restart the crs on all nodes.
[root@gridtest1 gridtest]# /grid/11.2.0.3/bin/crsctl stop crs -f
CRS-2791: Starting shutdown of Oracle High Availability Services-managed resources on 'gridtest1'
CRS-2673: Attempting to stop 'ora.mdnsd' on 'gridtest1'
CRS-2673: Attempting to stop 'ora.ctssd' on 'gridtest1'
CRS-2673: Attempting to stop 'ora.asm' on 'gridtest1'
CRS-2673: Attempting to stop 'ora.drivers.acfs' on 'gridtest1'
CRS-2677: Stop of 'ora.mdnsd' on 'gridtest1' succeeded
CRS-2677: Stop of 'ora.asm' on 'gridtest1' succeeded
CRS-2673: Attempting to stop 'ora.cluster_interconnect.haip' on 'gridtest1'
CRS-2677: Stop of 'ora.ctssd' on 'gridtest1' succeeded
CRS-2677: Stop of 'ora.cluster_interconnect.haip' on 'gridtest1' succeeded
CRS-2673: Attempting to stop 'ora.cssd' on 'gridtest1'
CRS-2677: Stop of 'ora.cssd' on 'gridtest1' succeeded
CRS-2673: Attempting to stop 'ora.gipcd' on 'gridtest1'
CRS-2677: Stop of 'ora.drivers.acfs' on 'gridtest1' succeeded
CRS-2677: Stop of 'ora.gipcd' on 'gridtest1' succeeded
CRS-2673: Attempting to stop 'ora.gpnpd' on 'gridtest1'
CRS-2677: Stop of 'ora.gpnpd' on 'gridtest1' succeeded
CRS-2793: Shutdown of Oracle High Availability Services-managed resources on 'gridtest1' has completed
CRS-4133: Oracle High Availability Services has been stopped.
[root@gridtest1 gridtest]# /grid/11.2.0.3/bin/crsctl start crs
CRS-4123: Oracle High Availability Services has been started.
[oracle@gridtest1 ~]$ crsctl check crs
CRS-4638: Oracle High Availability Services is online
CRS-4537: Cluster Ready Services is online
CRS-4529: Cluster Synchronization Services is online
CRS-4533: Event Manager is online
Start the crs in all the remaining nodes.
[oracle@gridtest1 ~]$ crsctl stat res -t
--------------------------------------------------------------------------------
NAME TARGET STATE SERVER STATE_DETAILS
--------------------------------------------------------------------------------
Local Resources
--------------------------------------------------------------------------------
ora.GRID_OCR_VOTING.dg
ONLINE ONLINE gridtest1
ONLINE ONLINE gridtest2
ora.GRID_TEST_DB.dg
ONLINE ONLINE gridtest1
ONLINE ONLINE gridtest2
ora.LISTENER.lsnr
ONLINE ONLINE gridtest1
ONLINE ONLINE gridtest2
ora.asm
ONLINE ONLINE gridtest1 Started
ONLINE ONLINE gridtest2 Started
ora.gsd
OFFLINE OFFLINE gridtest1
OFFLINE OFFLINE gridtest2
ora.net1.network
ONLINE ONLINE gridtest1
ONLINE ONLINE gridtest2
ora.ons
ONLINE ONLINE gridtest1
ONLINE ONLINE gridtest2
ora.registry.acfs
ONLINE ONLINE gridtest1
ONLINE ONLINE gridtest2
--------------------------------------------------------------------------------
Cluster Resources
--------------------------------------------------------------------------------
ora.LISTENER_SCAN1.lsnr
1 ONLINE ONLINE gridtest1
ora.cvu
1 ONLINE ONLINE gridtest1
ora.gridtest1.vip
1 ONLINE ONLINE gridtest1
ora.gridtest2.vip
1 ONLINE ONLINE gridtest2
ora.layatest.db
1 ONLINE ONLINE gridtest1 Open
ora.layatest.layaconn.svc
1 ONLINE ONLINE gridtest1
ora.oc4j
1 ONLINE ONLINE gridtest1
ora.scan1.vip
1 ONLINE ONLINE gridtest1
No comments:
Post a Comment