Clusterware Exclusive Mode to restore OCR

We execute crsctl start crs command as OS user root to start Oracle High Availability Service on the local server. We can invoke crsctl command in exclusive mode to perform exclusive cluster maintenance task as restoring OCR and VDs, troubleshooting root.sh and so on. While starting cluster in exclusive mode we need to stop cluster stack from all other nodes in the cluster. Starting Cluster in exclusive mode do not requires VD and network.

Following is the syntax.

crsctl start crs [-excl [-nocrs] [-cssonly]] | [-wait | -waithas | -nowait] | [-noautostart]

-excl Starts clusterware in exclusive mode

-nocrs Starts Clusterware with out CRS

-cssonly Starts CSS only

-wait Wait until the crs started completely and display all progress and status

-waithas Wait until startup is completed and display OHASD progress and status

-nowait Not to wait for OHASD to start

-noautostart Start only OHASD.

I am unable to start crs in my two node rac (rac01, rac02).  While digging more I came to know that OCR file is missing.

Trying to start crs normally from rac01.

./crsctl start crs

Few minutes later, I check the status of the crs 

./crsctl stat res -t

Checking the crs with crsctl check crs command and notice that CRS is not up, css is online and evmd is also not working.

./crsctl check crs

Checking the status of ocrfile but no luck.

./ocrcheck

Use adrci to find the diagnostic location and crs logs.

adrci

show homes

Checking alert file.

cd /u01/app/oracle/diag/crs/rac01/crs

tail -f alert.log

On accessing trace file it is noticed that OCR file is marked as unavailable.

cat /u01/app/oracle/diag/crs/rac01/crs/trace/crsd.trc 

Trying to stop crs on rac01 on which we have started crs previously.

./crsctl stop crs

Since stopping crs failed, -f option is used to stop crs forcefully.

 ./crsctl stop crs -f

Stopping crs on rac02 so that crs stack is not started on any node of the cluster.

./crsctl stop crs -f

Start Cluster stack on exclusive mode without crs -nocrs option.

./crsctl start crs -excl -nocrs

List the details of ocr backup files.

 ./ocrcheck -showbackup

Accessing the location of OCR file we noticed that backup file is already created on the location.

asmcmd

ls

cd DATA

cd rac-cluster

cd OCRBACKUP

ls

Restore OCR file with the command below and stop the crs that has been started on exclusive mode.

./ocrconfig -restore  +FRA:/rac-cluster/OCRBACKUP/backup00.ocr.257.968870687

./crsctl stop crs

Start CRS normally.

 ./crsctl start crs

Verify CRS file on ASM storage.

cd +DATA/rac-cluster/OCRFILE

ls

Verify crs status on the node.

./crsctl stat res -t

Now the OCR file is restored successfully.  We can start crs on another node too.

Additional OCR file integrity verification commands:
#ocrcheck
$ocrcheck -config
$ocrcheck -local -config
$cluvfy comp ocr -n all -verbose
$cluvfy comp ocr -n rac01 -verbose

$cluvfy comp ocr -n rac02 -verbose