Friday, March 2, 2018

Adding Second SCAN listeners on a Second Network to a 12c RAC Cluster

1:28 PM Posted by Dilli Maharjan No comments


Environment:
OS : Oracle Enterprise Linux 7.4
Oracle : Oracle Database 12 R2 RAC
nodes : rac01.localdomain, rac02.localdomain

    Generally if we use single switch and network for public interface of RAC then there might be outage in case of the network failure and network component failure. Lets say that if there is problem in network switch then database will not be available. We can provide network redundancy with the network bonding. With the help of network bonding we can create two set of network acting as one. 
   We can add second network in RAC and create VIP on second network. I am adding Second network with Switch II and interfaces enp0s9 in each rac nodes. In case there is any issue on first network all the request will go through second network. Adding second network with the subnet of 192.168.3.0/24 for load balancing and failover. Prior to Oracle 12c we can add VIP listeners on second network and scan listener can listen only in the default first network. From Oracle 12c we can add VIP listeners and SCAN listeners on second network too. 

Adding DNS record.

Editing my zone file with vi.


Following is my current DNS entry.

Adding new DNS entries.
rac01-bck       IN      A               192.168.3.101
rac02-bck       IN      A               192.168.3.102
rac01-bck-vip   IN      A               192.168.3.201
rac02-bck-vip   IN      A               192.168.3.202
rac-bck-scan    IN      A               192.168.3.203
rac-bck-scan    IN      A               192.168.3.204
rac-bck-scan    IN      A               192.168.3.205



Restarting DNS service
/etc/init.d/named restart



Verify that newly added DNS entries have been resolved.
nslookup rac01-bck.localdomain
nslookup rac02-bck.localdomain
nslookup rac-bck-scan.localdomain



I am using Virtual Box. Adding new interface on Virtual box. I am adding Adapter 3 that is attached to Host-only Adapter.



Adding adapter for second node 12cR2RAC2



Once network interface has been added on both nodes login to first node as user root. rac01 is the first node in my case and configure network with nmtui GUI tool.
nmtui



Below is the landing page Click on OK to continue.


Click on Add.


Select Ethernet and Click on Create.


Type enp0s9 in the profile name, Device name as enp0s9. Type Address as 192.168.3.101/24 and click on OK.


Once adding is competed click on back


Select and click Quit to quit from the interface.

Once network interface configuration is completed, verify IP address has been set properly with ipconfig command.
ipconfig



Repeat the same task on next node rac02 and verify IP address has been set properly.



Add following host entries on /etc/hosts file.
192.168.3.101  rac01-bck.localdomain
192.168.3.102  rac02-bck.localdomain 

192.168.3.201  rac01-bck-vip.localdomain
192.168.3.202  rac02-bck-vip.localdomain

192.168.3.203  rac-bck-scan.localdomain rac-bck-scan
192.168.3.204  rac-bck-scan.localdomain rac-bck-scan
192.168.3.205  rac-bck-scan.localdomain rac-bck-scan 



Verify new network addresses have been resolved . Use nslookup command to verify it.
nslookup rac01-bck
nslookup rac02-bck
nslookup rac-bck-scan



Change directory to GRID_HOME/bin and execute following statements to add network 192.168.3.0/24 to net number 2 and add vip associated with net number 2 to respected nodes.
cd /u01/app/12.2.0/grid/bin
./srvctl add network -netnum 2 -subnet 192.168.3.0/255.255.255.0
./srvctl add vip -node rac01 -address 192.168.3.201/255.255.255.0 -netnum 2
./srvctl add vip -node rac02 -address 192.168.3.202/255.255.255.0 -netnum 2



Verify that new network has been added and online on both nodes. Use crsctl stat res -t command to list all the resources and its status.
./crsctl stat res -t



In the same way verify newly added vip is listed on the crsctl stat command output.



Add second listener LISTENER_BCK to second network using TCP with port 1522.
./srvctl add listener -listener LISTENER_bck -netnum 2 -user oracle -endpoints TCP:1522 



Try to start new listener with no success. We need to create directory with name oracle under GRID_HOME/network/admin. The new listener configuration files will be stored on that location with names listener.ora.net.rac01 and listener.orac.net.rac02 on nodes rac01 and rac02 respectively.

cd /u01/app/12.2.0/grid/bin
./srvctl start listener -listener LISTENER_BCK



Creating oracle directory under $GRID_HOME/network/admin on rac01.
mkdir -p /u01/app/12.2.0/grid/network/admin/oracle
chown -R oracle:oinstall /u01/app/12.2.0/grid/network/admin/oracle 



Creating oracle directory under $GRID_HOME/network/admin on rac02.
mkdir -p /u01/app/12.2.0/grid/network/admin/oracle



Now we can start the newly created listener LISTENER_BCK successfully.
./srvctl start listener -listener LISTENER_BCK



Verify the new listener started successfully with crsctl command
./crsctl stat res -t



Add new scan name
./srvctl add scan -scanname rac-bck-scan -netnum 2



Verify new scan has been created and is currently offline.
./crsctl stat res -t



Add scan listener using TCP and port 1522 to net number 2
./srvctl add scan_listener -listener LISTSCAN -endpoints TCP:1522 -netnum 2


Add following tns configuration on tnsnames files on DB home directory.
on rac01
LISTENER =
        (DESCRIPTION =
                (ADDRESS = (PROTOCOL = TCP)(HOST = 192.168.1.201)(PORT = 1521))
)

LISTENER_BCK =
        (DESCRIPTION =
                (ADDRESS = (PROTOCOL = TCP)(HOST = 192.168.3.201)(PORT = 1522))
        )

R_LISTENER=
        (DESCRIPTION =
                (ADDRESS = (PROTOCOL = TCP)(HOST = 192.168.1.203)(PORT = 1521))
                (ADDRESS = (PROTOCOL = TCP)(HOST = 192.168.1.204)(PORT = 1521))
                (ADDRESS = (PROTOCOL = TCP)(HOST = 192.168.1.205)(PORT = 1521))
)

R_LISTENER_BCK=
        (DESCRIPTION =
                (ADDRESS = (PROTOCOL = TCP)(HOST = 192.168.3.203)(PORT = 1522))
                (ADDRESS = (PROTOCOL = TCP)(HOST = 192.168.3.204)(PORT = 1522))
                (ADDRESS = (PROTOCOL = TCP)(HOST = 192.168.3.205)(PORT = 1522))
)

on rac02
LISTENER =
        (DESCRIPTION =
                (ADDRESS = (PROTOCOL = TCP)(HOST = 192.168.1.202)(PORT = 1521))
)

LISTENER_BCK =
        (DESCRIPTION =
                (ADDRESS = (PROTOCOL = TCP)(HOST = 192.168.3.202)(PORT = 1522))
        )

R_LISTENER=
        (DESCRIPTION =
                (ADDRESS = (PROTOCOL = TCP)(HOST = 192.168.1.203)(PORT = 1521))
                (ADDRESS = (PROTOCOL = TCP)(HOST = 192.168.1.204)(PORT = 1521))
                (ADDRESS = (PROTOCOL = TCP)(HOST = 192.168.1.205)(PORT = 1521))
)

R_LISTENER_BCK=
        (DESCRIPTION =
                (ADDRESS = (PROTOCOL = TCP)(HOST = 192.168.3.203)(PORT = 1522))
                (ADDRESS = (PROTOCOL = TCP)(HOST = 192.168.3.204)(PORT = 1522))
                (ADDRESS = (PROTOCOL = TCP)(HOST = 192.168.3.205)(PORT = 1522))
)

Now login to rac database from one of the node and modify listener_networks parameter so that PMON or LREG process can register the services to listeners on both network.
In statement below we are adding two network with following details:

Network INetwork IIDescription
Namenetwork1network2Defines the name it can be anything
Local_ListenerLISTENERLISTENER_BCKDefines the local listener. It is defined on tnsnames.ora and contains the VIP of the server
Remote_Listenerrac-scan:1521rac-bck-scan:1522Scan name of the first and second network with the port on which listener is active.

alter system set listener_networks='((NAME=network1)(LOCAL_LISTENER=LISTENER)(REMOTE_LISTENER=rac-scan:1521))','((NAME=network2)(LOCAL_LISTENER=LISTENER_BCK)(REMOTE_LISTENER=rac-bck-scan:1522))' sid='*';



Start scan listener on net number 2
./srvctl start scan_listener -netnum 2



Verify the listener has been started using crsctl stat res -t command.
./crsctl stat res -t



Check listener status with lsnrctl command.
lsnrctl status



Check the status of LISTENER_BCK listener.
set current_listener LISTENER_BCK
status



On client site create tns setting to connect to RAC on both the scans. We can use LOAD_BALANCE and FAILOVER for high availability and load balancing.




Additionally create two tnsnames to connect on two scan address seperately.



Verify the connection using newly creating tnsnames.


Sunday, February 25, 2018

Clusterware Exclusive Mode to restore OCR

9:58 AM Posted by Dilli Maharjan , No comments

We execute crsctl start crs command as OS user root to start Oracle High Availability Service on the local server. We can invoke crsctl command in exclusive mode to perform exclusive cluster maintenance task as restoring OCR and VDs, troubleshooting root.sh and so on. While starting cluster in exclusive mode we need to stop cluster stack from all other nodes in the cluster. Starting Cluster in exclusive mode do not requires VD and network.

Following is the syntax.
crsctl start crs [-excl [-nocrs] [-cssonly]] | [-wait | -waithas | -nowait] | [-noautostart]

-excl Starts clusterware in exclusive mode
-nocrs Starts Clusterware with out CRS
-cssonly Starts CSS only

-wait Wait until the crs started completely and display all progress and status
-waithas Wait until startup is completed and display OHASD progress and status
-nowait Not to wait for OHASD to start

-noautostart Start only OHASD.

I am unable to start crs in my two node rac (rac01, rac02).  While digging more I came to know that OCR file is missing.


Trying to start crs normally from rac01.
./crsctl start crs



Few minutes later, I check the status of the crs 
./crsctl stat res -t



Checking the crs with crsctl check crs command and notice that CRS is not up, css is online and evmd is also not working.
./crsctl check crs



Checking the status of ocrfile but no luck.
./ocrcheck



Use adrci to find the diagnostic location and crs logs.
adrci
show homes



Checking alert file.
cd /u01/app/oracle/diag/crs/rac01/crs
tail -f alert.log



On accessing trace file it is noticed that OCR file is marked as unavailable.
cat /u01/app/oracle/diag/crs/rac01/crs/trace/crsd.trc 



Trying to stop crs on rac01 on which we have started crs previously.
./crsctl stop crs



Since stopping crs failed, -f option is used to stop crs forcefully.
 ./crsctl stop crs -f



Stopping crs on rac02 so that crs stack is not started on any node of the cluster.
./crsctl stop crs -f



Start Cluster stack on exclusive mode without crs -nocrs option.
./crsctl start crs -excl -nocrs



List the details of ocr backup files.
 ./ocrcheck -showbackup



Accessing the location of OCR file we noticed that backup file is already created on the location.
asmcmd
ls
cd DATA
cd rac-cluster
cd OCRBACKUP
ls



Restore OCR file with the command below and stop the crs that has been started on exclusive mode.
./ocrconfig -restore  +FRA:/rac-cluster/OCRBACKUP/backup00.ocr.257.968870687
./crsctl stop crs



Start CRS normally.
 ./crsctl start crs



Verify CRS file on ASM storage.
cd +DATA/rac-cluster/OCRFILE
ls



Verify crs status on the node.
./crsctl stat res -t


Now the OCR file is restored successfully.  We can start crs on another node too.

Additional OCR file integrity verification commands:
#ocrcheck
$ocrcheck -config
$ocrcheck -local -config
$cluvfy comp ocr -n all -verbose
$cluvfy comp ocr -n rac01 -verbose
$cluvfy comp ocr -n rac02 -verbose