1. No category

Virtualized 11gR2 Linux RAC
on a
Virtual Box Host
Ahbaid Gaffoor
Amazon.com/A9.com
Overview
1.
2.
3.
4.
5.
6.
Single Node Configuration
Shared Storage
ASM
Cloning Nodes
Installing Oracle RAC
Verification
2
1. Single Node Configuration
1.
2.
3.
4.
5.
6.
7.
8.
9.
Required Packages
Memory Configuration
Network Configuration
Kernel Configuration
User Limits
Security
NTP
User and Group Creation
VBox Guest Additions
3
[root@n1 setup]# more packages.txt
kernel-headers-2.6.18-194.el5.x86_64.rpm
libgomp-4.4.0-6.el5.x86_64.rpm
binutils-2.17.50.0.6-14.el5.x86_64.rpm
compat-libstdc++-33-3.2.3-61.x86_64.rpm
elfutils-libelf-0.137-3.el5.x86_64.rpm
elfutils-libelf-devel-0.137-3.el5.x86_64.rpm
elfutils-libelf-devel-static-0.137-3.el5.x86_64.rpm
gcc-4.1.2-48.el5.x86_64.rpm
gcc-c++-4.1.2-48.el5.x86_64.rpm
glibc-2.5-49.x86_64.rpm
glibc-common-2.5-49.x86_64.rpm
glibc-devel-2.5-49.x86_64.rpm
glibc-headers-2.5-49.x86_64.rpm
ksh-20100202-1.el5.x86_64.rpm
libaio-0.3.106-5.x86_64.rpm
libaio-devel-0.3.106-5.x86_64.rpm
libgcc-4.1.2-48.el5.x86_64.rpm
libstdc++-4.1.2-48.el5.x86_64.rpm
libstdc++-devel-4.1.2-48.el5.x86_64.rpm
make-3.81-3.el5.x86_64.rpm
sysstat-7.0.2-3.el5.x86_64.rpm
unixODBC-2.2.11-7.1.x86_64.rpm
unixODBC-devel-2.2.11-7.1.x86_64.rpm
Packages – Install Script
[root@n1 setup]# more install_packages.sh
umount /dev/cdom 1>/dev/null 2>&1
mount /dev/cdrom /media 1>/dev/null 2>&1
export CD=/media/Server
for p in `cat packages.txt`; do
echo "# $CD/$p"
rpm -Uvh --nodeps $CD/$p
done
1.1 REQUIRED PACKAGES
Packages - List
Automatic Memory Management and /dev/shm
[root@n1 ~]# df -h -t tmpfs
Filesystem
Size Used Avail Use% Mounted on
tmpfs
2.0G 0 2.0G 0% /dev/shm
Size /dev/shm for MEMORY_TARGET shared memory allocation of 3G
[root@n1 ~]# df -h -t tmpfs
Filesystem
Size Used Avail Use% Mounted on
[root@n1 ~]# mount -t tmpfs shmfs -o size=3000m /dev/shm
[root@n1 ~]# df -h -t tmpfs
Filesystem
Size Used Avail Use% Mounted on
shmfs
3.0G 0 3.0G 0% /dev/shm
Edit /etc/fstab for persistence
[root@n1 ~]# cat /etc/fstab
/dev/VolGroup00/LogVol00 /
ext3 defaults
11
LABEL=/boot
/boot
ext3 defaults
12
#tmpfs
/dev/shm
tmpfs defaults
00
tmpfs
/dev/shm
tmpfs size=3000m 0 0
devpts
/dev/pts
devpts gid=5,mode=620 0 0
sysfs
/sys
sysfs defaults
00
proc
/proc
proc defaults
00
/dev/VolGroup00/LogVol01 swap
swap defaults
00
1.2 MEMORY CONFIGURATION
[root@n1 ~]# umount tmpfs
Public IPs
n1 199.1.1.61
n2 199.1.1.62
n1-vip 199.1.1.161
n2-vip 199.1.1.162
Private IPs
n1 10.0.0.61
n2 10.0.0.62
SCAN (Single Client Access Name) IPs (>= 11gR2)
n-scan 199.1.1.151
n-scan 199.1.1.152
eth0 private interconnect configuration
[root@n1 ~]# cat /etc/sysconfig/network-scripts/ifcfg-eth0
# Intel Corporation 82540EM Gigabit Ethernet Controller
DEVICE=eth0
BOOTPROTO=static
BROADCAST=199.1.1.255
HWADDR= 08:00:27:0F:D3:AF
IPADDR=199.1.1.61
NETMASK=255.255.255.0
NETWORK=199.1.1.0
ONBOOT=yes
1.3 NETWORK CONFIGURATION
VIP IPs (<= 11gR2)
n1 (& n2) /etc/hosts file:
# n1,n2 public interfaces
199.1.1.61
n1.ilmtech.com n1
199.1.1.62
n2.ilmtech.com n2
# n1,n2 internal interfaces
10.0.0.61
n1-int.ilmtech.com n1-int
10.0.0.62
n2-int.ilmtech.com n2-int
# n1,n2 vip interfaces
199.1.1.161
n1-vip.ilmtech.com n1-vip
199.1.1.162
n2-vip.ilmtech.com n2-vip
1.3 NETWORK CONFIGURATION
# Do not remove the following line, or various programs
# that require network functionality will fail.
127.0.0.1
localhost.localdomain localhost
::1
localhost6.localdomain6 localhost6
[root@n1 ~]# cat /etc/sysconfig/network-scripts/ifcfg-eth2
# Intel Corporation 82540EM Gigabit Ethernet Controller
DEVICE=eth2
BOOTPROTO=static
BROADCAST=10.0.0.255
HWADDR=08:00:27:78:3E:63
IPADDR=10.0.0.61
NETMASK=255.255.255.0
NETWORK=10.0.0.0
ONBOOT=yes
[root@n1 ~]# /etc/init.d/network start
Bringing up loopback interface:
Bringing up interface eth0:
Bringing up interface eth2:
[ OK ]
[ OK ]
[ OK ]
[root@n1 ~]# ifconfig eth2
eth2 Link encap:Ethernet HWaddr 08:00:27:78:3E:63
inet addr:10.0.0.61 Bcast:10.0.0.255 Mask:255.255.255.0
inet6 addr: fe80::a00:27ff:fe78:3e63/64 Scope:Link
UP BROADCAST RUNNING MULTICAST MTU:1500 Metric:1
RX packets:115 errors:0 dropped:0 overruns:0 frame:0
TX packets:35 errors:0 dropped:0 overruns:0 carrier:0
collisions:0 txqueuelen:1000
RX bytes:15079 (14.7 KiB) TX bytes:5792 (5.6 KiB)
[root@n1 ~]# ping -c 1 10.0.0.61
PING 10.0.0.61 (10.0.0.61) 56(84) bytes of data.
64 bytes from 10.0.0.61: icmp_seq=1 ttl=64 time=0.020 ms
--- 10.0.0.61 ping statistics --1 packets transmitted, 1 received, 0% packet loss, time 0ms
rtt min/avg/max/mdev = 0.020/0.020/0.020/0.000 ms
1.3 NETWORK CONFIGURATION
eth2 private interconnect configuration
Edit /etc/sysctl.conf
# Oracle 11gR2 configuration
# http://download.oracle.com/docs/cd/E11882_01/install.112/e16768/toc.htm#BHCCADGD
# concurrent outstanding requests
fs.aio-max-nr = 1048576
# Shared Memory
kernel.shmall = 2097152
kernel.shmmax = 536870912
kernel.shmmni = 4096
# Semaphores semmsl semmns semopm semmni
kernel.sem = 250 32000 100 128
# Network
net.ipv4.ip_local_port_range = 9000 65500
net.core.rmem_default = 262144
net.core.rmem_max = 4194304
net.core.wmem_default = 262144
net.core.wmem_max = 1048586
# Huge Pages for Oracle group 501 (oracle user group)
vm.hugetlb_shm_group=501
1.4 KERNEL CONFIGURATION
# files
fs.file-max = 6815744
[root@n1 ~]# /sbin/sysctl -p
net.ipv4.ip_forward = 0
net.ipv4.conf.default.rp_filter = 1
net.ipv4.conf.default.accept_source_route = 0
kernel.sysrq = 0
kernel.core_uses_pid = 1
net.ipv4.tcp_syncookies = 1
kernel.msgmnb = 65536
kernel.msgmax = 65536
kernel.shmall = 4294967296
fs.aio-max-nr = 1048576
fs.file-max = 6815744
kernel.shmall = 2097152
kernel.shmmax = 536870912
kernel.shmmni = 4096
kernel.sem = 250 32000 100 128
net.ipv4.ip_local_port_range = 9000 65500
net.core.rmem_default = 262144
net.core.rmem_max = 4194304
net.core.wmem_default = 262144
net.core.wmem_max = 1048586
vm.hugetlb_shm_group = 501
1.4 KERNEL CONFIGURATION
Apply/etc/sysctl.conf changes
Edit /etc/security/limits.conf
nproc 2047
nproc 16384
nofile 1024
nofile 65536
stack 10240
1.5 USER LIMITS
# oracle limits
oracle
soft
oracle
hard
oracle
soft
oracle
hard
oracle
soft
Edit /etc/pam.d/login
# oracle
session required
pam_limits.so
Edit /etc/selinux/config
1.6 SECURITY
# SELINUX= can take one of these three values:
#
enforcing - SELinux security policy is enforced.
#
permissive - SELinux prints warnings instead of enforcing.
#
disabled - SELinux is fully disabled.
SELINUX=disabled
Edit /etc/sysconfig/ntpd
# -x sets the the slew/step threshold to 600 seconds
OPTIONS="-x -u ntp:ntp -p /var/run/ntpd.pid“
Restart ntpd
[ OK ]
[ OK ]
[ OK ]
1.7 NTP
[root@n1 ~]# /etc/init.d/ntpd restart
Shutting down ntpd:
ntpd: Synchronizing with time server:
Starting ntpd:
Create dba and oinstall groups
[root@n1 ~]# cat /etc/sysctl.conf | grep ^vm.hugetlb_shm_group
vm.hugetlb_shm_group=501
[root@n1 ~]# groupadd -g 501 dba
[root@n1 ~]# groupadd -g 502 oinstall
[root@n1 oracle]# mkdir -p /u01/app
[root@n1 oracle]# useradd -m -d /u01/app/oracle -u 501 -g oinstall -G dba -s /bin/bash oracle
[root@n1 oracle]# passwd oracle
Changing password for user oracle.
New UNIX password:
Retype new UNIX password:
passwd: all authentication tokens updated successfully.
Edit /etc/sudoers
# Allow oracle user full sudo access
oracle ALL=(ALL)
ALL
Create grid and db directories (as oracle)
[root@n1 oracle]# su - oracle
[oracle@n1 ~]$ mkdir -p /u01/app/grid/11.2
[oracle@n1 ~]$ mkdir -p /u01/app/oracle/product/db/11.2
1.8 USER AND GROUP CREATION
Create oracle user
Edit ~oracle/.bash_profile on n1 to include
export GRID_HOME=/u01/app/grid/11.2
export GRID_BIN=$GRID_HOME/bin
export LD_LIBRARY_PATH=$ORACLE_HOME/lib:/lib:/usr/lib:$LD_LIBRARY_PATH
export CLASSPATH=$ORACLE_HOME/JRE:$ORACLE_HOME/jlib:$ORACLE_HOME/rdbms/jlib:$CLASSPATH
export PATH=$ORACLE_BIN:$PATH
export TMP=/tmp
export TMPDIR=$TMP
# Oracle host n1 settings
export ORACLE_HOSTNAME=n1.ilmtech.com
export ORACLE_SID=n1
1.8 USER AND GROUP CREATION
# Oracle Common Settings
export ORACLE_BASE=/u01/app/oracle
export ORACLE_HOME=$ORACLE_BASE/product/db/11.2
export ORACLE_BIN=$ORACLE_HOME/bin
export ORACLE_UNQNAME=n
Mount VirtualBox ISO image from Host
D:\Program Files\Sun\VirtualBox\VBoxGuestAdditions.iso
Attach using Host Window: Menu -> Devices -> CD/DVD Devices -> More CD/DVD Images…
Mount and run VirtualBox Guest Additions from Guest
[root@n1 ~]# df -h | grep media
/dev/hdc
33M 33M 0 100% /media/VBOXADDITIONS_3.2.8_64453
[root@n1 ~]# cd /media/VBOXADDITIONS_3.2.8_64453/
[root@n1 VBOXADDITIONS_3.2.8_64453]# sh ./VBoxLinuxAdditions-amd64.run
Why?
•
•
•
•
•
•
Mouse Pointer Integration
Improved Video Resolution
Seamless windows
Time Synchronization
Shared Clipboard
Automated Logins
1.9 Vbox Guest Additions
Mount as CD
2. Shared Storage
1. Shared Disk Creation
2. Attaching Shared Disks
3. Partitioning Shared Disks
17
VBoxManage.exe
D:\Program Files\Sun\VirtualBox\VBoxManage.exe
Shared Disks new in 3.2.8 - http://forums.virtualbox.org/viewtopic.php?f=1&t=33517
Create Shared Disks
C:\>VBoxManage.exe createhd --filename Q:\VirtualBox\SSD\rac\n\shared\rac-n-asm01-vdi.disk --size 10240 --format VDI
--variant Fixed --type shareable --remember
Oracle VM VirtualBox Command Line Management Interface Version 3.2.8
(C) 2005-2010 Oracle Corporation
All rights reserved.
0%...10%...20%...30%...40%...50%...60%...70%...80%...90%...100%
Disk image created. UUID: d18e5c34-6604-464f-a495-e9aaa925697c
2.1 SHARED DISK CREATION
VBoxManage.exe createhd --filename rac-n-asm01-vdi.disk –size 10240 --format VDI --variant Fixed --type shareable –remember
..
VBoxManage.exe createhd --filename rac-n-asm05-vdi.disk –size 10240 --format VDI --variant Fixed --type shareable –remember
Attach Shared Disks to n1
C:\>VBoxManage.exe storageattach n1 --storagectl "SATA Controller" --port 1 --device 0 --type hdd
--medium Q:\VirtualBox\SSD\rac\n\shared\rac-n-asm01-vdi.disk
Oracle VM VirtualBox Command Line Management Interface Version 3.2.8
(C) 2005-2010 Oracle Corporation
All rights reserved.
2.2 ATTACHING SHARED DISKS
VBoxManage.exe storageattach n1 --storagectl “SATA Controller” --port 1 --device 0 --type hdd --medium rac-n-asm01-vdi.disk
..
VBoxManage.exe storageattach n1 --storagectl “SATA Controller” --port 5 --device 0 --type hdd --medium rac-n-asm05-vdi.disk
Find scsi Devices (SATA Controller)
scsi0
[root@n1 scsi]# fdisk -l /dev/sda
Disk /dev/sda: 42.9 GB, 42949672960 bytes
255 heads, 63 sectors/track, 5221 cylinders
Units = cylinders of 16065 * 512 = 8225280 bytes
Device Boot
/dev/sda1 *
/dev/sda2
Start
1
14
End Blocks Id System
13 104391 83 Linux
5221 41833260 8e Linux LVM
2.3 SHARED DISK PARTITIONING
[root@n1 scsi]# cat /proc/scsi/scsi
Attached devices:
Host: scsi0 Channel: 00 Id: 00 Lun: 00
Vendor: ATA Model: VBOX HARDDISK Rev: 1.0
Type: Direct-Access
ANSI SCSI revision: 05
Host: scsi1 Channel: 00 Id: 00 Lun: 00
Vendor: ATA Model: VBOX HARDDISK Rev: 1.0
Type: Direct-Access
ANSI SCSI revision: 05
Host: scsi2 Channel: 00 Id: 00 Lun: 00
Vendor: ATA Model: VBOX HARDDISK Rev: 1.0
Type: Direct-Access
ANSI SCSI revision: 05
Host: scsi3 Channel: 00 Id: 00 Lun: 00
Vendor: ATA Model: VBOX HARDDISK Rev: 1.0
Type: Direct-Access
ANSI SCSI revision: 05
Host: scsi4 Channel: 00 Id: 00 Lun: 00
Vendor: ATA Model: VBOX HARDDISK Rev: 1.0
Type: Direct-Access
ANSI SCSI revision: 05
Host: scsi5 Channel: 00 Id: 00 Lun: 00
Vendor: ATA Model: VBOX HARDDISK Rev: 1.0
Type: Direct-Access
ANSI SCSI revision: 05
Create Partitions on /dev/sdb (scsi1) .. /dev/sdf (scsi5)
[root@n1 scsi]# parted /dev/sdb mklabel gpt
Information: Don't forget to update /etc/fstab, if necessary.
[root@n1 scsi]# parted /dev/sdb print
Model: ATA VBOX HARDDISK (scsi)
Disk /dev/sdb: 10.7GB
Sector size (logical/physical): 512B/512B
Partition Table: gpt
10.7GB
Number Start End Size File system Name Flags
Information: Don't forget to update /etc/fstab, if necessary.
[root@n1 scsi]# parted /dev/sdb mkpart primary 0 10700
Information: Don't forget to update /etc/fstab, if necessary.
[root@n1 scsi]# parted /dev/sdd print
Model: ATA VBOX HARDDISK (scsi)
Disk /dev/sdb: 10.7GB
Sector size (logical/physical): 512B/512B
Partition Table: gpt
Number Start End Size File system Name
1 17.4kB 10.7GB 10.7GB
primary
Flags
Information: Don't forget to update /etc/fstab, if necessary.
2.3 SHARED DISK PARTITIONING
parted takes size in MB, 10.7GB is 10700MB
3. ASM
1. ASM Lib Installation
2. ASM Lib Configuration
3. ASM Disks
22
ASM Lib Installation
ASM Lib files included with OELR5U5 under $CD/Server
[root@n1 ~]# ls /media/Server/oracleasm*
/media/Server/oracleasm-2.6.18-194.el5-2.0.5-1.el5.x86_64.rpm
/media/Server/oracleasm-2.6.18-194.el5debug-2.0.5-1.el5.x86_64.rpm
/media/Server/oracleasm-2.6.18-194.el5xen-2.0.5-1.el5.x86_64.rpm
/media/Server/oracleasm-support-2.1.3-1.el5.x86_64.rpm
Oracle ASM lib available at: http://www.oracle.com/technetwork/topics/linux/downloads/rhel5-084877.html
oracleasmlib-2.0.4-1.el5.x86_64.rpm
[root@n1 ASM]# rpm -ivh oracleasm-support-2.1.3-1.el5.x86_64.rpm
warning: oracleasm-support-2.1.3-1.el5.x86_64.rpm: Header V3 DSA signature: NOKEY, key ID 1e5e0159
Preparing...
########################################### [100%]
1:oracleasm-support ########################################### [100%]
[root@n1 ASM]# rpm -ivh oracleasm-2.6.18-164.el5-2.0.5-1.el5.x86_64.rpm
warning: oracleasm-2.6.18-164.el5-2.0.5-1.el5.x86_64.rpm: Header V3 DSA signature: NOKEY, key ID 1e5e0159
Preparing...
########################################### [100%]
1:oracleasm-2.6.18-164.el########################################### [100%]
[root@n1 ASM]# rpm -ivh oracleasmlib-2.0.4-1.el5.x86_64.rpm
warning: oracleasmlib-2.0.4-1.el5.x86_64.rpm: Header V3 DSA signature: NOKEY, key ID 1e5e0159
Preparing...
########################################### [100%]
1:oracleasmlib
########################################### [100%]
3.1 ASM Lib INSTALLATION
Kernel Version
[root@n1 ~]# echo `uname -r`.`uname -i`
2.6.18-194.el5.x86_64
ASM Lib Configuration
[root@n1 ~]# /usr/sbin/oracleasm configure -i
Configuring the Oracle ASM library driver.
Default user to own the driver interface []: oracle
Default group to own the driver interface []: dba
Start Oracle ASM library driver on boot (y/n) [n]: y
Scan for Oracle ASM disks on boot (y/n) [y]: y
Writing Oracle ASM library driver configuration: done
Load oracleasm module
[root@n1 ~]# /usr/sbin/oracleasm init
Loading module "oracleasm": oracleasm
Mounting ASMlib driver filesystem: /dev/oracleasm
Verify oracleasm module
[root@n1 ~]# /usr/sbin/oracleasm status
Checking if ASM is loaded: yes
Checking if /dev/oracleasm is mounted: yes
3.2 ASM Lib CONFIGURATION
This will configure the on-boot properties of the Oracle ASM library
driver. The following questions will determine whether the driver is
loaded on boot and what permissions it will have. The current values
will be shown in brackets ('[]'). Hitting <ENTER> without typing an
answer will keep that current value. Ctrl-C will abort.
Mark ASM Disk Header /dev/sdb .. /dev/sdf
[root@n1 ~]# which oracleasm
/usr/sbin/oracleasm
[root@n1 ~]# oracleasm createdisk DISK1 /dev/sdb1
Writing disk header: done
Instantiating disk: done
[root@n1 ~]# oracleasm createdisk DISK3 /dev/sdd1
Writing disk header: done
Instantiating disk: done
[root@n1 ~]# oracleasm createdisk DISK4 /dev/sde1
Writing disk header: done
Instantiating disk: done
[root@n1 ~]# oracleasm createdisk DISK5 /dev/sdf1
Writing disk header: done
Instantiating disk: done
List ASM Disks
[root@n1 ~]# oracleasm listdisks
DISK1
DISK2
DISK3
DISK4
DISK5
3.3 ASM DISKS
[root@n1 ~]# oracleasm createdisk DISK2 /dev/sdc1
Writing disk header: done
Instantiating disk: done
Oracle ASM and Devices
• ASM Disks can show up with different device paths
• Can use udev to control device naming
• Oracle ASM scans for disk header tag of “ORCLDISK”
• Oracle ASM scans for disk header label of “DISKn”
•Another node can mount disks on different devices and have ASM detect drives correctly
ASM Disk Header (tag and label)
Non ASM Disk Header
[root@n1 ~]# dd if=/dev/sda1 bs=128 count=1 | od --format=a
1+0 records in
1+0 records out
128 bytes (128 B) copied, 1.3e-05 seconds, 9.8 MB/s
0000000 nul nul nul nul nul nul nul nul nul nul nul nul nul nul nul nul
*
0000200
Tag: ORCLDISK
Label: DISK1
3.3 ASM DISKS
[root@n1 ~]# dd if=/dev/sdb1 bs=128 count=1 | od --format=a
1+0 records in
1+0 records out
128 bytes (128 B) copied, 0.011425 seconds, 11.2 kB/s
0000000 nul nul nul nul nul nul nul nul nul nul nul nul u I S K
0000020 nul nul nul nul nul nul nul nul nul nul nul nul nul nul nul nul
0000040 O R C L D I S K D I S K 1 nul nul nul
0000060 nul nul nul nul nul nul nul nul nul nul nul nul nul nul nul nul
*
0000200
Oracle ASM Disk Devices
[root@n1 ~]# cd /dev/oracleasm/disks
0 Sep 15 17:54 .
0 Sep 15 17:54 ..
8, 17 Sep 15 18:02 DISK1
8, 33 Sep 15 18:02 DISK2
8, 49 Sep 15 18:02 DISK3
8, 65 Sep 15 18:02 DISK4
8, 81 Sep 15 18:02 DISK5
Major device number 8:
http://www.kernel.org/pub/linux/docs/device-list/devices.txt
8
block
SCSI disk devices (0-15)
0 = /dev/sda First SCSI disk whole disk
16 = /dev/sdb Second SCSI disk whole disk
32 = /dev/sdc Third SCSI disk whole disk
...
240 = /dev/sdp Sixteenth SCSI disk whole disk
3.3 ASM DISKS
[root@n1 disks]# ls -al
total 0
drwxr-xr-x 1 root root
drwxr-xr-x 4 root root
brw-rw---- 1 oracle dba
brw-rw---- 1 oracle dba
brw-rw---- 1 oracle dba
brw-rw---- 1 oracle dba
brw-rw---- 1 oracle dba
4. Cloning Nodes
1.
2.
3.
4.
5.
6.
Cloning n1 (non-ASM) Disks
Create n2 Guest
Change Hostname
Change Network Addresses
Attach Shared Storage
Verify ASM Disks on n1 & n2
28
Shutdown n1
[root@n1 ~]# shutdown –h now
Clone n1 non-ASM hard disk n1-disk0.vdi to n2-disk0.vdi
C:\>VBoxManage.exe clonehd "X:\Virtual Machines\virtualbox\11gRAC\n1-disk0.vdi"
"X:\Virtual Machines\virtualbox\11gRAC\n2-disk0.vdi" --remember
Oracle VM VirtualBox Command Line Management Interface Version 3.2.8
(C) 2005-2010 Oracle Corporation
All rights reserved.
0%...10%...20%...30%...40%...50%...60%...70%...80%...90%...100%
Clone hard disk created in format 'VDI'. UUID: 0b80e4d8-e8f7-4ccb-96fb-3d5769d45b53
4.1 CLONING n1 (non-ASM) DISKS
VBoxManage.exe clonehd <sourcefile> <targetfile> –remember
Create n2 Guest
4.2 CREATE n2 GUEST (using cloned disk)
• Same as n2
• Use cloned hard disk n2-disk0.vdi
• Same networking interfaces eth0, eth1: bridged and eth2 : host only
4.2 CREATE n2 GUEST (using cloned disk)
Create n2 Guest
4.2 CREATE n2 GUEST (using cloned disk)
Create n2 Guest
[root@n1 ~]# cat /etc/sysconfig/network-scripts/ifcfg-eth0
# Intel Corporation 82540EM Gigabit Ethernet Controller
DEVICE=eth0
BOOTPROTO=static
BROADCAST=199.1.1.255
HWADDR=08:00:27:09:0f:7d
IPADDR=199.1.1.62
NETMASK=255.255.255.0
NETWORK=199.1.1.0
ONBOOT=yes
Edit ifcfg-eth2 network script
[root@n1 ~]# cat /etc/sysconfig/network-scripts/ifcfg-eth2
# Intel Corporation 82540EM Gigabit Ethernet Controller
DEVICE=eth2
BOOTPROTO=static
BROADCAST=10.0.0.255
HWADDR=08:00:27:1b:07:ee
IPADDR=10.0.0.62
NETMASK=255.255.255.0
NETWORK=10.0.0.0
ONBOOT=yes
Reboot n2:
[root@n1 ~]# shutdown –r now
Hostname now n2:
[root@n2 ~]# hostname
n2.ilmtech.com
4.4 CHANGE NETWORK ADDRESSES
Edit ifcfg-eth0 network script
Attach Shared Disks to n2
C:\>VBoxManage.exe storageattach n2 --storagectl "SATA Controller" --port 1 --device 0 --type hdd
--medium Q:\VirtualBox\SSD\rac\n\shared\rac-n-asm01-vdi.disk
Oracle VM VirtualBox Command Line Management Interface Version 3.2.8
(C) 2005-2010 Oracle Corporation
All rights reserved.
4.5 ATTACH SHARED STORAGE (to n2)
VBoxManage.exe storageattach n2 --storagectl “SATA Controller” --port 1 --device 0 --type hdd --medium rac-n-asm01-vdi.disk
..
VBoxManage.exe storageattach n2 --storagectl “SATA Controller” --port 5 --device 0 --type hdd --medium rac-n-asm05-vdi.disk
Attach Shared Disks to n2
C:\>VBoxManage.exe storageattach n2 --storagectl "SATA Controller" --port 1 --device 0 --type hdd
--medium Q:\VirtualBox\SSD\rac\n\shared\rac-n-asm01-vdi.disk
Oracle VM VirtualBox Command Line Management Interface Version 3.2.8
(C) 2005-2010 Oracle Corporation
All rights reserved.
4.5 ATTACH SHARED STORAGE (to n2)
VBoxManage.exe storageattach n2 --storagectl “SATA Controller” --port 1 --device 0 --type hdd --medium rac-n-asm01-vdi.disk
..
VBoxManage.exe storageattach n2 --storagectl “SATA Controller” --port 5 --device 0 --type hdd --medium rac-n-asm05-vdi.disk
Verify Oracle ASM Disk Devices on n1 and n2:
[root@n2 ~]# oracleasm listdisks
DISK1
DISK2
DISK3
DISK4
DISK5
4.6 VERIFY ASM DISKS
[root@n1 ~]# oracleasm listdisks
DISK1
DISK2
DISK3
DISK4
DISK5
5. Installing Oracle RAC
1.
2.
3.
4.
5.
6.
Environment
Cluster Verification
Grid Infrastructure
Database Binaries
Database Creation – DBCA
Verification
37
Environment Variables
[oracle@n2 ~]$ set | grep ^ORACLE
ORACLE_BASE=/u01/app/oracle
ORACLE_BIN=/u01/app/oracle/product/db/11.2/bin
ORACLE_HOME=/u01/app/oracle/product/db/11.2
ORACLE_HOSTNAME=n2.ilmtech.com
ORACLE_SID=n2
ORACLE_UNQNAME=n
SSH equivalence
[oracle@n1 ~]$ ssh-keygen -t rsa
Generating public/private rsa key pair.
Enter file in which to save the key (//u01/app/oracle/.ssh/id_rsa):
Created directory '/u01/app/oracle/.ssh'.
Enter passphrase (empty for no passphrase):
Enter same passphrase again:
Your identification has been saved in /u01/app/oracle/.ssh/id_rsa.
Your public key has been saved in /u01/app/oracle/.ssh/id_rsa.pub.
The key fingerprint is:
fb:01:95:10:de:55:fd:9d:ae:cf:c7:ed:9a:e6:01:59 [email protected]
[oracle@n2 ~]$ cd ~/.ssh/
[oracle@n2 .ssh]$ cp id_rsa.pub authorized_keys
[oracle@n2 .ssh]$ tar -cvf s.tar *
authorized_keys
id_rsa
id_rsa.pub
# scp s.tar to n1, create ~oracle/.ssh with same privs, untar s.tar there
5.1 Environment
[oracle@n1 ~]$ set | grep ^ORACLE
ORACLE_BASE=/u01/app/oracle
ORACLE_BIN=/u01/app/oracle/product/db/11.2/bin
ORACLE_HOME=/u01/app/oracle/product/db/11.2
ORACLE_HOSTNAME=n1.ilmtech.com
ORACLE_SID=n1
ORACLE_UNQNAME=n
Cluster Verification – runcluvfy.sh (Grid Infrastructure Install Directory)
[oracle@n1 grid]$ pwd
/u01/app/oracle/installer/grid
[oracle@n1 grid]$ ./runcluvfy.sh stage -pre crsinst -n n1,n2 -verbose 1>runcluvfy.log 2>&1
[oracle@n1 grid]$ sudo rpm -qa | grep glibc-devel
[sudo] password for oracle:
glibc-devel-2.5-49
[oracle@n1 grid]$ ssh root@n2 rpm -qa | grep glibc-devel
glibc-devel-2.5-49
Cluster Verification – runcluvfy.sh (some checks)
[oracle@n1 grid]$ cat runcluvfy.log | grep -i ^Checking
Checking node reachability...
Checking user equivalence...
Checking node connectivity...
Checking hosts config file...
Checking for multiple users with UID value 0
Checking Core file name pattern consistency...
Checking to make sure user "oracle" is not in "root" group
Checking daemon liveness...
Checking NTP daemon command line for slewing option "-x"
Checking NTP daemon's boot time configuration, in file "/etc/sysconfig/ntpd", for slewing option "-x"
Checking on nodes "[n2, n1]"...
…..
(Full listing in Appendix A)
…..
5.2 Cluster Verification
[oracle@n1 grid]$ cat runcluvfy.log | grep -i failed
n2
missing
glibc-devel-2.5 (i386) failed
n1
missing
glibc-devel-2.5 (i386) failed
Result: Package existence check failed for "glibc-devel-2.5 (i386)"
Grid Infrastrucutre - Installation
[oracle@n1 grid]$ pwd
/u01/app/oracle/installer/grid
[oracle@n1 grid]$ ./runInstaller
5.3 Grid Infrastructure Installation
• Run from Grid Infrastructure Installer directory
• Both n1 and n2 are up
• ssh equivalence in place (runcluvfy.sh)
Grid Infrastrucutre - Installation
5.3 Grid Infrastructure Installation
• “Install and Configure Grid Infrastructure for a Cluster”
Grid Infrastrucutre - Installation
5.3 Grid Infrastructure Installation
• “Typical Installation”
Grid Infrastrucutre - Installation
[oracle@n1 grid]$ dig +short n-scan.ilmtech.com
199.1.1.151
199.1.1.152
5.3 Grid Infrastructure Installation
• SCAN Name: n-scan (DNS entry)
• Can use /etc/hosts but ignore errors on n-scan lookups
Grid Infrastrucutre - Installation
5.3 Grid Infrastructure Installation
• Add node n2
Grid Infrastrucutre - Installation
5.3 Grid Infrastructure Installation
• Set SCAN Name: “n-scan”
Grid Infrastrucutre - Installation
5.3 Grid Infrastructure Installation
• Specify Public and Private network interfaces
•
•
•
•
•
Oracle Base for Grid Infrastructure
Grid Home
Cluster Registry Type – ASM
Password – Complex! (Oracl3ASM in my example)
OSASM Group - dba
5.3 Grid Infrastructure Installation
Grid Infrastrucutre - Installation
Grid Infrastrucutre - Installation
5.3 Grid Infrastructure Installation
• Redundancy – External
• Candidate Disks (all ASM disks)
Grid Infrastrucutre - Installation
5.3 Grid Infrastructure Installation
• Grid Inventory Location
Grid Infrastrucutre - Installation
5.3 Grid Infrastructure Installation
• Prerequisite Checks (Ignoring glibc-devel-2.5)
Grid Infrastrucutre - Installation
5.3 Grid Infrastructure Installation
• Review Summary
Grid Infrastrucutre - Installation
5.3 Grid Infrastructure Installation
• Installing….
Grid Infrastrucutre - Installation
5.3 Grid Infrastructure Installation
• Installing….
Grid Infrastrucutre - Installation
5.3 Grid Infrastructure Installation
• Root scripts on n1 and n2
Grid Infrastrucutre - Installation
[root@n1 ~]# /u01/app/oraInventory/orainstRoot.sh
Changing permissions of /u01/app/oraInventory.
Adding read,write permissions for group.
Removing read,write,execute permissions for world.
Changing groupname of /u01/app/oraInventory to oinstall.
The execution of the script is complete.
5.3 Grid Infrastructure Installation
• Root scripts on n1 and n2
• /u01/app/oraInventory/orainstRoot.sh
Grid Infrastrucutre - Installation
• Root scripts on n1 and n2
• /u01/app/11.2.0/grid/root.sh
The following environment variables are set as:
ORACLE_OWNER= oracle
ORACLE_HOME= /u01/app/11.2.0/grid
Enter the full pathname of the local bin directory: [/usr/local/bin]:
Copying dbhome to /usr/local/bin ...
Copying oraenv to /usr/local/bin ...
Copying coraenv to /usr/local/bin ...
Creating /etc/oratab file...
Entries will be added to the /etc/oratab file as needed by
Database Configuration Assistant when a database is created
Finished running generic part of root.sh script.
……..
(Full listing is in Appendix B)
……..
5.3 Grid Infrastructure Installation
[root@n1 ~]# /u01/app/11.2.0/grid/root.sh
Running Oracle 11g root.sh script...
Grid Infrastrucutre - Installation
5.3 Grid Infrastructure Installation
• After running scripts, installation continues
Grid Infrastrucutre - Installation
5.3 Grid Infrastructure Installation
• Installation Successful
Grid Infrastrucutre - Installation
5.3 Grid Infrastructure Installation
• Installation Done
Database Binaries
• Run from Database Installer directory
• Both n1 and n2 are up
• ssh equivalence in place
[oracle@n1 grid]$ pwd
/u01/app/oracle/installer/database
5.4 Database Binaries
[oracle@n1 grid]$ ./runInstaller
Database Binaries
5.4 Database Binaries
• Uncheck Oracle Support Security Updates
• Accept warning
Database Binaries
5.4 Database Binaries
• Choose “Create and configure a database”
Database Binaries
5.4 Database Binaries
• Choose “Server Class”
Database Binaries
5.4 Database Binaries
• Choose “Real Application Clusters database installation”
• Ensure both nodes n1 & n2 are checked
Database Binaries
5.4 Database Binaries
• Choose “Typical install”
Database Binaries
5.4 Database Binaries
• Storage – ASM
• Password Complexity (Oracl3ASM used in example)
• Global database nane – n.<domain>
Database Binaries
5.4 Database Binaries
• Prerequisite Checks
• Ignoring glibc-devel-2.5 (verified it’s installed)
Database Binaries
5.4 Database Binaries
• Summary
Database Binaries
5.4 Database Binaries
• Installing…..
Database Binaries
5.4 Database Binaries
• Installing…..
Database Binaries
5.4 Database Binaries
• Done (skipped DBCA)
Database Creation - DBCA
• Run from Database Home
• Both n1 and n2 are up
• ssh equivalence in place
[oracle@n1 11.2]$ ./dbca
5.5 Database Creation
[oracle@n1 11.2]$ pwd
/u01/app/oracle/product/db/11.2
Database Creation - DBCA
5.5 Database Creation
• Choose “Oracle Real Application Clusters database”
Database Creation - DBCA
5.5 Database Creation
• “Create a Database”
Database Creation - DBCA
5.5 Database Creation
• “General Purpose”
Database Creation - DBCA
5.5 Database Creation
• Cluster database configuration
Database Creation - DBCA
5.5 Database Creation
• No Enterprise Manager
Database Creation - DBCA
5.5 Database Creation
• Passwords same
• Oracl3ASM to be consistent in this example
Database Creation - DBCA
5.5 Database Creation
• Storage Type ASM
• Use OMF in +DATA (ASM)
Database Creation - DBCA
5.5 Database Creation
• Use Flash Recovery
• Use +DATA (ASM) for Flash Recovery Area
Database Creation - DBCA
5.5 Database Creation
• Skip Sample Schemas
• No Custom Scripts
Database Creation - DBCA
5.5 Database Creation
• Memory, Sizing, Char Sets, Connection Mode
• Accept Defaults (Guest VM has enough RAM allocated)
Database Creation - DBCA
5.5 Database Creation
• Storage in +DATA (ASM)
Database Creation - DBCA
5.5 Database Creation
• Create Database
• Save and review the scripts
Database Creation - DBCA
5.5 Database Creation
• Scripts generating
• Ok on HTML dialog that comes next
Database Creation - DBCA
5.5 Database Creation
• Installing
Database Creation - DBCA
5.5 Database Creation
• Installing…..
Database Creation - DBCA
5.5 Database Creation
• Done
Verification
• Instances
• Database
[oracle@n1 ~]$ ps -ef|grep pmon|grep -v grep
oracle 3569 1 0 04:20 ?
00:00:00 asm_pmon_+ASM1
oracle 6185 1 0 04:34 ?
00:00:00 ora_pmon_n1
[oracle@n2 ~]$ ps -ef|grep pmon|grep -v grep
oracle 3722 1 0 04:20 ?
00:00:00 asm_pmon_+ASM2
oracle 6047 1 0 04:35 ?
00:00:00 ora_pmon_n2
[oracle@n2 ~]$ srvctl config database
n
[oracle@n2 ~]$ srvctl config database –d n
Database unique name: n
Database name: n
Oracle home: /u01/app/oracle/product/db/11.2
Oracle user: oracle
Spfile: +DATA/n/spfilen.ora
Domain: ilmtech.com
Start options: open
Stop options: immediate
Database role: PRIMARY
Management policy: AUTOMATIC
Server pools: n
Database instances: n1,n2
Disk Groups: DATA
Services:
Database is administrator managed
5.6 Verification
[oracle@n1 ~]$ srvctl config database
n
Verification
• Query instances
[oracle@n2 ~]$ sqlplus "/ as sysdba"
SQL*Plus: Release 11.2.0.1.0 Production on Thu Nov 11 05:04:32 2010
Connected to:
Oracle Database 11g Enterprise Edition Release 11.2.0.1.0 - 64bit Production
With the Partitioning, Real Application Clusters, Automatic Storage Management, OLAP,
Data Mining and Real Application Testing options
SQL> select * from v$active_instances;
INST_NUMBER INST_NAME
----------- -----------------------------------------------------------1 n1.ilmtech.com:n1
2 n2.ilmtech.com:n2
5.6 Verification
Copyright (c) 1982, 2009, Oracle. All rights reserved.
References
• Books:
• Linux Recipes for Oracle DBAs by Kuhn, Kim and Lopez (Apress)
• Oracle Automatic Storage Management by Vengurlekar, Vallath, Long
(Oracle Press)
• DNS and Bind by Liu and Albitz (O’Reilly)
References
• Web:
• http://www.oracle-base.com/articles/rac/ArticlesRac.php#11gR2
• http://tahiti.oracle.com documentation
Appendix A : runcluvfy.log
./runcluvfy.sh stage -pre crsinst -n n1,n2 -verbose 1>runcluvfy.log 2>&1
Appendix B : root.log
[root@n1 ~]# /u01/app/11.2.0/grid/root.sh