标签:RAC MB dev 11grac asm File Oracle disk OCR2
【11grac】Oracle RAC 更换存储实验
回到未来望过去 2019-05-14 10:49:45 478 收藏 2 分类专栏: rac rac专栏收录该内容 27 篇文章0 订阅 订阅专栏实验环境准备:
RHEL 6.5 + Oracle 11.2.0.4 RAC (2nodes)
- OCR和Voting Disk使用的是OCR1磁盘组,底层对应3个1G大小的共享LUN,一般冗余;
- DATA使用的是DATA1磁盘组,底层对应的是3个5G大小的共享LUN,外部冗余;
- FRA使用的是FRA1磁盘组,底层对应的是1个5G大小的共享LUN,外部冗余;
- 额外添加7个共享LUN(3个1G大小,4个5G大小),用来模拟新的存储映射出来的共享LUN;
我这里是使用Oracle VM VirtualBox 4.3虚拟机软件模拟这个环境,
默认初始的RAC环境已经安装完毕,且已经存有业务数据。
对于RAC环境的安装可参考:
- Linux平台 Oracle 11gR2 RAC安装Part1:准备工作
- Linux平台 Oracle 11gR2 RAC安装Part2:GI安装
- Linux平台 Oracle 11gR2 RAC安装Part3:DB安装
模拟的业务数据可参考 模拟业务最小测试用例01 第1~5步进行初始化。
初始的RAC环境配置信息:
- #public ip
- 192.168.56.150 jyrac1
- 192.168.56.152 jyrac2
- #private ip
- 10.10.10.11 jyrac1-priv
- 10.10.10.12 jyrac2-priv
- #virtual ip
- 192.168.56.151 jyrac1-vip
- 192.168.56.153 jyrac2-vip
- #scan ip
- 192.168.56.160 jyrac-scan
本次需求:
更换RAC的共享存储,即把之前的存储设备划出的共享LUN上的所有数据全部迁移新的存储设备划分的LUN上。
具体就涉及到OCR,voting disk迁移,DATA迁移。
在我这里实验具体就是指OCR1,DATA1,FRA1磁盘组的迁移。
一、准备工作
二、更换存储
一、准备工作
1.1查看asm磁盘组
select name, TOTAL_MB, FREE_MB, state, type, COMPATIBILITY from v$asm_diskgroup;
- SQL> select name, TOTAL_MB, FREE_MB, state, type, COMPATIBILITY from v$asm_diskgroup;
- NAME TOTAL_MB FREE_MB STATE TYPE COMPATIBILITY
- DATA1 15360 13502 MOUNTED EXTERN 11.2.0.0.0
- FRA1 5120 4790 MOUNTED EXTERN 11.2.0.0.0
- OCR1 3072 2146 MOUNTED NORMAL 11.2.0.0.0
1.2查看asm的磁盘信息
select GROUP_NUMBER, DISK_NUMBER, TOTAL_MB, FREE_MB, NAME, PATH from v$asm_disk order by 1,2;
- SQL> set linesize 1000
- SQL> r
- 1* select GROUP_NUMBER, DISK_NUMBER, TOTAL_MB, FREE_MB, NAME, PATH from v$asm_disk order by 1,2
- GROUP_NUMBER DISK_NUMBER TOTAL_MB FREE_MB NAME PATH
- 1 0 1024 714 OCR1_0000 /dev/asm-diskb
- 1 1 1024 716 OCR1_0001 /dev/asm-diskc
- 1 2 1024 716 OCR1_0002 /dev/asm-diskd
- 2 0 5120 4552 DATA1_0000 /dev/asm-diske
- 2 1 5120 4549 DATA1_0001 /dev/asm-diskf
- 2 2 5120 4549 DATA1_0002 /dev/asm-diskg
- 3 0 5120 4790 FRA1_0000 /dev/asm-diskh
- 7 rows selected.
1.3查看存储设备信息,确定新的存储
这里实验环境是使用virtualbox模拟添加7个大小和之前一样的共享存储;
即分别为 1G大小的3个(OCR2),5G大小的3个(DATA1),5G大小的1个(FRA1)。
NewStorage1
NewStorage2
NewStorage3
NewStorage4
NewStorage5
NewStorage6
NewStorage7
在系统中就是/dev/sdi j k l m n o
udev绑定(两个节点root执行)
- --New Shared Disks
- for i in i j k l m n o;
- do
- echo "KERNEL==\"sd*\", BUS==\"scsi\", PROGRAM==\"/sbin/scsi_id --whitelisted --replace-whitespace --device=/dev/\$name\", RESULT==\"`/sbin/scsi_id --whitelisted --replace-whitespace --device=/dev/sd$i`\", NAME=\"asm-disk$i\", OWNER=\"grid\", GROUP=\"asmadmin\", MODE=\"0660\"" >> /etc/udev/rules.d/99-oracle-asmdevices.rules
- done
udev绑定后对应就是asm-diski,j,k,l,m,n,o
start_udev (两个节点root执行)
查看某个磁盘大小:
- [root@jyrac1 ~]# fdisk -l /dev/asm-diski
- Disk /dev/asm-diski: 1073 MB, 1073741824 bytes
- 255 heads, 63 sectors/track, 130 cylinders
- Units = cylinders of 16065 * 512 = 8225280 bytes
- Sector size (logical/physical): 512 bytes / 512 bytes
- I/O size (minimum/optimal): 512 bytes / 512 bytes
- Disk identifier: 0x00000000
确定/dev/asm-diski,/dev/asm-diskj,/dev/asm-diskk对应的是新加的1G大小的3个盘
用于创建新的OCR2
二、更换存储
2.1迁移OCR1到OCR2
2.1.1 创建新的OCR2磁盘组
su – grid
sqlplus / as sysasm
CREATE DISKGROUP OCR2 NORMAL REDUNDANCY DISK '/dev/asm-diski', '/dev/asm-diskj', '/dev/asm-diskk' ATTRIBUTE 'compatible.asm'='11.2';
- SQL> CREATE DISKGROUP OCR2 NORMAL REDUNDANCY DISK '/dev/asm-diski', '/dev/asm-diskj', '/dev/asm-diskk' ATTRIBUTE 'compatible.asm'='11.2';
- Diskgroup created.
查看两个节点是否添加了OCR2:
- --节点1:
- SQL> r
- 1* select name, TOTAL_MB, FREE_MB, state, type, COMPATIBILITY from v$asm_diskgroup
- NAME TOTAL_MB FREE_MB STATE TYPE COMPATIBILITY
- ------------------------------------------------------------ ---------- ---------- ---------------------- ------------ ------------------------------------------------------------------------------------------------------------------------
- DATA1 15360 13502 MOUNTED EXTERN 11.2.0.0.0
- FRA1 5120 4790 MOUNTED EXTERN 11.2.0.0.0
- OCR1 3072 2146 MOUNTED NORMAL 11.2.0.0.0
- OCR2 3072 2913 MOUNTED NORMAL 11.2.0.0.0
- --节点2:
- SQL> select name, TOTAL_MB, FREE_MB, state, type, COMPATIBILITY from v$asm_diskgroup
- 2 ;
- NAME TOTAL_MB FREE_MB STATE TYPE COMPATIBILITY
- ------------------------------------------------------------ ---------- ---------- ---------------------- ------------ ------------------------------------------------------------------------------------------------------------------------
- DATA1 15360 13502 MOUNTED EXTERN 11.2.0.0.0
- FRA1 5120 4790 MOUNTED EXTERN 11.2.0.0.0
- OCR1 3072 2146 MOUNTED NORMAL 11.2.0.0.0
- OCR2 0 0 DISMOUNTED 0.0.0.0.0
发现节点2需要手动mount新加的ASM磁盘组
sqlplus / as sysasm
- SQL> alter diskgroup OCR2 mount;
- Diskgroup altered.
2.1.2 添加OCR信息到OCR2
more /etc/oracle/ocr.loc
ocrconfig -add +OCR2
ocrcheck -config
more /etc/oracle/ocr.loc
- [root@jyrac1 ~]# more /etc/oracle/ocr.loc
- ocrconfig_loc=+OCR1
- local_only=FALSE
- [root@jyrac1 ~]# ocrconfig -add +OCR2
- [root@jyrac1 ~]#
- [root@jyrac1 ~]# ocrcheck -config
- Oracle Cluster Registry configuration is :
- Device/File Name : +OCR1
- Device/File Name : +OCR2
- [root@jyrac1 ~]# more /etc/oracle/ocr.loc
- #Device/file getting replaced by device +OCR2
- ocrconfig_loc=+OCR1
- ocrmirrorconfig_loc=+OCR2
- local_only=false
- [root@jyrac1 ~]# ocrcheck && crsctl query css votedisk
- Status of Oracle Cluster Registry is as follows :
- Version : 3
- Total space (kbytes) : 262120
- Used space (kbytes) : 3116
- Available space (kbytes) : 259004
- ID : 1718040627
- Device/File Name : +OCR1
- Device/File integrity check succeeded
- Device/File Name : +OCR2
- Device/File integrity check succeeded
- Device/File not configured
- Device/File not configured
- Device/File not configured
- Cluster registry integrity check succeeded
- Logical corruption check succeeded
- ## STATE File Universal Id File Name Disk group
- -- ----- ----------------- --------- ---------
- 1. ONLINE ab583d47c34a4f0ebfd46c8b04d7332b (/dev/asm-diskb) [OCR1]
- 2. ONLINE 853e95652d994fe3bfdf21ca23e65dfc (/dev/asm-diskc) [OCR1]
- 3. ONLINE 9f712a096a7b4f9bbfd811bff6462157 (/dev/asm-diskd) [OCR1]
- Located 3 voting disk(s).
2.1.3迁移VOTE DISK到新创建的ASM磁盘组OCR2上
crsctl replace votedisk +OCR2 (一个节点root执行)
先查询:
- SQL> select GROUP_NUMBER,NAME,STATE,type,TOTAL_MB,free_mb,VOTING_FILES,COMPATIBILITY from v$asm_diskgroup;
- GROUP_NUMBER NAME STATE TYPE TOTAL_MB FREE_MB VO COMPATIBILITY
- ------------ ------------------------------------------------------------ ---------------------- ------------ ---------- ---------- -- ------------------------------------------------------------------------------------------------------------------------
- 1 DATA1 MOUNTED EXTERN 15360 13502 N 11.2.0.0.0
- 2 FRA1 MOUNTED EXTERN 5120 4790 N 11.2.0.0.0
- 3 OCR1 MOUNTED NORMAL 3072 2146 Y 11.2.0.0.0
- 4 OCR2 MOUNTED NORMAL 3072 2244 N 11.2.0.0.0
替换为新添加的OCR2:
- [root@jyrac1 ~]# crsctl replace votedisk +OCR2
- Successful addition of voting disk 83f3dea9f4c24fccbf05eaa163c5c087.
- Successful addition of voting disk ad21c1f0452e4f4abff906127ff9fcc3.
- Successful addition of voting disk 2771819c65984f85bf366e74ee85781a.
- Successful deletion of voting disk ab583d47c34a4f0ebfd46c8b04d7332b.
- Successful deletion of voting disk 853e95652d994fe3bfdf21ca23e65dfc.
- Successful deletion of voting disk 9f712a096a7b4f9bbfd811bff6462157.
- Successfully replaced voting disk group with +OCR2.
- CRS-4266: Voting file(s) successfully replaced
再查询:
- SQL> r
- 1* select GROUP_NUMBER,NAME,STATE,type,TOTAL_MB,free_mb,VOTING_FILES,COMPATIBILITY from v$asm_diskgroup
- GROUP_NUMBER NAME STATE TYPE TOTAL_MB FREE_MB VO COMPATIBILITY
- ------------ ------------------------------------------------------------ ---------------------- ------------ ---------- ---------- -- ------------------------------------------------------------------------------------------------------------------------
- 1 DATA1 MOUNTED EXTERN 15360 13502 N 11.2.0.0.0
- 2 FRA1 MOUNTED EXTERN 5120 4790 N 11.2.0.0.0
- 3 OCR1 MOUNTED NORMAL 3072 2242 N 11.2.0.0.0
- 4 OCR2 MOUNTED NORMAL 3072 2148 Y 11.2.0.0.0
发现已经voting file已经在OCR2上,然后在两个节点root查询:
ocrcheck && crsctl query css votedisk
- [root@jyrac1 ~]# ocrcheck && crsctl query css votedisk
- Status of Oracle Cluster Registry is as follows :
- Version : 3
- Total space (kbytes) : 262120
- Used space (kbytes) : 3116
- Available space (kbytes) : 259004
- ID : 1718040627
- Device/File Name : +OCR1
- Device/File integrity check succeeded
- Device/File Name : +OCR2
- Device/File integrity check succeeded
- Device/File not configured
- Device/File not configured
- Device/File not configured
- Cluster registry integrity check succeeded
- Logical corruption check succeeded
- ## STATE File Universal Id File Name Disk group
- -- ----- ----------------- --------- ---------
- 1. ONLINE 83f3dea9f4c24fccbf05eaa163c5c087 (/dev/asm-diski) [OCR2]
- 2. ONLINE ad21c1f0452e4f4abff906127ff9fcc3 (/dev/asm-diskj) [OCR2]
- 3. ONLINE 2771819c65984f85bf366e74ee85781a (/dev/asm-diskk) [OCR2]
- Located 3 voting disk(s).
- [root@jyrac2 ~]# ocrcheck && crsctl query css votedisk
- Status of Oracle Cluster Registry is as follows :
- Version : 3
- Total space (kbytes) : 262120
- Used space (kbytes) : 3116
- Available space (kbytes) : 259004
- ID : 1718040627
- Device/File Name : +OCR1
- Device/File integrity check succeeded
- Device/File Name : +OCR2
- Device/File integrity check succeeded
- Device/File not configured
- Device/File not configured
- Device/File not configured
- Cluster registry integrity check succeeded
- Logical corruption check succeeded
- ## STATE File Universal Id File Name Disk group
- -- ----- ----------------- --------- ---------
- 1. ONLINE 83f3dea9f4c24fccbf05eaa163c5c087 (/dev/asm-diski) [OCR2]
- 2. ONLINE ad21c1f0452e4f4abff906127ff9fcc3 (/dev/asm-diskj) [OCR2]
- 3. ONLINE 2771819c65984f85bf366e74ee85781a (/dev/asm-diskk) [OCR2]
- Located 3 voting disk(s).
2.1.4 创建ASM实例spfile到新创建的OCR2 ASM磁盘组上
一个节点grid用户登录ASM实例执行:
create pfile='/tmp/asmpfile.ora' from spfile;
create spfile='+OCR2' from pfile='/tmp/asmpfile.ora';
- SQL> show parameter pfile
- NAME TYPE VALUE
- ------------------------------------ ---------------------- ------------------------------
- spfile string +OCR1/jyrac-scan/asmparameterf
- ile/registry.253.919993739
- SQL> create pfile='/tmp/asmpfile.ora' from spfile;
- File created.
- SQL> create spfile='+OCR2' from pfile='/tmp/asmpfile.ora';
- File created.
- SQL> show parameter pfile
- NAME TYPE VALUE
- ------------------------------------ ---------------------- ------------------------------
- spfile string +OCR1/jyrac-scan/asmparameterf
- ile/registry.253.919993739
2.1.5 删除OCR原位置ASM磁盘组OCR1
一个节点root执行:
[root@jyrac1 ~]# ocrconfig -delete +OCR1
两个节点root执行:
查看OCR和VOTE新状态与位置
[root@jyrac1 ~]# ocrcheck && crsctl query css votedisk
- [root@jyrac1 ~]# ocrconfig -delete +OCR1
- [root@jyrac1 ~]# ocrcheck && crsctl query css votedisk
- Status of Oracle Cluster Registry is as follows :
- Version : 3
- Total space (kbytes) : 262120
- Used space (kbytes) : 3116
- Available space (kbytes) : 259004
- ID : 1718040627
- Device/File Name : +OCR2
- Device/File integrity check succeeded
- Device/File not configured
- Device/File not configured
- Device/File not configured
- Device/File not configured
- Cluster registry integrity check succeeded
- Logical corruption check succeeded
- ## STATE File Universal Id File Name Disk group
- -- ----- ----------------- --------- ---------
- 1. ONLINE 83f3dea9f4c24fccbf05eaa163c5c087 (/dev/asm-diski) [OCR2]
- 2. ONLINE ad21c1f0452e4f4abff906127ff9fcc3 (/dev/asm-diskj) [OCR2]
- 3. ONLINE 2771819c65984f85bf366e74ee85781a (/dev/asm-diskk) [OCR2]
- Located 3 voting disk(s).
- 节点2的ocrcheck输出也没问题,略。
2.1.6 重启整个集群
两个节点root执行:
crsctl stop crs && crsctl start crs
2.1.7 将旧的磁盘组drop掉
检查参数文件:
- SQL> show parameter pfile
- NAME TYPE VALUE
- ------------------------------------ ---------------------- ------------------------------
- spfile string +OCR2/jyrac-scan/asmparameterf
- ile/registry.253.920008067
检查磁盘组状态:
- Select GROUP_NUMBER,NAME,STATE,type,TOTAL_MB,free_mb,VOTING_FILES,COMPATIBILITY from v$asm_diskgroup;
- select GROUP_NUMBER,DISK_NUMBER,STATE,REDUNDANCY,TOTAL_MB,FREE_MB,name,path,failgroup from v$asm_disk order by GROUP_NUMBER;
最后将旧的磁盘组mount并drop掉:
alter diskgroup OCR1 mount;
drop diskgroup OCR1 including contents;
- SQL> alter diskgroup OCR1 mount;
- Diskgroup altered.
- SQL> drop diskgroup OCR1 including contents;
- Diskgroup dropped.
注:旧的磁盘组只允许在一个节点mount,如果发现多个节点mount,需要在其他节点dismount,否则会删除不掉旧的磁盘组。
2.2迁移DATA1
2.2.1 数据磁盘组DATA1添加
确定/dev/asm-diskl,/dev/asm-diskm,/dev/asm-diskn对应的是新加的5G大小的3个盘
用于数据磁盘组DATA1添加
alter diskgroup DATA1 add disk '/dev/asm-diskl' rebalance power 7;
alter diskgroup DATA1 add disk '/dev/asm-diskm', '/dev/asm-diskn' rebalance power 11;
- SQL> alter diskgroup DATA1 add disk '/dev/asm-diskl' rebalance power 7;
- Diskgroup altered.
- SQL> alter diskgroup DATA1 add disk '/dev/asm-diskm', '/dev/asm-diskn' rebalance power 11;
- Diskgroup altered.
这里说明可以单个添加,也可以多个添加。
2.2.2 查询同步状态
- set linesize 200
- col NAME for a10
- select name,allocation_unit_size,state,type,free_mb,required_mirror_free_mb req_mi_fr_mb,usable_file_mb
- from v$asm_diskgroup;
2.2.3 删除原存储的LUN
所有磁盘添加进ASM磁盘组后,删除原存储的LUN。
查询LUN path与asm_disk_name的对应关系
select name,path from v$asm_disk;
- SQL> select name,path from v$asm_disk
- NAME PATH
- ---------- ------------------------------------------------------------
- /dev/asm-diskc
- /dev/asm-diskd
- /dev/asm-disko
- /dev/asm-diskb
- OCR2_0001 /dev/asm-diskj
- OCR2_0000 /dev/asm-diski
- DATA1_0003 /dev/asm-diskl
- DATA1_0004 /dev/asm-diskm
- DATA1_0005 /dev/asm-diskn
- OCR2_0002 /dev/asm-diskk
- FRA1_0000 /dev/asm-diskh
- DATA1_0001 /dev/asm-diskf
- DATA1_0002 /dev/asm-diskg
- DATA1_0000 /dev/asm-diske
- 14 rows selected.
确定是DATA1_0000,DATA1_0001,DATA1_0002
删除磁盘组DATA1中旧存储对应的LUN:
- alter diskgroup DATA1 drop disk DATA1_0000 rebalance power 7;
- alter diskgroup DATA1 drop disk DATA1_0001,DATA1_0002 rebalance power 7;
2.2.4 确认存储置换完成
- select disk_number,path,total_mb,free_mb from v$asm_disk;
- DISK_NUMBER PATH TOTAL_MB FREE_MB
- ----------- ------------------------------------------------------------ ---------- ----------
- 0 /dev/asm-diskc 0 0
- 1 /dev/asm-diskd 0 0
- 2 /dev/asm-disko 0 0
- 3 /dev/asm-diske 0 0
- 4 /dev/asm-diskf 0 0
- 5 /dev/asm-diskg 0 0
- 6 /dev/asm-diskb 0 0
- 1 /dev/asm-diskj 1024 715
- 0 /dev/asm-diski 1024 717
- 3 /dev/asm-diskl 5120 4502
- 4 /dev/asm-diskm 5120 4499
- 5 /dev/asm-diskn 5120 4501
- 2 /dev/asm-diskk 1024 714
- 0 /dev/asm-diskh 5120 4790
- 14 rows selected.
2.3迁移FRA1
FRA1磁盘组和DATA1磁盘组的添加一样,
- alter diskgroup FRA1 add disk '/dev/asm-disko' rebalance power 7;
- alter diskgroup FRA1 drop disk FRA1_0000 rebalance power 7;
- select disk_number,path,total_mb,free_mb from v$asm_disk;
最后确定所有盘都置换完成:
- SQL> select disk_number,path,total_mb,free_mb from v$asm_disk;
- DISK_NUMBER PATH TOTAL_MB FREE_MB
- ----------- ------------------------------------------------------------ ---------- ----------
- 0 /dev/asm-diskc 0 0
- 1 /dev/asm-diskd 0 0
- 3 /dev/asm-diske 0 0
- 4 /dev/asm-diskf 0 0
- 5 /dev/asm-diskg 0 0
- 6 /dev/asm-diskb 0 0
- 1 /dev/asm-diskj 1024 715
- 0 /dev/asm-diski 1024 717
- 1 /dev/asm-disko 5120 4943
- 3 /dev/asm-diskl 5120 4502
- 4 /dev/asm-diskm 5120 4499
- 5 /dev/asm-diskn 5120 4501
- 2 /dev/asm-diskk 1024 714
- 0 /dev/asm-diskh 5120 4965
- 14 rows selected.
- SQL> r
- 1* select disk_number,path,total_mb,free_mb from v$asm_disk
- DISK_NUMBER PATH TOTAL_MB FREE_MB
- ----------- ------------------------------------------------------------ ---------- ----------
- 0 /dev/asm-diskc 0 0
- 1 /dev/asm-diskd 0 0
- 2 /dev/asm-diskh 0 0
- 3 /dev/asm-diske 0 0
- 4 /dev/asm-diskf 0 0
- 5 /dev/asm-diskg 0 0
- 6 /dev/asm-diskb 0 0
- 1 /dev/asm-diskj 1024 715
- 0 /dev/asm-diski 1024 717
- 1 /dev/asm-disko 5120 4790
- 3 /dev/asm-diskl 5120 4502
- 4 /dev/asm-diskm 5120 4499
- 5 /dev/asm-diskn 5120 4501
- 2 /dev/asm-diskk 1024 714
- 14 rows selected.
最终确定要替换的LUN都没有数据了,主机侧就可以操作了。
2.4测试
查看集群各资源状态:
crsctl stat res -t
查看OCR信息:
ocrcheck && crsctl query css votedisk
查询ASM实例的spfile文件:
- sqlplus / as sysasm <<EOF
- show parameter pfile
- EOF
查询之前的业务数据:
- -- 业务查询SQL 1
- select * from t1, t2 where t1.id = t2.t1_id and t1.n = 19;
- -- 业务查询SQL 2
- select * from t1, t2 where t1.id = t2.t1_id;
至此,整个RAC环境更换存储的实验就全部完成了。
标签:RAC,MB,dev,11grac,asm,File,Oracle,disk,OCR2 来源: https://www.cnblogs.com/yaoyangding/p/15484442.html
本站声明: 1. iCode9 技术分享网(下文简称本站)提供的所有内容,仅供技术学习、探讨和分享; 2. 关于本站的所有留言、评论、转载及引用,纯属内容发起人的个人观点,与本站观点和立场无关; 3. 关于本站的所有言论和文字,纯属内容发起人的个人观点,与本站观点和立场无关; 4. 本站文章均是网友提供,不完全保证技术分享内容的完整性、准确性、时效性、风险性和版权归属;如您发现该文章侵犯了您的权益,可联系我们第一时间进行删除; 5. 本站为非盈利性的个人网站,所有内容不会用来进行牟利,也不会利用任何形式的广告来间接获益,纯粹是为了广大技术爱好者提供技术内容和技术思想的分享性交流网站。