Perder o OCR no Oracle RAC 19c é daquelas situações que ninguém quer encarar. Aqui, no entanto, provoquei esse cenário: danifiquei o disco onde o OCR estava armazenado, simulando uma falha para ilustrar um modo direto e prático de resolver. O OCR é como o coração do cluster, guardando todas as informações essenciais para o sistema operar bem.
Antes de explicar a solução, fica o aviso: vou quebrar e demonstrar, mas isso é algo que você NÃO deve fazer!
Passo 0: Corromper o Disco do OCR
O Que Você NÃO Deve Fazer
[grid@srv1 ~]$ crsctl stat res -t
--------------------------------------------------------------------------------
Name Target State Server State details
--------------------------------------------------------------------------------
Local Resources
--------------------------------------------------------------------------------
ora.LISTENER.lsnr
ONLINE ONLINE srv1 STABLE
ONLINE ONLINE srv2 STABLE
ora.chad
ONLINE ONLINE srv1 STABLE
ONLINE ONLINE srv2 STABLE
ora.net1.network
ONLINE ONLINE srv1 STABLE
ONLINE ONLINE srv2 STABLE
ora.ons
ONLINE ONLINE srv1 STABLE
ONLINE ONLINE srv2 STABLE
--------------------------------------------------------------------------------
Cluster Resources
--------------------------------------------------------------------------------
ora.ASMNET1LSNR_ASM.lsnr(ora.asmgroup)
1 ONLINE ONLINE srv1 STABLE
2 ONLINE ONLINE srv2 STABLE
3 ONLINE OFFLINE STABLE
ora.CONFIG.dg(ora.asmgroup)
1 ONLINE ONLINE srv1 STABLE
2 ONLINE ONLINE srv2 STABLE
3 OFFLINE OFFLINE STABLE
ora.DATA.dg(ora.asmgroup)
1 ONLINE ONLINE srv1 STABLE
2 ONLINE ONLINE srv2 STABLE
3 OFFLINE OFFLINE STABLE
ora.FRA.dg(ora.asmgroup)
1 ONLINE ONLINE srv1 STABLE
2 ONLINE ONLINE srv2 STABLE
3 OFFLINE OFFLINE STABLE
ora.LISTENER_SCAN1.lsnr
1 ONLINE ONLINE srv2 STABLE
ora.LISTENER_SCAN2.lsnr
1 ONLINE ONLINE srv1 STABLE
ora.LISTENER_SCAN3.lsnr
1 ONLINE ONLINE srv1 STABLE
ora.asm(ora.asmgroup)
1 ONLINE ONLINE srv1 Started,STABLE
2 ONLINE ONLINE srv2 Started,STABLE
3 OFFLINE OFFLINE STABLE
ora.asmnet1.asmnetwork(ora.asmgroup)
1 ONLINE ONLINE srv1 STABLE
2 ONLINE ONLINE srv2 STABLE
3 OFFLINE OFFLINE STABLE
ora.auto.db
1 OFFLINE OFFLINE Instance Shutdown,ST
ABLE
2 OFFLINE OFFLINE Instance Shutdown,ST
ABLE
ora.cvu
1 ONLINE ONLINE srv1 STABLE
ora.qosmserver
1 ONLINE ONLINE srv1 STABLE
ora.scan1.vip
1 ONLINE ONLINE srv2 STABLE
ora.scan2.vip
1 ONLINE ONLINE srv1 STABLE
ora.scan3.vip
1 ONLINE ONLINE srv1 STABLE
ora.srvora1.vip
1 ONLINE ONLINE srv1 STABLE
ora.srvora2.vip
1 ONLINE ONLINE srv2 STABLE
--------------------------------------------------------------------------------
[grid@srv1 ~]$
[grid@srv1 ~]$ ocrcheck
Status of Oracle Cluster Registry is as follows :
Version : 4
Total space (kbytes) : 491684
Used space (kbytes) : 84284
Available space (kbytes) : 407400
ID : 982382141
Device/File Name : +CONFIG
Device/File integrity check succeeded
Device/File not configured
Device/File not configured
Device/File not configured
Device/File not configured
Cluster registry integrity check succeeded
Logical corruption check bypassed due to non-privileged user
[grid@srv1 ~]$ sqlplus / as sysasm
SQL*Plus: Release 19.0.0.0.0 - Production on Mon Oct 21 09:39:36 2024
Version 19.3.0.0.0
Copyright (c) 1982, 2019, Oracle. All rights reserved.
Connected to:
Oracle Database 19c Enterprise Edition Release 19.0.0.0.0 - Production
Version 19.3.0.0.0
SQL> @ams_disks.sql
SQL>
SQL> SET LINESIZE 150;
SQL> SET PAGESIZE 50;
SQL>
SQL> COLUMN name FORMAT A20;
SQL> COLUMN path FORMAT A50;
SQL> COLUMN state FORMAT A10;
SQL> COLUMN total_mb FORMAT 9999999;
SQL> COLUMN free_mb FORMAT 9999999;
SQL>
SQL> SELECT name, path, state, total_mb, free_mb
2 FROM v$asm_disk
3 WHERE group_number = (SELECT group_number FROM v$asm_diskgroup WHERE name = 'CONFIG');
NAME PATH STATE TOTAL_MB FREE_MB
-------------------- ----------------------------------------------- ---------- -------- --------
CONFIG_0000 /dev/oracleasm/disks/DISK1 NORMAL 10236 9876
[grid@srv1 ~]$ exit
logout
[root@srvora1 ~]# ls -l /dev | grep " $(oracleasm querydisk -d DISK1| grep -o '[0-9]\+,[0-9]\+' | cut -d',' -f1)," | grep " $(oracleasm querydisk -d DISK1 | grep -o '[0-9]\+,[0-9]\+' | cut -d',' -f2)"
rw-rw----. 1 root disk 8, 17 Oct 21 09:28 sdb1
[root@srv1 ~]#
[root@srv1 ~]#
[root@srv1 ~]# fdisk -l /dev/sdb
Disk /dev/sdb: 10.7 GB, 10737418240 bytes, 20971520 sectors
Units = sectors of 1 * 512 = 512 bytes
Sector size (logical/physical): 512 bytes / 512 bytes
I/O size (minimum/optimal): 512 bytes / 512 bytes
Disk label type: dos
Disk identifier: 0x5d62374c
Device Boot Start End Blocks Id System
/dev/sdb1 2048 20971519 10484736 83 Linux
[root@srv1 ~]#
[root@srv1 ~]# dd if=/dev/urandom of=/dev/sdb1 bs=1M count=10000 status=progress
6047137792 bytes (6.0 GB) copied, 52.376175 s, 115 MB/s
7823425536 bytes (7.8 GB) copied, 76.612367 s, 102 MB/s
10463739904 bytes (10 GB) copied, 110.908806 s, 94.3 MB/s
10000+0 records in
10000+0 records out
10485760000 bytes (10 GB) copied, 140.469 s, 74.6 MB/s
[root@srv1 ~]#
[root@srv1 ~]# fsck /dev/sdb1
fsck from util-linux 2.23.2
e2fsck 1.45.4 (23-Sep-2019)
ext2fs_open2: Bad magic number in super-block
fsck.ext2: Superblock invalid, trying backup blocks...
fsck.ext2: Bad magic number in super-block while trying to open /dev/sdb1
The superblock could not be read or does not describe a valid ext2/ext3/ext4
filesystem. If the device is valid and it really contains an ext2/ext3/ext4
filesystem (and not swap or ufs or something else), then the superblock
is corrupt, and you might try running e2fsck with an alternate superblock:
e2fsck -b 8193 <device>
or
e2fsck -b 32768 <device>
[root@srv1 ~]#
[root@srv1 ~]#
[root@srv1 ~]# Antes de realizar a formatação, cuidei para garantir que, nesse disco, estivesse apenas o OCR.
Após a formatação, o Oracle Clusterware pode continuar funcionando por um tempo, pois o OCR está em cache. No entanto, na próxima vez que o cluster tentar acessar o OCR diretamente (como em um restart), ele vai falhar.
Parar Tudo e Tentar Subir
Depois de formatar o disco e forçar o problema, vamos parar intencionalmente todos os serviços e tentar reiniciar o cluster, para ver o impacto da perda do OCR.
[root@srv1 ~]# crsctl stop cluster -all
CRS-2673: Attempting to stop 'ora.crsd' on 'srv1'
CRS-2790: Starting shutdown of Cluster Ready Services-managed resources on server 'srv1'
CRS-2673: Attempting to stop 'ora.ibge.db' on 'srv1'
CRS-2673: Attempting to stop 'ora.chad' on 'srv1'
CRS-2673: Attempting to stop 'ora.crsd' on 'srv2'
CRS-2790: Starting shutdown of Cluster Ready Services-managed resources on server 'srv2'
CRS-2673: Attempting to stop 'ora.qosmserver' on 'srv2'
CRS-2673: Attempting to stop 'ora.chad' on 'srv2'
CRS-2673: Attempting to stop 'ora.ibge.db' on 'srv2'
CRS-2677: Stop of 'ora.ibge.db' on 'srv1' succeeded
CRS-33673: Attempting to stop resource group 'ora.asmgroup' on server 'srv1'
CRS-2673: Attempting to stop 'ora.CONFIG.dg' on 'srv1'
CRS-2673: Attempting to stop 'ora.DATA.dg' on 'srv1'
CRS-2673: Attempting to stop 'ora.FRA.dg' on 'srv1'
CRS-2673: Attempting to stop 'ora.LISTENER.lsnr' on 'srv1'
CRS-2673: Attempting to stop 'ora.LISTENER_SCAN1.lsnr' on 'srv1'
CRS-2677: Stop of 'ora.CONFIG.dg' on 'srv1' succeeded
CRS-2677: Stop of 'ora.FRA.dg' on 'srv1' succeeded
CRS-2677: Stop of 'ora.DATA.dg' on 'srv1' succeeded
CRS-2673: Attempting to stop 'ora.asm' on 'srv1'
CRS-2677: Stop of 'ora.LISTENER.lsnr' on 'srv1' succeeded
CRS-2673: Attempting to stop 'ora.srv1.vip' on 'srv1'
CRS-2677: Stop of 'ora.LISTENER_SCAN1.lsnr' on 'srv1' succeeded
CRS-2673: Attempting to stop 'ora.scan1.vip' on 'srv1'
CRS-2677: Stop of 'ora.srv1.vip' on 'srv1' succeeded
CRS-2677: Stop of 'ora.scan1.vip' on 'srv1' succeeded
CRS-2677: Stop of 'ora.asm' on 'srv1' succeeded
CRS-2673: Attempting to stop 'ora.ASMNET1LSNR_ASM.lsnr' on 'srv1'
CRS-2677: Stop of 'ora.ibge.db' on 'srv2' succeeded
CRS-2673: Attempting to stop 'ora.LISTENER.lsnr' on 'srv2'
CRS-2673: Attempting to stop 'ora.LISTENER_SCAN2.lsnr' on 'srv2'
CRS-2673: Attempting to stop 'ora.LISTENER_SCAN3.lsnr' on 'srv2'
CRS-2677: Stop of 'ora.chad' on 'srv1' succeeded
CRS-2673: Attempting to stop 'ora.cvu' on 'srv2'
CRS-2677: Stop of 'ora.LISTENER.lsnr' on 'srv2' succeeded
CRS-2677: Stop of 'ora.LISTENER_SCAN2.lsnr' on 'srv2' succeeded
CRS-2673: Attempting to stop 'ora.scan2.vip' on 'srv2'
CRS-2673: Attempting to stop 'ora.srv2.vip' on 'srv2'
CRS-2677: Stop of 'ora.LISTENER_SCAN3.lsnr' on 'srv2' succeeded
CRS-2673: Attempting to stop 'ora.scan3.vip' on 'srv2'
CRS-2677: Stop of 'ora.scan2.vip' on 'srv2' succeeded
CRS-2677: Stop of 'ora.scan3.vip' on 'srv2' succeeded
CRS-2677: Stop of 'ora.srv2.vip' on 'srv2' succeeded
CRS-2677: Stop of 'ora.ASMNET1LSNR_ASM.lsnr' on 'srv1' succeeded
CRS-2673: Attempting to stop 'ora.asmnet1.asmnetwork' on 'srv1'
CRS-2677: Stop of 'ora.asmnet1.asmnetwork' on 'srv1' succeeded
CRS-2677: Stop of 'ora.chad' on 'srv2' succeeded
CRS-33677: Stop of resource group 'ora.asmgroup' on server 'srv1' succeeded.
CRS-33673: Attempting to stop resource group 'ora.asmgroup' on server 'srv2'
CRS-2673: Attempting to stop 'ora.CONFIG.dg' on 'srv2'
CRS-2673: Attempting to stop 'ora.DATA.dg' on 'srv2'
CRS-2673: Attempting to stop 'ora.FRA.dg' on 'srv2'
CRS-2673: Attempting to stop 'ora.ons' on 'srv1'
CRS-2677: Stop of 'ora.DATA.dg' on 'srv2' succeeded
CRS-2677: Stop of 'ora.CONFIG.dg' on 'srv2' succeeded
CRS-2677: Stop of 'ora.FRA.dg' on 'srv2' succeeded
CRS-2673: Attempting to stop 'ora.asm' on 'srv2'
CRS-4549: Unexpected disconnect while executing shutdown request.
CRS-4549: Unexpected disconnect while executing shutdown request.
Cluster registry server error. Server error stack -
PROC-00026: Error while accessing the physical storage
- end of server error stackCluster registry server error. Server error stack -
PROC-00026: Error while accessing the physical storage
- end of server error stackCluster Ready Service aborted due to Oracle Cluster Registry error [PROC-26: Error while accessing the physical storage Storage layer error [Insufficient quorum to open OCR devices] [0]]. Details at (:CRSD00111:) in /u01/app/grid/diag/crs/srv2/crs/trace/crsd.trc.
Cluster Ready Service aborted due to Oracle Cluster Registry error [PROC-26: Error while accessing the physical storage Storage layer error [Insufficient quorum to open OCR devices] [0]]. Details at (:CRSD00111:) in /u01/app/grid/diag/crs/srv1/crs/trace/crsd.trc.
##PODEMOS OBSERVAR ERRO AO TENTAR BAIXAR TUDÃO##
##COMO BAIXOU COM ERRO EU DECIDI REININCIAR OS SERVIDORES##
[root@srv1 ~]# crsctl stat res -t -init
--------------------------------------------------------------------------------
Name Target State Server State details
--------------------------------------------------------------------------------
Cluster Resources
--------------------------------------------------------------------------------
ora.asm
1 ONLINE ONLINE srv1 STABLE
ora.cluster_interconnect.haip
1 ONLINE ONLINE srv1 STABLE
ora.crf
1 ONLINE ONLINE srv1 STABLE
ora.crsd
1 OFFLINE ONLINE srv1 STOPPING
ora.cssd
1 ONLINE ONLINE srv1 STABLE
ora.cssdmonitor
1 ONLINE ONLINE srv1 STABLE
ora.ctssd
1 ONLINE ONLINE srv1 OBSERVER,STABLE
ora.diskmon
1 OFFLINE OFFLINE STABLE
ora.evmd
1 ONLINE ONLINE srv1 STABLE
ora.gipcd
1 ONLINE ONLINE srv1 STABLE
ora.gpnpd
1 ONLINE ONLINE srv1 STABLE
ora.mdnsd
1 ONLINE ONLINE srv1 STABLE
ora.storage
1 ONLINE OFFLINE srv1 STARTING
--------------------------------------------------------------------------------
[root@srv1 ~]# reboot
Connection to 192.168.56.71 closed by remote host.
Connection to 192.168.56.71 closed.
[.root@salto ~]$
[root@srv2~]# reboot
Connection to 192.168.56.71 closed by remote host.
Connection to 192.168.56.71 closed.
[.root@salto ~]$
Mas no final do comando, começaram a aparecer diversos erros. Diante disso, decidi dar um reboot em ambos os nodes. Após o reboot, o próximo passo foi verificar a subida do cluster.
[root@srv1 ~]# crsctl stat res -t
CRS-4535: Cannot communicate with Cluster Ready Services
CRS-4000: Command Status failed, or completed with errors.
[root@srv1 ~]# crsctl stat res -t -init
--------------------------------------------------------------------------------
Name Target State Server State details
--------------------------------------------------------------------------------
Cluster Resources
--------------------------------------------------------------------------------
ora.asm
1 ONLINE ONLINE srv1 Started,STABLE
ora.cluster_interconnect.haip
1 ONLINE ONLINE srv1 STABLE
ora.crf
1 ONLINE ONLINE srv1 STABLE
ora.crsd
1 ONLINE OFFLINE STABLE
ora.cssd
1 ONLINE ONLINE srv1 STABLE
ora.cssdmonitor
1 ONLINE ONLINE srv1 STABLE
ora.ctssd
1 ONLINE ONLINE srv1 OBSERVER,STABLE
ora.diskmon
1 OFFLINE OFFLINE STABLE
ora.evmd
1 ONLINE ONLINE srv1 STABLE
ora.gipcd
1 ONLINE ONLINE srv1 STABLE
ora.gpnpd
1 ONLINE ONLINE srv1 STABLE
ora.mdnsd
1 ONLINE ONLINE srv1 STABLE
ora.storage
1 ONLINE OFFLINE STABLE
--------------------------------------------------------------------------------
[root@srv1 ~]# tail -n 5 /u01/app/grid/diag/crs/srv1/crs/trace/alert.log
2024-10-10 22:15:03.461 [ORAROOTAGENT(2581)]CRS-5818: Aborted command 'start' for resource 'ora.storage'. Details at (:CRSAGF00113:) {0:5:3} in /u01/app/grid/diag/crs/srv1/crs/trace/ohasd_orarootagent_root.trc.
2024-10-10 22:15:03.935 [OHASD(2459)]CRS-2757: Command 'Start' timed out waiting for response from the resource 'ora.storage'. Details at (:CRSPE00221:) {0:5:3} in /u01/app/grid/diag/crs/srv1/crs/trace/ohasd.trc.
2024-10-10 22:15:03.926 [ORAROOTAGENT(2581)]CRS-5017: The resource action "ora.storage start" encountered the following error:
2024-10-10 22:15:03.926+Storage::start action aborted. For details refer to "(:CLSN00107:)" in "/u01/app/grid/diag/crs/srv1/crs/trace/ohasd_orarootagent_root.trc".
2024-10-10 22:16:07.580 [ORAROOTAGENT(2581)]CRS-5019: All OCR locations are on ASM disk groups [CONFIG], and none of these disk groups are mounted. Details are at "(:CLSN00140:)" in "/u01/app/grid/diag/crs/srv1/crs/trace/ohasd_orarootagent_root.trc".
[root@srv1 ~]#
Verificar o log, ficou claro o seguinte erro:
Ou seja, o Disk Group CONFIG não estava montado, o que resultou na falha de acesso ao OCR.
Agora o problema tá criado. Hora de resolver.
Recriar o DiskGroup CONFIG
Agora, vamos recriar o DiskGroup CONFIG. Para isso, você precisa dos discos físicos que vão compor o DiskGroup.
[root@srv1 ~]# oracleasm listdisks
DISK2
DISK3
[root@srv1 ~]# oracleasm scandisks
Reloading disk partitions: done
Cleaning any stale ASM disks...
Scanning system for ASM disks...
[root@srv1 ~]#
[root@srv1 ~]#
[root@srv1 ~]# fdisk /dev/sdb
Welcome to fdisk (util-linux 2.23.2).
Changes will remain in memory only, until you decide to write them.
Be careful before using the write command.
Command (m for help): d
Selected partition 1
Partition 1 is deleted
Command (m for help): w
The partition table has been altered!
Calling ioctl() to re-read partition table.
Syncing disks.
[root@srv1 ~]# fdisk /dev/sdb
Welcome to fdisk (util-linux 2.23.2).
Changes will remain in memory only, until you decide to write them.
Be careful before using the write command.
Command (m for help): n
Partition type:
p primary (0 primary, 0 extended, 4 free)
e extended
Select (default p):
Using default response p
Partition number (1-4, default 1):
First sector (2048-20971519, default 2048):
Using default value 2048
Last sector, +sectors or +size{K,M,G} (2048-20971519, default 20971519):
Using default value 20971519
Partition 1 of type Linux and of size 10 GiB is set
Command (m for help): w
The partition table has been altered!
Calling ioctl() to re-read partition table.
Syncing disks.
[root@srv1 ~]# oracleasm createdisk DISK1 /dev/sdb1
Writing disk header: done
Instantiating disk: done
[root@srv1 ~]# oracleasm scandisks
Reloading disk partitions: done
Cleaning any stale ASM disks...
Scanning system for ASM disks...
[root@srv1 ~]# oracleasm listdisks
DISK1
DISK2
DISK3
[root@srv1 ~]# su - grid
Last login: Thu Oct 10 14:59:41 -03 2024
[grid@srv1 ~]$ sqlplus / as sysasm
SQL*Plus: Release 19.0.0.0.0 - Production on Thu Oct 10 15:55:01 2024
Version 19.3.0.0.0
Copyright (c) 1982, 2019, Oracle. All rights reserved.
Connected to:
Oracle Database 19c Enterprise Edition Release 19.0.0.0.0 - Production
Version 19.3.0.0.0
SQL> CREATE DISKGROUP CONFIG EXTERNAL REDUNDANCY DISK '/dev/oracleasm/disks/DISK1';
Diskgroup created.
SQL>
##FAZER scandisks NO NODE2
[root@srv2 ~]# oracleasm scandisks
Reloading disk partitions: done
Cleaning any stale ASM disks...
Scanning system for ASM disks...
Instantiating disk "DISK1"
[root@srv2 ~]# oracleasm listdisks
DISK1
DISK2
DISK3
[root@srv2 ~]#Passo 1: Localizar os Backups do OCR
[root@srv1 ~]# ocrconfig -showbackup
PROT-26: Oracle Cluster Registry backup locations were retrieved from a local copy
srv2 2024/10/06 23:33:48 +CONFIG:/rac/OCRBACKUP/backup00.ocr.263.1181691181 724960844
srv2 2024/10/06 16:53:12 +CONFIG:/rac/OCRBACKUP/backup01.ocr.262.1181667159 724960844
srv2 2024/10/06 12:52:37 +CONFIG:/rac/OCRBACKUP/backup02.ocr.264.1181652725 724960844
srv1 2024/10/05 08:53:00 +CONFIG:/rac/OCRBACKUP/day.ocr.259.1181551981 724960844
srv1 2024/10/05 08:53:00 +CONFIG:/rac/OCRBACKUP/week.ocr.260.1181551983 724960844
PROT-25: Manual backups for the Oracle Cluster Registry are not available
[root@srv1 ~]#
[root@srv1 ~]# find $ORACLE_BASE -name "*.ocr"
/u01/app/grid/crsdata/srv1/ocr/day.ocr
/u01/app/grid/crsdata/srv1/ocr/backup00.ocr
[root@srv1 ~]# ls -ltra /u01/app/grid/crsdata/srv1/ocr/
total 4444
drwxrwxr-x. 25 grid oinstall 4096 Jul 13 2020 ..
-rw-r--r--. 1 root root 2273280 Oct 5 01:53 day.ocr
drwxr-xr-x. 2 grid oinstall 41 Oct 5 09:53 .
-rw-r--r--. 1 root root 2273280 Oct 5 09:53 backup00.ocr
[root@srv1 ~]#Os backups do OCR estavam armazenados no diskgroup que foi destruído. Todos os backups automáticos estavam no DG CONFIG, que eu corrompi, então não tem como restaurar a partir deles. No entanto, por padrão, também existe um backup do OCR que é realizado no filesystem do sistema operacional, vou tentar restaurar por ele.
Passo 2: Parar o CRS e Iniciar em Modo Exclusivo Sem Cluster
##NODE02
[root@srv2 ~]# crsctl stop crs -f
CRS-2791: Starting shutdown of Oracle High Availability Services-managed resources on 'srv2'
CRS-2673: Attempting to stop 'ora.asm' on 'srv2'
CRS-2673: Attempting to stop 'ora.mdnsd' on 'srv2'
CRS-2677: Stop of 'ora.asm' on 'srv2' succeeded
CRS-2673: Attempting to stop 'ora.cluster_interconnect.haip' on 'srv2'
CRS-2677: Stop of 'ora.mdnsd' on 'srv2' succeeded
CRS-2677: Stop of 'ora.cluster_interconnect.haip' on 'srv2' succeeded
CRS-2673: Attempting to stop 'ora.ctssd' on 'srv2'
CRS-2673: Attempting to stop 'ora.evmd' on 'srv2'
CRS-2677: Stop of 'ora.ctssd' on 'srv2' succeeded
CRS-2677: Stop of 'ora.evmd' on 'srv2' succeeded
CRS-2673: Attempting to stop 'ora.cssd' on 'srv2'
CRS-2677: Stop of 'ora.cssd' on 'srv2' succeeded
CRS-2673: Attempting to stop 'ora.crf' on 'srv2'
CRS-2673: Attempting to stop 'ora.gpnpd' on 'srv2'
CRS-2677: Stop of 'ora.gpnpd' on 'srv2' succeeded
CRS-2677: Stop of 'ora.crf' on 'srv2' succeeded
CRS-2673: Attempting to stop 'ora.gipcd' on 'srv2'
CRS-2677: Stop of 'ora.gipcd' on 'srv2' succeeded
CRS-2793: Shutdown of Oracle High Availability Services-managed resources on 'srv2' has completed
CRS-4133: Oracle High Availability Services has been stopped.
[root@srv2 ~]#
##NODE 01
[root@srv1 ~]# crsctl stop crs -f
CRS-2791: Starting shutdown of Oracle High Availability Services-managed resources on 'srv1'
CRS-2673: Attempting to stop 'ora.ctssd' on 'srv1'
CRS-2673: Attempting to stop 'ora.evmd' on 'srv1'
CRS-2673: Attempting to stop 'ora.asm' on 'srv1'
CRS-2673: Attempting to stop 'ora.mdnsd' on 'srv1'
CRS-2677: Stop of 'ora.ctssd' on 'srv1' succeeded
CRS-2677: Stop of 'ora.evmd' on 'srv1' succeeded
CRS-2677: Stop of 'ora.mdnsd' on 'srv1' succeeded
CRS-2677: Stop of 'ora.asm' on 'srv1' succeeded
CRS-2673: Attempting to stop 'ora.cluster_interconnect.haip' on 'srv1'
CRS-2677: Stop of 'ora.cluster_interconnect.haip' on 'srv1' succeeded
CRS-2673: Attempting to stop 'ora.cssd' on 'srv1'
CRS-2677: Stop of 'ora.cssd' on 'srv1' succeeded
CRS-2673: Attempting to stop 'ora.gpnpd' on 'srv1'
CRS-2673: Attempting to stop 'ora.crf' on 'srv1'
CRS-2677: Stop of 'ora.gpnpd' on 'srv1' succeeded
CRS-2677: Stop of 'ora.crf' on 'srv1' succeeded
CRS-2673: Attempting to stop 'ora.gipcd' on 'srv1'
CRS-2677: Stop of 'ora.gipcd' on 'srv1' succeeded
CRS-2793: Shutdown of Oracle High Availability Services-managed resources on 'srv1' has completed
CRS-4133: Oracle High Availability Services has been stopped.
[root@srv1 ~]#
[root@srv1 ~]#
[root@srv1 ~]# crsctl start crs -excl -nocrs
CRS-4123: Oracle High Availability Services has been started.
CRS-2672: Attempting to start 'ora.evmd' on 'srv1'
CRS-2672: Attempting to start 'ora.mdnsd' on 'srv1'
CRS-2676: Start of 'ora.mdnsd' on 'srv1' succeeded
CRS-2676: Start of 'ora.evmd' on 'srv1' succeeded
CRS-2672: Attempting to start 'ora.gpnpd' on 'srv1'
CRS-2676: Start of 'ora.gpnpd' on 'srv1' succeeded
CRS-2672: Attempting to start 'ora.cssdmonitor' on 'srv1'
CRS-2672: Attempting to start 'ora.gipcd' on 'srv1'
CRS-2676: Start of 'ora.cssdmonitor' on 'srv1' succeeded
CRS-2676: Start of 'ora.gipcd' on 'srv1' succeeded
CRS-2672: Attempting to start 'ora.cssd' on 'srv1'
CRS-2672: Attempting to start 'ora.diskmon' on 'srv1'
CRS-2676: Start of 'ora.diskmon' on 'srv1' succeeded
CRS-2676: Start of 'ora.cssd' on 'srv1' succeeded
CRS-2672: Attempting to start 'ora.cluster_interconnect.haip' on 'srv1'
CRS-2672: Attempting to start 'ora.ctssd' on 'srv1'
CRS-2676: Start of 'ora.ctssd' on 'srv1' succeeded
CRS-2676: Start of 'ora.cluster_interconnect.haip' on 'srv1' succeeded
CRS-2672: Attempting to start 'ora.asm' on 'srv1'
CRS-2676: Start of 'ora.asm' on 'srv1' succeeded
[root@srv1 ~]#Passo 3: Restaurar o OCR a Partir do Backup no Filesystem
[root@srv1 ~]# asmcmd lsdg
State Type Rebal Sector Logical_Sector Block AU Total_MB Free_MB Req_mir_free_MB Usable_file_MB Offline_disks Voting_files Name
MOUNTED EXTERN N 512 512 4096 1048576 10239 10187 0 10187 0 N CONFIG/
MOUNTED EXTERN N 512 512 4096 4194304 15356 10668 0 10668 0 N DATA/
MOUNTED EXTERN N 512 512 4096 4194304 15356 14332 0 14332 0 Y FRA/
[root@srv1 ~]#
[root@srv1 ~]# find $ORACLE_BASE -name "*.ocr"
/u01/app/grid/crsdata/srv1/ocr/day.ocr
/u01/app/grid/crsdata/srv1/ocr/backup00.ocr
[root@srv1 ~]#
[root@srv1 ~]# ocrconfig -restore /u01/app/grid/crsdata/srv1/ocr/backup00.ocr
[root@srv1 ~]#
[root@srv1 ~]# ocrcheck
Status of Oracle Cluster Registry is as follows :
Version : 4
Total space (kbytes) : 491684
Used space (kbytes) : 84088
Available space (kbytes) : 407596
ID : 1774280553
Device/File Name : +CONFIG
Device/File integrity check succeeded
Device/File not configured
Device/File not configured
Device/File not configured
Device/File not configured
Cluster registry integrity check succeeded
Logical corruption check succeeded
[root@srv1 ~]#Deu certo a restauração, agora vamos baixar tudo e subir normalmente
Com o DiskGroup CONFIG e o OCR restaurados, está tudo pronto para voltarmos ao funcionamento normal. Vamos parar o cluster e reiniciá-lo de por completo.
Passo 4: Reiniciar o Cluster
##NODE01##
[root@srv1 ~]# crsctl stop crs
CRS-2791: Starting shutdown of Oracle High Availability Services-managed resources on 'srv1'
CRS-2673: Attempting to stop 'ora.ctssd' on 'srv1'
CRS-2673: Attempting to stop 'ora.evmd' on 'srv1'
CRS-2673: Attempting to stop 'ora.asm' on 'srv1'
CRS-2673: Attempting to stop 'ora.mdnsd' on 'srv1'
CRS-2677: Stop of 'ora.evmd' on 'srv1' succeeded
CRS-2677: Stop of 'ora.ctssd' on 'srv1' succeeded
CRS-2677: Stop of 'ora.mdnsd' on 'srv1' succeeded
CRS-2677: Stop of 'ora.asm' on 'srv1' succeeded
CRS-2673: Attempting to stop 'ora.cluster_interconnect.haip' on 'srv1'
CRS-2677: Stop of 'ora.cluster_interconnect.haip' on 'srv1' succeeded
CRS-2673: Attempting to stop 'ora.cssd' on 'srv1'
CRS-2677: Stop of 'ora.cssd' on 'srv1' succeeded
CRS-2673: Attempting to stop 'ora.gipcd' on 'srv1'
CRS-2673: Attempting to stop 'ora.gpnpd' on 'srv1'
CRS-2677: Stop of 'ora.gpnpd' on 'srv1' succeeded
CRS-2677: Stop of 'ora.gipcd' on 'srv1' succeeded
CRS-2793: Shutdown of Oracle High Availability Services-managed resources on 'srv1' has completed
CRS-4133: Oracle High Availability Services has been stopped.
[root@srv1 ~]# crsctl start crs -wait
CRS-4123: Starting Oracle High Availability Services-managed resources
CRS-2672: Attempting to start 'ora.mdnsd' on 'srv1'
CRS-2672: Attempting to start 'ora.evmd' on 'srv1'
CRS-2676: Start of 'ora.mdnsd' on 'srv1' succeeded
CRS-2676: Start of 'ora.evmd' on 'srv1' succeeded
CRS-2672: Attempting to start 'ora.gpnpd' on 'srv1'
CRS-2676: Start of 'ora.gpnpd' on 'srv1' succeeded
CRS-2672: Attempting to start 'ora.gipcd' on 'srv1'
CRS-2676: Start of 'ora.gipcd' on 'srv1' succeeded
CRS-2672: Attempting to start 'ora.crf' on 'srv1'
CRS-2672: Attempting to start 'ora.cssdmonitor' on 'srv1'
CRS-2676: Start of 'ora.cssdmonitor' on 'srv1' succeeded
CRS-2672: Attempting to start 'ora.cssd' on 'srv1'
CRS-2672: Attempting to start 'ora.diskmon' on 'srv1'
CRS-2676: Start of 'ora.diskmon' on 'srv1' succeeded
CRS-2676: Start of 'ora.crf' on 'srv1' succeeded
CRS-2676: Start of 'ora.cssd' on 'srv1' succeeded
CRS-2672: Attempting to start 'ora.cluster_interconnect.haip' on 'srv1'
CRS-2672: Attempting to start 'ora.ctssd' on 'srv1'
CRS-2676: Start of 'ora.ctssd' on 'srv1' succeeded
CRS-2676: Start of 'ora.cluster_interconnect.haip' on 'srv1' succeeded
CRS-2672: Attempting to start 'ora.asm' on 'srv1'
CRS-2676: Start of 'ora.asm' on 'srv1' succeeded
CRS-2672: Attempting to start 'ora.storage' on 'srv1'
CRS-2676: Start of 'ora.storage' on 'srv1' succeeded
CRS-2672: Attempting to start 'ora.crsd' on 'srv1'
CRS-2676: Start of 'ora.crsd' on 'srv1' succeeded
CRS-6023: Starting Oracle Cluster Ready Services-managed resources
CRS-6017: Processing resource auto-start for servers: srv1
CRS-2672: Attempting to start 'ora.srv1.vip' on 'srv1'
CRS-2672: Attempting to start 'ora.scan3.vip' on 'srv1'
CRS-2672: Attempting to start 'ora.scan2.vip' on 'srv1'
CRS-2672: Attempting to start 'ora.scan1.vip' on 'srv1'
CRS-2672: Attempting to start 'ora.srv2.vip' on 'srv1'
CRS-2672: Attempting to start 'ora.ons' on 'srv1'
CRS-2672: Attempting to start 'ora.chad' on 'srv1'
CRS-2672: Attempting to start 'ora.qosmserver' on 'srv1'
CRS-2676: Start of 'ora.srv1.vip' on 'srv1' succeeded
CRS-2672: Attempting to start 'ora.LISTENER.lsnr' on 'srv1'
CRS-2676: Start of 'ora.srv2.vip' on 'srv1' succeeded
CRS-2676: Start of 'ora.scan3.vip' on 'srv1' succeeded
CRS-2672: Attempting to start 'ora.LISTENER_SCAN3.lsnr' on 'srv1'
CRS-2676: Start of 'ora.scan1.vip' on 'srv1' succeeded
CRS-2672: Attempting to start 'ora.LISTENER_SCAN1.lsnr' on 'srv1'
CRS-2676: Start of 'ora.scan2.vip' on 'srv1' succeeded
CRS-2672: Attempting to start 'ora.LISTENER_SCAN2.lsnr' on 'srv1'
CRS-2676: Start of 'ora.chad' on 'srv1' succeeded
CRS-2676: Start of 'ora.LISTENER.lsnr' on 'srv1' succeeded
CRS-2676: Start of 'ora.ons' on 'srv1' succeeded
CRS-2676: Start of 'ora.LISTENER_SCAN1.lsnr' on 'srv1' succeeded
CRS-2676: Start of 'ora.LISTENER_SCAN3.lsnr' on 'srv1' succeeded
CRS-2676: Start of 'ora.LISTENER_SCAN2.lsnr' on 'srv1' succeeded
CRS-2676: Start of 'ora.qosmserver' on 'srv1' succeeded
CRS-6016: Resource auto-start has completed for server srv1
CRS-6024: Completed start of Oracle Cluster Ready Services-managed resources
CRS-4123: Oracle High Availability Services has been started.
[root@srv1 ~]#
[root@srv1 ~]# crsctl stat res -t
--------------------------------------------------------------------------------
Name Target State Server State details
--------------------------------------------------------------------------------
Local Resources
--------------------------------------------------------------------------------
ora.LISTENER.lsnr
ONLINE ONLINE srv1 STABLE
ora.chad
ONLINE ONLINE srv1 STABLE
ora.net1.network
ONLINE ONLINE srv1 STABLE
ora.ons
ONLINE ONLINE srv1 STABLE
--------------------------------------------------------------------------------
Cluster Resources
--------------------------------------------------------------------------------
ora.ASMNET1LSNR_ASM.lsnr(ora.asmgroup)
1 ONLINE ONLINE srv1 STABLE
2 ONLINE OFFLINE STABLE
3 ONLINE OFFLINE STABLE
ora.DATA.dg(ora.asmgroup)
1 ONLINE ONLINE srv1 STABLE
2 OFFLINE OFFLINE STABLE
3 OFFLINE OFFLINE STABLE
ora.FRA.dg(ora.asmgroup)
1 ONLINE ONLINE srv1 STABLE
2 OFFLINE OFFLINE STABLE
3 OFFLINE OFFLINE STABLE
ora.LISTENER_SCAN1.lsnr
1 ONLINE ONLINE srv1 STABLE
ora.LISTENER_SCAN2.lsnr
1 ONLINE ONLINE srv1 STABLE
ora.LISTENER_SCAN3.lsnr
1 ONLINE ONLINE srv1 STABLE
ora.CONFIG.dg(ora.asmgroup)
1 OFFLINE OFFLINE STABLE
2 OFFLINE OFFLINE STABLE
3 OFFLINE OFFLINE STABLE
ora.asm(ora.asmgroup)
1 ONLINE ONLINE srv1 Started,STABLE
2 ONLINE OFFLINE STABLE
3 OFFLINE OFFLINE STABLE
ora.asmnet1.asmnetwork(ora.asmgroup)
1 ONLINE ONLINE srv1 STABLE
2 ONLINE OFFLINE STABLE
3 OFFLINE OFFLINE STABLE
ora.cvu
1 ONLINE ONLINE srv1 STABLE
ora.qosmserver
1 ONLINE ONLINE srv1 STABLE
ora.ibge.db
1 ONLINE ONLINE srv1 Open,HOME=/u01/app/o
racle/product/19.0.0
/db_1,STABLE
2 ONLINE ONLINE srv2 Open,HOME=/u01/app/o
racle/product/19.0.0
/db_1,STABLE
ora.scan1.vip
1 ONLINE ONLINE srv1 STABLE
ora.scan2.vip
1 ONLINE ONLINE srv1 STABLE
ora.scan3.vip
1 ONLINE ONLINE srv1 STABLE
ora.srv1.vip
1 ONLINE ONLINE srv1 STABLE
ora.srv2.vip
1 ONLINE INTERMEDIATE srv1 FAILED OVER,STABLE
--------------------------------------------------------------------------------
[root@srv1 ~]#
##NODE02##
[root@srv2 ~]#
[root@srv2 ~]#
[root@srv2 ~]# crsctl start crs -wait
]CRS-4123: Starting Oracle High Availability Services-managed resources
CRS-2672: Attempting to start 'ora.mdnsd' on 'srv2'
CRS-2672: Attempting to start 'ora.evmd' on 'srv2'
CRS-2676: Start of 'ora.evmd' on 'srv2' succeeded
CRS-2676: Start of 'ora.mdnsd' on 'srv2' succeeded
CRS-2672: Attempting to start 'ora.gpnpd' on 'srv2'
CRS-2676: Start of 'ora.gpnpd' on 'srv2' succeeded
CRS-2672: Attempting to start 'ora.gipcd' on 'srv2'
CRS-2676: Start of 'ora.gipcd' on 'srv2' succeeded
CRS-2672: Attempting to start 'ora.crf' on 'srv2'
CRS-2672: Attempting to start 'ora.cssdmonitor' on 'srv2'
CRS-2676: Start of 'ora.cssdmonitor' on 'srv2' succeeded
CRS-2672: Attempting to start 'ora.cssd' on 'srv2'
CRS-2672: Attempting to start 'ora.diskmon' on 'srv2'
CRS-2676: Start of 'ora.diskmon' on 'srv2' succeeded
CRS-2676: Start of 'ora.crf' on 'srv2' succeeded
CRS-2676: Start of 'ora.cssd' on 'srv2' succeeded
CRS-2672: Attempting to start 'ora.cluster_interconnect.haip' on 'srv2'
CRS-2672: Attempting to start 'ora.ctssd' on 'srv2'
CRS-2676: Start of 'ora.ctssd' on 'srv2' succeeded
CRS-2676: Start of 'ora.cluster_interconnect.haip' on 'srv2' succeeded
CRS-2672: Attempting to start 'ora.asm' on 'srv2'
CRS-2676: Start of 'ora.asm' on 'srv2' succeeded
CRS-2672: Attempting to start 'ora.storage' on 'srv2'
CRS-2676: Start of 'ora.storage' on 'srv2' succeeded
CRS-2672: Attempting to start 'ora.crsd' on 'srv2'
CRS-2676: Start of 'ora.crsd' on 'srv2' succeeded
CRS-6017: Processing resource auto-start for servers: srv2
CRS-2673: Attempting to stop 'ora.srv2.vip' on 'srv1'
CRS-2673: Attempting to stop 'ora.LISTENER_SCAN1.lsnr' on 'srv1'
CRS-2672: Attempting to start 'ora.ons' on 'srv2'
CRS-2672: Attempting to start 'ora.chad' on 'srv2'
CRS-2677: Stop of 'ora.srv2.vip' on 'srv1' succeeded
CRS-2672: Attempting to start 'ora.srv2.vip' on 'srv2'
CRS-2677: Stop of 'ora.LISTENER_SCAN1.lsnr' on 'srv1' succeeded
CRS-2673: Attempting to stop 'ora.scan1.vip' on 'srv1'
CRS-2677: Stop of 'ora.scan1.vip' on 'srv1' succeeded
CRS-2672: Attempting to start 'ora.scan1.vip' on 'srv2'
CRS-2676: Start of 'ora.srv2.vip' on 'srv2' succeeded
CRS-2672: Attempting to start 'ora.LISTENER.lsnr' on 'srv2'
CRS-2676: Start of 'ora.scan1.vip' on 'srv2' succeeded
CRS-2672: Attempting to start 'ora.LISTENER_SCAN1.lsnr' on 'srv2'
CRS-2676: Start of 'ora.chad' on 'srv2' succeeded
CRS-2676: Start of 'ora.LISTENER.lsnr' on 'srv2' succeeded
CRS-2676: Start of 'ora.ons' on 'srv2' succeeded
CRS-2676: Start of 'ora.LISTENER_SCAN1.lsnr' on 'srv2' succeeded
CRS-6016: Resource auto-start has completed for server srv2
CRS-6024: Completed start of Oracle Cluster Ready Services-managed resources
CRS-4123: Oracle High Availability Services has been started.
[root@srv2 ~]# ]
bash: ]: command not found...
[root@srv2 ~]#
[root@srv2 ~]# crsctl stat res -t
--------------------------------------------------------------------------------
Name Target State Server State details
--------------------------------------------------------------------------------
Local Resources
--------------------------------------------------------------------------------
ora.LISTENER.lsnr
ONLINE ONLINE srv1 STABLE
ONLINE ONLINE srv2 STABLE
ora.chad
ONLINE ONLINE srv1 STABLE
ONLINE ONLINE srv2 STABLE
ora.net1.network
ONLINE ONLINE srv1 STABLE
ONLINE ONLINE srv2 STABLE
ora.ons
ONLINE ONLINE srv1 STABLE
ONLINE ONLINE srv2 STABLE
--------------------------------------------------------------------------------
Cluster Resources
--------------------------------------------------------------------------------
ora.ASMNET1LSNR_ASM.lsnr(ora.asmgroup)
1 ONLINE ONLINE srv1 STABLE
2 ONLINE OFFLINE STABLE
3 ONLINE OFFLINE STABLE
ora.DATA.dg(ora.asmgroup)
1 ONLINE ONLINE srv1 STABLE
2 ONLINE ONLINE srv2 STABLE
3 OFFLINE OFFLINE STABLE
ora.FRA.dg(ora.asmgroup)
1 ONLINE ONLINE srv1 STABLE
2 ONLINE ONLINE srv2 STABLE
3 OFFLINE OFFLINE STABLE
ora.LISTENER_SCAN1.lsnr
1 ONLINE ONLINE srv2 STABLE
ora.LISTENER_SCAN2.lsnr
1 ONLINE ONLINE srv1 STABLE
ora.LISTENER_SCAN3.lsnr
1 ONLINE ONLINE srv1 STABLE
ora.CONFIG.dg(ora.asmgroup)
1 OFFLINE OFFLINE STABLE
2 OFFLINE OFFLINE STABLE
3 OFFLINE OFFLINE STABLE
ora.asm(ora.asmgroup)
1 ONLINE ONLINE srv1 Started,STABLE
2 ONLINE ONLINE srv2 Started,STABLE
3 OFFLINE OFFLINE STABLE
ora.asmnet1.asmnetwork(ora.asmgroup)
1 ONLINE ONLINE srv1 STABLE
2 ONLINE OFFLINE STABLE
3 OFFLINE OFFLINE STABLE
ora.cvu
1 ONLINE ONLINE srv1 STABLE
ora.qosmserver
1 ONLINE ONLINE srv1 STABLE
ora.ibge.db
1 ONLINE ONLINE srv1 Open,HOME=/u01/app/o
racle/product/19.0.0
/db_1,STABLE
2 ONLINE ONLINE srv2 Open,HOME=/u01/app/o
racle/product/19.0.0
/db_1,STABLE
ora.scan1.vip
1 ONLINE ONLINE srv2 STABLE
ora.scan2.vip
1 ONLINE ONLINE srv1 STABLE
ora.scan3.vip
1 ONLINE ONLINE srv1 STABLE
ora.srv1.vip
1 ONLINE ONLINE srv1 STABLE
ora.srv2.vip
1 ONLINE ONLINE srv2 STABLE
--------------------------------------------------------------------------------
[root@srv2 ~]#
Agora, com o DiskGroup CONFIG recriado e o OCR restaurado, o cluster voltou a funcionar normalmente. Garantimos que, mesmo após a exclusão do DiskGroup, conseguimos restaurar e reestabelecer o ambiente. Lembre-se de sempre ter backups de segurança em múltiplos locais.
🧙♂️🧌#20241027 #DBASobrinho #GuinaNãoTinhaDó #BóBó #CaceteDeAgulha #OracleACE #GuinaNoPortão #OGuinaFoieDeuMaisTrês #MeuModo #DemarcoMeuEspaço #VivãoEVivendo🧙♂️🧌
