测试环境存储磁盘问题，导致RAC一个节点CRS启动失败
linux 5.6 *64, 2节点RAC 11.2.0.4
如下，按照日志，说明信息。
主机重启，发现CRS进程并未自动启动完成。a1:/picclife/app/oracle$ crsctl check crs
CRS-4638: Oracle High Availability Services is online
CRS-4535: Cannot communicate with Cluster Ready Services
CRS-4530: Communications failure contacting Cluster Synchronization Services daemon
CRS-4534: Cannot communicate with Event Manager
查询当前节点信息，发现ASM实例 OFFLINE
a1:/picclife/app/grid$ crsctl stat res -t -init
--------------------------------------------------------------------------------
NAME TARGET STATE SERVER STATE_DETAILS
--------------------------------------------------------------------------------
Cluster Resources
--------------------------------------------------------------------------------
ora.asm
1 ONLINE OFFLINE Instance Shutdown
ora.cluster_interconnect.haip
1 ONLINE OFFLINE
ora.crf
1 ONLINE ONLINE a1
ora.crsd
1 ONLINE OFFLINE
ora.cssd
1 ONLINE OFFLINE STARTING
ora.cssdmonitor
1 ONLINE ONLINE a1
ora.ctssd
1 ONLINE OFFLINE
ora.diskmon
1 OFFLINE OFFLINE
ora.evmd
1 ONLINE OFFLINE
ora.gipcd
1 ONLINE ONLINE a1
ora.gpnpd
1 ONLINE ONLINE a1
ora.mdnsd
1 ONLINE ONLINE a1
手工启动ASM实例，报错
grid$sqlplus / as sysasm
SQL> startup
ORA-01078: failure in processing system parameters
ORA-29701: unable to connect to Cluster Synchronization Service
查询集群Alert,发现启动资源失败报错，提升表决磁盘信息存在问题，并且指向日志
grid$cd $ORACLE_HOME/log/node_name/
grid$tail -200f a*.log
2020-01-18 10:46:30.622:
[ohasd(2344)]CRS-2765:Resource 'ora.cssdmonitor' has failed on server 'a1'.
2020-01-18 10:46:32.278:
[cssd(4941)]CRS-1713:CSSD daemon is started in clustered mode
2020-01-18 10:46:32.329:
[cssd(4941)]CRS-1714:Unable to discover any voting files, retrying discovery in 15 seconds; Details at (:CSSNM00070:) in /picclife/app/11.2.0/grid/log/a1/cssd/ocssd.log
2020-01-18 10:46:33.920:
[ohasd(2344)]CRS-2767:Resource state recovery not attempted for 'ora.diskmon' as its target state is OFFLINE
2020-01-18 10:46:33.920:
[ohasd(2344)]CRS-2769:Unable to failover resource 'ora.diskmon'.
2020-01-18 10:46:47.335:
[cssd(4941)]CRS-1714:Unable to discover any voting files, retrying discovery in 15 seconds; Details at (:CSSNM00070:) in /picclife/app/11.2.0/grid/log/a1/cssd/ocssd.log
2020-01-18 10:47:02.340:
[cssd(4941)]CRS-1714:Unable to discover any voting files, retrying discovery in 15 seconds; Details at (:CSSNM00070:) in /picclife/app/11.2.0/grid/log/a1/cssd/ocssd.log
根据集群alert日志，指向，查询ocssd.log日志， clssnmvDiskVerify: Successful discovery of 0 disks 未发现存在一块磁盘！！！
$ tail -200f /picclife/app/11.2.0/grid/log/a1/cssd/ocssd.log
2020-01-18 10:47:47.358: [ CSSD][1087162688]clssnmReadDiscoveryProfile: voting file discovery string(/dev/asm*)
2020-01-18 10:47:47.358: [ CSSD][1087162688]clssnmvDDiscThread: using discovery string /dev/asm* for initial discovery
2020-01-18 10:47:47.358: [ SKGFD][1087162688]Discovery with str:/dev/asm*:
2020-01-18 10:47:47.358: [ SKGFD][1087162688]UFS discovery with :/dev/asm*:
2020-01-18 10:47:47.358: [ SKGFD][1087162688]Execute glob on the string /dev/asm*
2020-01-18 10:47:47.358: [ SKGFD][1087162688]OSS discovery with :/dev/asm*:
2020-01-18 10:47:47.358: [ CSSD][1087162688]clssnmvDiskVerify: Successful discovery of 0 disks
2020-01-18 10:47:47.358: [ CSSD][1087162688]clssnmCompleteInitVFDiscovery: Completing initial voting file discovery
2020-01-18 10:47:47.358: [ CSSD][1087162688]clssnmvFindInitialConfigs: No voting files found
2020-01-18 10:47:47.358: [ CSSD][1087162688](:CSSNM00070:)clssnmCompleteInitVFDiscovery: Voting file not found. Retrying discovery in 15 seconds
2020-01-18 10:47:47.486: [ CSSD][1100994880]clssscSelect: cookie accept request 0xdee1a0
2020-01-18 10:47:47.486: [ CSSD][1100994880]clssgmAllocProc: (0x122c840) allocated
2020-01-18 10:47:47.486: [ CSSD][1100994880]clssgmClientConnectMsg: properties of cmProc 0x122c840 - 1,2,3,4,5
2020-01-18 10:47:47.486: [ CSSD][1100994880]clssgmClientConnectMsg: Connect from con(0x1416) proc(0x122c840) pid(2686) version 11:2:1:4, properties: 1,2,3,4,5
2020-01-18 10:47:47.486: [ CSSD][1100994880]clssgmClientConnectMsg: msg flags 0x0000
2020-01-18 10:47:47.725: [ CSSD][1100994880]clssscSelect: cookie accept request 0x1222de0
2020-01-18 10:47:47.725: [ CSSD][1100994880]clssscevtypSHRCON: getting client with cmproc 0x1222de0
2020-01-18 10:47:47.725: [ CSSD][1100994880]clssgmRegisterClient: proc(4/0x1222de0), client(73/0x122d7d0)
2020-01-18 10:47:47.725: [ CSSD][1100994880]clssgmExecuteClientRequest(): type(6) size(684) only connect and exit messages are allowed before lease acquisition proc(0x1222de0) client(0x122d7d0)
查询测试环境，默认存储是/dev/asm*磁盘，未发现asm磁盘
[root@a1 dev]# ls -lrt asm*
ls: asm*: No such file or directory
[root@a1 ~]# fdisk -l
存储问题，未加载ASM磁盘，系统磁盘问题，丢盘了，重新虚拟机加载磁盘，重启主机后，问题解决。
来源：https://www.cnblogs.com/lvcha001/p/12218529.html