记⼀次Ceph故障处理
ceph存储集群是建⽴在两台服务器上⾯，两台服务器各有4个OSD节点，上班的时候发现，两台服务器上其中⼀台服务器上4个OSD节点全部掉线，重启OSD节点后恢复正常。查看OSD节点⽇志后发现，整个故障过程如下：
1. 单个OSD节点接收不到另外⼀台服务器节点上的所有OSD⼼跳信息，⽇志记录如下
2016-03-21 17:43:42.426051 7f716e0a7700 -1 osd.0 70 heartbeat_check: no reply from osd.5 since back 2016-03-21 17:43:22.233321 front 2016-03-21 17:43:41.335367 (cutoff 2016-03-21 17:43:22.426050)
2016-03-21 17:43:42.426061 7f716e0a7700 -1 osd.0 70 heartbeat_check: no reply from osd.6 since back 2016-03-21 17:43:22.233321 front 2016-03-21 17:43:41.335367 (cutoff 2016-03-21 17:43:22.426050)
2016-03-21 17:43:42.426072 7f716e0a7700 -1 osd.0 70 heartbeat_check: no reply from osd.7 since back 2016-03-21 17:43:22.233321 front 2016-03-21 17:43:41.335367 (cutoff 2016-03-21 17:43:22.426050)
2016-03-21 17:43:42.426075 7f716e0a7700 -1 osd.0 70 heartbeat_check: no reply from osd.8 since back 2016-03-21 17:43:22.233321 front 2016-03-21 17:43:41.335367 (cutoff 2016-03-21 17:43:22.426050)
2. 单个OSD节点接收不到相同服务器上其他节点的OSD⼼跳信息；
2016-03-21 17:43:53.427874 7f716e0a7700 -1 osd.0 74 heartbeat_check: no reply from osd.1 since back 2016-03-21 17:43:26.134353 front 2016-03-21 17:43:51.036748 (cutoff 2016-03-21 17:43:33.427873)
2016-03-21 17:43:53.427911 7f716e0a7700 -1 osd.0 74 heartbeat_check: no reply from osd.3 since back 2016-03-21 17:43:26.134353 front 2016-03-21 17:43:51.036748 (cutoff 2016-03-21 17:43:33.427873)
2016-03-21 17:44:07.937952 7f71410e4700 -1 osd.0 75 heartbeat_check: no reply from osd.2 ever on either front or back, first ping nt 2016-03-
顺畅的意思21 17:43:47.636418 (cutoff 2016-03-21 17:43:47.937950)
3. 该OSD节点被map标记为down状态，⽇志提⽰是：map wrongly mark me down；
2016-03-21 18:52:58.365106 7f715610e700 0 log_channel(cluster) log [WRN] : map e83 wrongly marked me down
4. OSD节点⽆法绑定到⽹卡6800-7300中任意⼀个端⼝；
2016-03-21 18:52:58.366288 7f715610e700 -1 accepter.accepter.bind unable to bind to 192.168.10.210:7300 on any port in range 6800-
7300: (99) Cannot assign requested address
2016-03-21 18:52:58.366318 7f715610e700 -1 accepter.accepter.bind was unable to bind. Trying again in 5 conds
5. OSD节点准备进⼊down状态，开始存储⽂件，进⼊down状态。
2016-03-21 18:53:18.383806 7f715610e700 0 osd.0 83 prepare_to_stop starting shutdown头痛吃什么药
2016-03-21 18:53:18.383817 7f715610e700 -1 osd.0 83 shutdown
2016-03-21 18:53:18.383996 7f715610e700 20 osd.0 83 kicking pg 4.76
进步之星
然后查看其他mds⽇志，mon⽇志均没有查到原因所在，于是去查看系统⽇志/var/log/messages，查看故障出现那个时间点的⽇志，这才找到原因所在，⽇志如下：
Mar 21 17:43:23 cephadmin1 kernel: e1000: enp5s0 NIC Link is Down
Mar 21 17:43:23 cephadmin1 NetworkManager[1021]: <info> (enp5s0): link disconnected (deferring action for 4 conds)
大地的声音
Mar 21 17:43:28 cephadmin1 NetworkManager[1021]: <info> (enp5s0): link disconnected (calling deferred action)第一次坐飞机作文
Mar 21 17:43:28 cephadmin1 NetworkManager[1021]: <info> (enp5s0): device state change: activated -> unavailable (reason 'carrier-
changed') [100 20 40]
Mar 21 17:43:28 cephadmin1 NetworkManager[1021]: <info> (enp5s0): canceled DHCP transaction, DHCP client pid 32494
Mar 21 17:43:28 cephadmin1 NetworkManager[1021]: <info> (enp5s0): DHCPv6 state changed bound -> done
Mar 21 17:43:28 cephadmin1 avahi-daemon[973]: Withdrawing address record for 192.168.10.210 on enp5s0.
Mar 21 17:43:28 cephadmin1 avahi-daemon[973]: Leaving mDNS multicast group on interface enp5s0.IPv4 with address 192.168.10.210.
Mar 21 17:43:28 cephadmin1 avahi-daemon[973]: Interface enp5s0.IPv4 no longer relevant for mDNS.
Mar 21 17:43:28 cephadmin1 avahi-daemon[973]: Withdrawing address record for 2001:470:4b:34f:207:e9ff:fe08:51 on enp5s0.
Mar 21 17:43:28 cephadmin1 avahi-daemon[973]: Withdrawing address record for fd26:b3dd:d727:0:207:e9ff:fe08:51 on enp5s0.
Mar 21 17:43:28 cephadmin1 avahi-daemon[973]: Withdrawing address record for fd26:b3dd:d727::863 on enp5s0.
Mar 21 17:43:28 cephadmin1 NetworkManager[1021]: <info> Policy t 'enp3s0' (enp3s0) as default for IPv6 routing and DNS.
Mar 21 17:43:28 cephadmin1 dbus-daemon: dbus[987]: [system] Activating via systemd: rvice name='_dispatcher' unit='-dispatcher.rvice'
牙龈出血什么原因Mar 21 17:43:28 cephadmin1 dbus[987]: [system] Activating via systemd: rvice name='_dispatcher' unit='dbus-
-dispatcher.rvice'
Mar 21 17:43:28 cephadmin1 systemd: Starting Network Manager Script
Mar 21 17:43:28 cephadmin1 dbus[987]: [system] Successfully activated rvice '_dispatcher'
Mar 21 17:43:28 cephadmin1 systemd: Started Network Manager Script Dispatcher Service.
Mar 21 17:43:28 cephadmin1 dbus-daemon: dbus[987]: [system] Successfully activated rvice '_dispatcher'
Mar 21 17:43:28 cephadmin1 nm-dispatcher: Dispatching action 'down' for enp5s0
Mar 21 17:43:41 cephadmin1 ceph-osd: 2016-03-21 17:43:41.426082 7efd2c5b4700 -
1 osd.3 70 heartbeat_check: no reply from osd.5 since back 2016-03-21 17:43:21.362339 front 2016-03-21 17:43:41.064144 (cutoff 2016-03-
悠悠人体艺术21 17:43:21.426080)
Mar 21 17:43:41 cephadmin1 ceph-osd: 2016-03-21 17:43:41.426157 7efd2c5b4700 -
1 osd.3 70 heartbeat_check: no reply from osd.6 since back 2016-03-21 17:43:21.362339 front 2016-03-21 17:43:41.064144 (cutoff 2016-03-
21 17:43:21.426080)
从⽇志中我们可以清楚地看到，这个时间点，ceph集群⽤来传输数据的⽹卡down掉了，然后NetworkManager开始处理⽹卡的⼀些信息并尝试重启⽹卡，但是重启失败。突然想起来以前在⼀篇⽂章中看到过，CentOS系统中⾃带的network程序会和GNOME桌⾯带的NetworManger冲突，导致⽹卡出现问题。这⾥应该也是相同的原因。于是关掉NetworkManger并关掉它的⾃启动程序。
systemctl stop NetworkManger
systemctl disable NetworkManager
幼儿园课程过了两天每天观察，再没出现过单个服务器所有OSD节点全部down掉的故障，说明故障原因确实在这⾥。
