Pacemaker Corosync: различия между версиями

Материал из noname.com.ua
Перейти к навигацииПерейти к поиску
 
(не показаны 52 промежуточные версии этого же участника)
Строка 9: Строка 9:
 
Это заметка о том как восстанавливать разваленный кластер, что бы не потерять если еще раз понадобится.
  
Это заметка о том как восстанавливать разваленный кластер, что бы не потерять если еще раз понадобится.
   
  +
На любом контроллере (в этом вом вопросе все контроллеры равнозначны):
−
From any controller in the cluster, disable MySQL resource in Pacemaker by running the following command:
  
 
<PRE>
  
<PRE>
 
pcs resource disable clone_p_mysqld
  
pcs resource disable clone_p_mysqld
 
</PRE>
  
</PRE>
  +
Дождаться пока все процессы <code>mysqld</code> будут остановлены<BR>
−
Wait a certain amount of time for MySQL to shut down.
  
  +
−
Verify that clone set clone_p_mysql is stopped on all controllers:
  
  +
Убедиться что ресурс <code>clone_p_mysql</code> остановлен на всех контроллерах:
  +
<PRE>
 
pcs status resources
  
pcs status resources
  +
</PRE>
−
On every controller in the cluster remove content of MySQL data directory (or move it to a different place):
  
  +
На всех контроллерах зачистить (перенести или удалить) данные mysql:
 
<PRE>
  
<PRE>
 
mv /var/lib/mysql/* /tmp/mysql/
  
mv /var/lib/mysql/* /tmp/mysql/
 
</PRE>
  
</PRE>
  +
Выбрать один контроллер как основной для восстановления (это может быть любой контроллер), далее называем его
−
Choose one of controllers that you are going to restore first. As an example, we will choose controller node named as controller-x
  
  +
<code>controller-x</code>
−
Copy database backup to MySQL data directory on controller-x:
  
  +
  +
Восстановить бекап (это конечно хорошо, если он есть) на <code>controller-x</code>:
 
<PRE>
  
<PRE>
 
cp -R /ext-volume/mysql-backup/* /var/lib/mysql/
  
cp -R /ext-volume/mysql-backup/* /var/lib/mysql/
 
</PRE>
  
</PRE>
  +
Поправить права на <code>controller-x</code>:
−
Change the owner of MySQL data directory on controller-x:
  
 
<PRE>
  
<PRE>
 
chown -R mysql:mysql /var/lib/mysql
  
chown -R mysql:mysql /var/lib/mysql
 
</PRE>
  
</PRE>
  +
Для <code>OCF</code> скриптов нужны переменные окружения, их нужно установить перед запуском скрипта <code>mysql-wss</code><BR>
−
Export variables for mysql-wss and start mysqld on controller-x:
  
  +
После чего стартовать базу на выбранном контроллере <code>controller-x</code>.
  +
<BR>
  +
Обратить внимание что первый (и только первый) контроллер запускается с параметром <code>--wsrep-new-cluster</code>
 
<PRE>
  
<PRE>
 
export OCF_RESOURCE_INSTANCE=p_mysqld
  
export OCF_RESOURCE_INSTANCE=p_mysqld
Строка 40: Строка 48:
 
/usr/lib/ocf/resource.d/fuel/mysql-wss start
  
/usr/lib/ocf/resource.d/fuel/mysql-wss start
 
</PRE>
  
</PRE>
−
Execute monitor operation on controller-x to update Galera GTID in Pacemaker cluster configuration:
 +
Запустить monitor operation на <code>controller-x</code> to update Galera GTID in Pacemaker cluster configuration,
  +
<BR>
  +
Тоже нужны переменные окружения
 
<PRE>
  
<PRE>
 
/usr/lib/ocf/resource.d/fuel/mysql-wss monitor
  
/usr/lib/ocf/resource.d/fuel/mysql-wss monitor
 
</PRE>
  
</PRE>
  +
Запустить на остальных 2-х контроллерах:
−
Export variables for mysql-wss and start mysqld on all other controllers:
  
 
<PRE>
  
<PRE>
 
export OCF_RESOURCE_INSTANCE=p_mysqld
  
export OCF_RESOURCE_INSTANCE=p_mysqld
Строка 54: Строка 64:
 
/usr/lib/ocf/resource.d/fuel/mysql-wss start
  
/usr/lib/ocf/resource.d/fuel/mysql-wss start
 
</PRE>
  
</PRE>
  +
С любого контроллера включить ресурс mysql в Pacemaker:
−
From any controller in the cluster, enable MySQL resource in Pacemaker by running the following command:
  
 
<PRE>
  
<PRE>
 
pcs resource enable clone_p_mysqld
  
pcs resource enable clone_p_mysqld
 
</PRE>
  
</PRE>
  +
Убедиться что ресурс стартовал на всех контроллерах:
−
Verify that clone set clone_p_mysqld is running on all controllers:
  
 
<PRE>
  
<PRE>
 
pcs status resources
  
pcs status resources
 
</PRE>
  
</PRE>
  +
  +
  +
https://clusterlabs.org/pacemaker/doc/2.1/Pacemaker_Explained/html/nodes.html#tracking-node-health
  +
  +
=Что делать если все упало из-за занятого места на дисках=
  +
  +
==Как проверить?==
  +
Узнать что есть проблемы с диском с точки зрения Pacemaker можно из лога <code>/var/log/pacemaker.log</code>
  +
<PRE>
  +
# grep 'health_disk.*value="red"' /var/log/pacemaker.log
  +
May 10 02:53:21 [9160] node-X.default.ltd cib: info: cib_perform_op: ++ /cib/status/node_state[@id='X']/transient_attributes[@id='X']/instance_attributes[@id='status-X']: <nvpair id="status-X-#health_disk" name="#health_disk" value="red"/>
  +
</PRE>
  +
  +
​After the health disk change you will see a health strategy: "migrate-on-red"
  +
<PRE>
  +
May 10 02:53:21 [9164] node-X.default.ltd pengine: info: apply_system_health: Applying automated node health strategy: migrate-on-red
  +
</PRE>
  +
После чего какие-то ресурсы не могут стартовать 0 обращать внимание на
  +
* <code>Applying automated node health strategy: migrate-on-red</code>
  +
* <code>Resource p_haproxy:0 cannot run anywhere</code>
  +
  +
{{#spoiler:show=Пример лога|
  +
  +
<PRE>
  +
On a Cluster of only one Controller; all services leave the node and can't find a suitable node
  +
May 10 02:53:51 [9164] node-X.default.ltd pengine: info: determine_online_status: Node node-X.default.ltd is online
  +
May 10 02:53:51 [9164] node-X.default.ltd pengine: info: determine_op_status: Operation monitor found resource p_vrouter:0 active on node-X.default.ltd
  +
May 10 02:53:51 [9164] node-X.default.ltd pengine: info: apply_system_health: Applying automated node health strategy: migrate-on-red
  +
May 10 02:53:51 [9164] node-X.default.ltd pengine: info: apply_system_health: Node node-X.default.ltd has an combined system health of -1000000
  +
May 10 02:53:51 [9164] node-X.default.ltd pengine: info: native_print: sysinfo_node-X.default.ltd (ocf::pacemaker:SysInfo): Stopped
  +
May 10 02:53:51 [9164] node-X.default.ltd pengine: info: clone_print: Clone Set: clone_p_vrouter [p_vrouter]
  +
May 10 02:53:51 [9164] node-X.default.ltd pengine: info: short_print: Stopped: [ node-X.default.ltd ]
  +
May 10 02:53:51 [9164] node-X.default.ltd pengine: info: native_print: vip__management (ocf::fuel:ns_IPaddr2): Stopped
  +
May 10 02:53:51 [9164] node-X.default.ltd pengine: info: native_print: vip__vrouter_pub (ocf::fuel:ns_IPaddr2): Stopped
  +
May 10 02:53:51 [9164] node-X.default.ltd pengine: info: native_print: vip__vrouter (ocf::fuel:ns_IPaddr2): Stopped
  +
May 10 02:53:51 [9164] node-X.default.ltd pengine: info: native_print: vip__public (ocf::fuel:ns_IPaddr2): Stopped
  +
May 10 02:53:51 [9164] node-X.default.ltd pengine: info: clone_print: Clone Set: clone_p_haproxy [p_haproxy]
  +
May 10 02:53:51 [9164] node-X.default.ltd pengine: info: short_print: Stopped: [ node-X.default.ltd ]
  +
May 10 02:53:51 [9164] node-X.default.ltd pengine: info: clone_print: Clone Set: clone_p_mysql [p_mysql]
  +
May 10 02:53:51 [9164] node-X.default.ltd pengine: info: short_print: Stopped: [ node-X.default.ltd ]
  +
May 10 02:53:51 [9164] node-X.default.ltd pengine: info: clone_print: Master/Slave Set: master_p_rabbitmq-server [p_rabbitmq-server]
  +
May 10 02:53:51 [9164] node-X.default.ltd pengine: info: short_print: Stopped: [ node-X.default.ltd ]
  +
May 10 02:53:51 [9164] node-X.default.ltd pengine: info: clone_print: Clone Set: clone_p_dns [p_dns]
  +
May 10 02:53:51 [9164] node-X.default.ltd pengine: info: short_print: Stopped: [ node-X.default.ltd ]
  +
May 10 02:53:51 [9164] node-X.default.ltd pengine: info: clone_print: Master/Slave Set: master_p_conntrackd [p_conntrackd]
  +
May 10 02:53:51 [9164] node-X.default.ltd pengine: info: short_print: Stopped: [ node-X.default.ltd ]
  +
May 10 02:53:51 [9164] node-X.default.ltd pengine: info: clone_print: Clone Set: clone_p_heat-engine [p_heat-engine]
  +
May 10 02:53:51 [9164] node-X.default.ltd pengine: info: short_print: Stopped: [ node-X.default.ltd ]
  +
May 10 02:53:51 [9164] node-X.default.ltd pengine: info: clone_print: Clone Set: clone_p_neutron-plugin-openvswitch-agent [p_neutron-plugin-openvswitch-agent]
  +
May 10 02:53:51 [9164] node-X.default.ltd pengine: info: short_print: Stopped: [ node-X.default.ltd ]
  +
May 10 02:53:51 [9164] node-X.default.ltd pengine: info: clone_print: Clone Set: clone_p_neutron-l3-agent [p_neutron-l3-agent]
  +
May 10 02:53:51 [9164] node-X.default.ltd pengine: info: short_print: Stopped: [ node-X.default.ltd ]
  +
May 10 02:53:51 [9164] node-X.default.ltd pengine: info: clone_print: Clone Set: clone_p_neutron-dhcp-agent [p_neutron-dhcp-agent]
  +
May 10 02:53:51 [9164] node-X.default.ltd pengine: info: short_print: Stopped: [ node-X.default.ltd ]
  +
May 10 02:53:51 [9164] node-X.default.ltd pengine: info: clone_print: Clone Set: clone_p_neutron-metadata-agent [p_neutron-metadata-agent]
  +
May 10 02:53:51 [9164] node-X.default.ltd pengine: info: short_print: Stopped: [ node-X.default.ltd ]
  +
May 10 02:53:51 [9164] node-X.default.ltd pengine: info: clone_print: Clone Set: clone_ping_vip__public [ping_vip__public]
  +
May 10 02:53:51 [9164] node-X.default.ltd pengine: info: short_print: Stopped: [ node-X.default.ltd ]
  +
May 10 02:53:51 [9164] node-X.default.ltd pengine: info: clone_print: Clone Set: clone_p_ntp [p_ntp]
  +
May 10 02:53:51 [9164] node-X.default.ltd pengine: info: short_print: Stopped: [ node-X.default.ltd ]
  +
May 10 02:53:51 [9164] node-X.default.ltd pengine: info: native_color: Resource sysinfo_node-X.default.ltd cannot run anywhere
  +
May 10 02:53:51 [9164] node-X.default.ltd pengine: info: rsc_merge_weights: clone_p_vrouter: Rolling back scores from clone_p_dns
  +
May 10 02:53:51 [9164] node-X.default.ltd pengine: info: native_color: Resource p_vrouter:0 cannot run anywhere
  +
May 10 02:53:51 [9164] node-X.default.ltd pengine: info: rsc_merge_weights: clone_p_haproxy: Rolling back scores from vip__management
  +
May 10 02:53:51 [9164] node-X.default.ltd pengine: info: rsc_merge_weights: clone_p_haproxy: Rolling back scores from vip__public
  +
May 10 02:53:51 [9164] node-X.default.ltd pengine: info: native_color: Resource p_haproxy:0 cannot run anywhere
  +
May 10 02:53:51 [9164] node-X.default.ltd pengine: info: native_color: Resource vip__management cannot run anywhere
  +
May 10 02:53:51 [9164] node-X.default.ltd pengine: info: rsc_merge_weights: vip__vrouter_pub: Rolling back scores from master_p_conntrackd
  +
May 10 02:53:51 [9164] node-X.default.ltd pengine: info: rsc_merge_weights: vip__vrouter_pub: Rolling back scores from vip__vrouter
  +
May 10 02:53:51 [9164] node-X.default.ltd pengine: info: native_color: Resource vip__vrouter_pub cannot run anywhere
  +
May 10 02:53:51 [9164] node-X.default.ltd pengine: info: native_color: Resource vip__vrouter cannot run anywhere
  +
May 10 02:53:51 [9164] node-X.default.ltd pengine: info: native_color: Resource vip__public cannot run anywhere
  +
May 10 02:53:51 [9164] node-X.default.ltd pengine: info: native_color: Resource p_mysql:0 cannot run anywhere
  +
May 10 02:53:51 [9164] node-X.default.ltd pengine: info: native_color: Resource p_rabbitmq-server:0 cannot run anywhere
  +
May 10 02:53:51 [9164] node-X.default.ltd pengine: info: master_color: master_p_rabbitmq-server: Promoted 0 instances of a possible 1 to master
  +
May 10 02:53:51 [9164] node-X.default.ltd pengine: notice: clone_rsc_colocation_rh: Cannot pair p_dns:0 with instance of clone_p_vrouter
  +
May 10 02:53:51 [9164] node-X.default.ltd pengine: info: native_color: Resource p_dns:0 cannot run anywhere
  +
May 10 02:53:51 [9164] node-X.default.ltd pengine: info: native_color: p_conntrackd:0: Rolling back scores from vip__vrouter_pub
  +
May 10 02:53:51 [9164] node-X.default.ltd pengine: info: native_color: Resource p_conntrackd:0 cannot run anywhere
  +
May 10 02:53:51 [9164] node-X.default.ltd pengine: info: master_color: master_p_conntrackd: Promoted 0 instances of a possible 1 to master
  +
May 10 02:53:51 [9164] node-X.default.ltd pengine: info: native_color: Resource p_heat-engine:0 cannot run anywhere
  +
May 10 02:53:51 [9164] node-X.default.ltd pengine: info: native_color: Resource p_neutron-plugin-openvswitch-agent:0 cannot run anywhere
  +
May 10 02:53:51 [9164] node-X.default.ltd pengine: info: native_color: Resource p_neutron-l3-agent:0 cannot run anywhere
  +
May 10 02:53:51 [9164] node-X.default.ltd pengine: info: native_color: Resource p_neutron-dhcp-agent:0 cannot run anywhere
  +
May 10 02:53:51 [9164] node-X.default.ltd pengine: info: native_color: Resource p_neutron-metadata-agent:0 cannot run anywhere
  +
May 10 02:53:51 [9164] node-X.default.ltd pengine: info: native_color: Resource ping_vip__public:0 cannot run anywhere
  +
May 10 02:53:51 [9164] node-X.default.ltd pengine: info: native_color: Resource p_ntp:0 cannot run anywhere
  +
May 10 02:53:51 [9164] node-X.default.ltd pengine: info: LogActions: Leave sysinfo_node-X.default.ltd (Stopped)
  +
May 10 02:53:51 [9164] node-X.default.ltd pengine: info: LogActions: Leave p_vrouter:0 (Stopped)
  +
May 10 02:53:51 [9164] node-X.default.ltd pengine: info: LogActions: Leave vip__management (Stopped)
  +
May 10 02:53:51 [9164] node-X.default.ltd pengine: info: LogActions: Leave vip__vrouter_pub (Stopped)
  +
May 10 02:53:51 [9164] node-X.default.ltd pengine: info: LogActions: Leave vip__vrouter (Stopped)
  +
May 10 02:53:51 [9164] node-X.default.ltd pengine: info: LogActions: Leave vip__public (Stopped)
  +
May 10 02:53:51 [9164] node-X.default.ltd pengine: info: LogActions: Leave p_haproxy:0 (Stopped)
  +
May 10 02:53:51 [9164] node-X.default.ltd pengine: info: LogActions: Leave p_mysql:0 (Stopped)
  +
May 10 02:53:51 [9164] node-X.default.ltd pengine: info: LogActions: Leave p_rabbitmq-server:0 (Stopped)
  +
May 10 02:53:51 [9164] node-X.default.ltd pengine: info: LogActions: Leave p_dns:0 (Stopped)
  +
May 10 02:53:51 [9164] node-X.default.ltd pengine: info: LogActions: Leave p_conntrackd:0 (Stopped)
  +
May 10 02:53:51 [9164] node-X.default.ltd pengine: info: LogActions: Leave p_heat-engine:0 (Stopped)
  +
May 10 02:53:51 [9164] node-X.default.ltd pengine: info: LogActions: Leave p_neutron-plugin-openvswitch-agent:0 (Stopped)
  +
May 10 02:53:51 [9164] node-X.default.ltd pengine: info: LogActions: Leave p_neutron-l3-agent:0 (Stopped)
  +
May 10 02:53:51 [9164] node-X.default.ltd pengine: info: LogActions: Leave p_neutron-dhcp-agent:0 (Stopped)
  +
May 10 02:53:51 [9164] node-X.default.ltd pengine: info: LogActions: Leave p_neutron-metadata-agent:0 (Stopped)
  +
May 10 02:53:51 [9164] node-X.default.ltd pengine: info: LogActions: Leave ping_vip__public:0 (Stopped)
  +
May 10 02:53:51 [9164] node-X.default.ltd pengine: info: LogActions: Leave p_ntp:0 (Stopped)
  +
</PRE>
  +
}}
  +
  +
==Как решать?==
  +
Проверить место
  +
<PRE>
  +
# df -h / /var/log /var/lib/mysql
  +
</PRE>
  +
Обычно проблемные
  +
* /var/log
  +
* /var/lib/mysql
  +
После того как место очищено - рестарт
  +
<PRE>
  +
# service pacemaker stop
  +
# service pacemaker start
  +
</PRE>
  +
==Проверка настроек==
  +
<PRE>
  +
# crm configure show cib-bootstrap-options
  +
property cib-bootstrap-options: \
  +
dc-version=1.1.12-561c4cf \
  +
cluster-infrastructure=corosync \
  +
no-quorum-policy=stop \
  +
cluster-recheck-interval=190s \
  +
stonith-enabled=false \
  +
start-failure-is-fatal=false \
  +
symmetric-cluster=false \
  +
last-lrm-refresh=1461091450 \
  +
node-health-strategy=migrate-on-red
  +
</PRE>
  +
Тут обратить внимание на <code>node-health-strategy=migrate-on-red</code>
  +
<BR>
  +
есть примитив Pacemaker/Corosync который мониторит "живость" ноды, его можно посмотреть так:
  +
<PRE>
  +
# crm configure show sysinfo_*
  +
primitive sysinfo_node-1.default.tld ocf:pacemaker:SysInfo \
  +
op monitor interval=15s \
  +
params disks="/ /var/log /var/lib/mysql" min_disk_free=512M disk_unit=M
  +
</PRE>
  +
* <code>/</code>
  +
* <code>/var/log</code>
  +
* <code>/var/lib/mysql</code>
  +
  +
=Проверка статуса репликации Galera=
  +
<PRE>
  +
SHOW GLOBAL STATUS LIKE 'wsrep_%';
  +
</PRE>
  +
  +
=Если кластер "не собирается" - поднять таймауты=
  +
  +
ProblemSymptoms of the problem being solved, objectives of the procedure or additional details about the question being asked that summarize what the customer is experiencing.
  +
MySQL is not running on one of the controller nodes
  +
<PRE>
  +
# pcs status
  +
Clone Set: clone_p_mysql [p_mysql]
  +
Started: [ b05-39-controller.domain.tld b06-39-controller.domain.tld ]
  +
Stopped: [ b05-38-controller.domain.tld ]
  +
</PRE>
  +
Make sure MySQL is stopped on problematic node
  +
<PRE>
  +
# ps -ef | grep mysql
  +
root 14878 5566 0 14:33 pts/0 00:00:00 grep --color=auto mysql
  +
</PRE>
  +
Edit the default timeout for start operation
  +
<PRE>
  +
# crm configure edit p_mysql
  +
</PRE>
  +
или
  +
<PRE>
  +
# crm configure edit p_mysqld
  +
</PRE>
  +
Set the temporary timeout value for p_mysql-start-0 to be 1200
  +
<PRE>
  +
# pcs resource show p_mysql
  +
Resource: p_mysql (class=ocf provider=fuel type=mysql-wss)
  +
Attributes: test_user=wsrep_sst test_passwd=??? socket=/var/run/mysqld/mysqld.sock
  +
Operations: monitor interval=60 timeout=55 (p_mysql-monitor-60)
  +
start interval=0 timeout=1200 (p_mysql-start-0)
  +
stop interval=0 timeout=120 (p_mysql-stop-0)
  +
</PRE>
  +
Cleanup the p_mysql resource
  +
<PRE>
  +
# crm resource cleanup p_mysql
  +
</PRE>
  +
Wait for 15-20 minutes for synchronization to be completed
  +
Run pcs status again to check mysql is back up and running on the problematic controller node
  +
<PRE>
  +
# pcs status
  +
Clone Set: clone_p_mysql [p_mysql]
  +
Started: [ b05-38-controller.domain.tld b05-39-controller.domain.tld b06-39-controller.domain.tld ]
  +
Ensure that the cluster is synced again
  +
| wsrep_local_state_comment | Synced |
  +
| wsrep_cert_index_size | 899 |
  +
| wsrep_causal_reads | 0 |
  +
| wsrep_incoming_addresses | 10.128.0.133:3307,10.128.0.132:3307,10.128.0.134:3307
  +
</PRE>
  +
Reset the timeout value for p_mysql-start-0 to default value.
  +
  +
=Ходовые команды CRM/PCS=
  +
==CRM==
  +
  +
{| class="wikitable"
  +
|+ The most frequently used CRM commands
  +
|-
  +
! Action !! Command !! Example
  +
|-
  +
|Show cluster status
  +
||crm status
  +
||crm status
  +
|-
  +
|Show status of resource
  +
||crm resource status
  +
||crm resource status
  +
|-
  +
|Cleanup resource status
  +
||crm resource cleanup <resource>
  +
||crm resource cleanup p_neutron-dhcp-agent
  +
|-
  +
|Start a resource
  +
||crm resource start <resource>
  +
||crm resource start p_neutron-dhcp-agent
  +
|-
  +
|Stop a resource
  +
||crm resource stop <resource>
  +
||crm resource stop p_neutron-dhcp-agent
  +
|-
  +
|Restart a resource
  +
||crm resource restart <resource>
  +
||crm resource restart p_neutron-dhcp-agent
  +
|-
  +
|Put a resource into manage mode
  +
||crm resource manage <resource>
  +
||crm resource manage p_neutron-dhcp-agent
  +
|-
  +
|Put a resource into unmanage mode
  +
||crm resource unmanage <resource>
  +
||crm resource unmanage p_neutron-dhcp-agent
  +
|-
  +
|Migrate a resource to another node
  +
||crm resource migrate <resource> [<node>] [<lifetime>] [force]
  +
||crm resource migrate p_neutron-dhcp-agent
  +
|}
  +
  +
==PCS==
  +
  +
{| class="wikitable"
  +
|+ The frequently used PCS commands
  +
|-
  +
! Action !! Command !! Example
  +
|-
  +
|Show cluster status
  +
||pcs status
  +
||pcs status
  +
|-
  +
|Show status of resource
  +
||pcs status resources
  +
||pcs status resources
  +
|-
  +
|Cleanup resource status
  +
||pcs resource cleanup resource_id
  +
||pcs resource cleanup p_neutron-plugin-openvswitch-agent
  +
|-
  +
|Put a resource into manage mode
  +
||pcs manage <resource id> ... [resource n]
  +
||
  +
|-
  +
|Put a resource into unmanage mode
  +
||pcs unmanage <resource id> ... [resource n]
  +
||
  +
|-
  +
|Saves the raw xml from the CIB into a file
  +
||pcs cluster cib
  +
||pcs cluster cib
  +
|-
  +
|Display full cluster config
  +
||pcs config
  +
||pcs config
  +
|-
  +
|Stopping Cluster Services / Force stop
  +
||
  +
* pcs cluster stop [--all] [node] [...]
  +
* pcs cluster kill
  +
||pcs cluster stop node-1
  +
|-
  +
|Standby Mode
  +
The specified node is no longer able to host resources
  +
||
  +
* pcs cluster standby node | --all
  +
* pcs cluster unstandby node | --all
  +
||
  +
* pcs cluster standby node-1
  +
* pcs cluster unstandby node-1
  +
|-
  +
|Enabling Cluster Resources
  +
||pcs resource enable resource_id [--wait[=n]]
  +
||pcs resource enable p_neutron-l3-agent
  +
|-
  +
|Disabling Cluster Resources
  +
||pcs resource disable resource_id [--wait[=n]]
  +
||pcs resource disable p_neutron-l3-agent
  +
|-
  +
|Prevent the resource id specified from running on the node
  +
||pcs resource ban <resource id> [node] [--master]
  +
||pcs resource ban p_neutron-plugin-openvswitch-agent node-1
  +
|-
  +
|Remove constraints created by move and/or ban on the specified resource
  +
||pcs resource clear <resource id> [node] [--master]
  +
||pcs resource clear p_neutron-plugin-openvswitch-agent node-1
  +
|-
  +
|Maintenance Mode tells the cluster to go to a "hands off" mode
  +
||pcs property set maintenance-mode=[true|false]
  +
||
  +
* pcs property set maintenance-mode=true
  +
* pcs property set maintenance-mode=false
  +
|-
  +
|Move resource off current node (and optionally onto destination node)
  +
||pcs move <resource id> [destination node] [--master]
  +
||pcs resource move p_neutron-dhcp-agent node-1
  +
|-
  +
|Force the specified resource to start on this node ignoring the cluster
  +
recommendations and print the output from starting the resource
  +
||pcs debug-start <resource id> [--full]
  +
||pcs resource debug-start p_ceilometer-agent-central
  +
|-
  +
|Show options specific resource
  +
||pcs resource show <resource id>
  +
||pcs resource show p_neutron-l3-agent
  +
|-
  +
|Show current failcount for specified resource
  +
||pcs failcount show <resource id> [node]
  +
||pcs resource failcount show p_neutron-l3-agent
  +
|-
  +
|Restart metadata-agent on a specific node
  +
||pcs resource ban p_neutron-metadata-agent node-1.domain.tld
  +
||pcs resource clear p_neutron-metadata-agent node-1.domain.tld
  +
|}
  +
  +
=Еще один документ по восстановлению кластера=
  +
  +
  +
This is brief document like step-by-step how to for restoration in case 2 of 3 controllers are down (hardware malfunction, power outage, etc). There is no quorum on pacemaker. Cloud is down.
  +
current state:
  +
* 1 controller up or 1 controller healthy.
  +
* 2 controllers down or not available or not functional
  +
Choose 1st controller – better will be controller with MySQL up and running. But on most cases, with such failures mysql is down on all nodes.
  +
node–1 – alive controller (mysql database should not be corrupted) we will back up this node
  +
node–2 – down or malfunction controller
  +
node–3 – down or malfunction controller
  +
Recover cloud on the one controller
  +
  +
For restoring node–1 as a single controller we need:
  +
* Fix pacemaker quorum policy
  +
* Fix MySQL server
  +
* * Fix RabbitMQ service
  +
==Fix other core services (Neutron, Nova, heat, etc)==
  +
===Pacemaker===
  +
  +
Switch node–2 and node–3 to maintenance/unmanage state. just for make sure they will not interrupt restoring process.
  +
* <code>$ crm node maintenance node-2</code>
  +
* $ <code>crm node maintenance node-3</code>
  +
Check:
  +
<PRE>
  +
$ pcs cluster cib |grep maint
  +
<nvpair id="nodes-1-maintenance" name="maintenance" value="off"/>
  +
<nvpair id="nodes-2-maintenance" name="maintenance" value="on"/>
  +
<nvpair id="nodes-3-maintenance" name="maintenance" value="on"/>
  +
</PRE>
  +
<PRE>
  +
$ pcs status
  +
</PRE>
  +
Disable mysql on node–1, if that node is not donor from pacemaker prospective we need manually start it as donor. Also we need disable zabbix (in case it installed). High load on MySQL can caused problem during replication or even break replication process.
  +
  +
* <code>$ pcs resource disable clone_p_mysql</code>
  +
* <code>$ pcs resource disable p_zabbix-server</code>
  +
* <code>$ pcs status</code>
  +
Switch no-quorum-policy to ignore. We need that for start all pacemaker’s services without quorum. Usually, this step is restoring RabbitMQ.
  +
  +
* <code>$ crm configure property no-quorum-policy=ignore</code>
  +
* <code>$ crm configure show</code>
  +
* <code>$ pcs cluster cib |grep no-quorum</code>
  +
* <code>$ pcs status</code>
  +
===RabbitMQ===
  +
  +
Restart rabbitmq messaging service. In some cases it is mandatory, for make sure that service is up, running and operational.
  +
<BR>
  +
Stop p_rabbitmq-server on pacemaker
  +
<BR>
  +
<PRE>
  +
$ crm resource stop p_rabbitmq-server
  +
</PRE>
  +
Ensure that rabbitmq down and kill it if necessary. If you are using murano please do not kill murano rabbit: kill only rabbit with line like 'pluginsexpanddir "/var/lib/rabbitmq/mnesia/rabbit@node-1-plugins-expand".
  +
<PRE>
  +
$ ps -ef|grep rabbit
  +
</PRE>
  +
Start p_rabbitmq-server by pacemaker
  +
* <code>$ crm resource start p_rabbitmq-server</code>
  +
* <code>$ pcs status</code>
  +
Ensure that rabbitmq running
  +
* <code>$ rabbitmqctl status</code>
  +
* <code>$ rabbitmqctl cluster_status</code>
  +
  +
===MySQL===
  +
  +
Ensure that there is no MySQL is running and kill it if necessary.
  +
<PRE>
  +
$ ps -ef|grep mysql
  +
</PRE>
  +
Start MySQL as donor for galera cluster.
  +
<PRE>
  +
$ export OCF_RESOURCE_INSTANCE=p_mysql
  +
$ export OCF_ROOT=/usr/lib/ocf
  +
$ export OCF_RESKEY_socket=/var/run/mysqld/mysqld.sock
  +
$ export OCF_RESKEY_additional_parameters="--wsrep-new-cluster"
  +
$ /usr/lib/ocf/resource.d/fuel/mysql-wss start
  +
</code>
  +
Check mysql in a couple of minutes
  +
<PRE>
  +
$ mysql -e "show status like 'wsrep_%';"
  +
</PRE>
  +
Create mysql backup just in case (if necessary)
  +
<PRE>
  +
$ mkdir /tmp/db-backup; \
  +
innobackupex --no-timestamp --socket=/var/run/mysqld/mysqld.sock /tmp/db-backup; \
  +
innobackupex --use-memory=1G --lock-wait-query-type=all --apply-log --socket=/var/run/mysqld/mysqld.sock /tmp/db-backup
  +
</PRE>
  +
You can specify just some special list of tables, Excluding zabbix. please refer to Percona Partial Backups. For example backup without zabbix we will have:
  +
<PRE>
  +
$ mysql -e "SELECT CONCAT(table_schema,'.',table_name) FROM information_schema.tables WHERE table_schema not like 'zabbix'; " > db-tables.txt
  +
$ innobackupex --tables-file=db-tables.txt --socket=/var/run/mysqld/mysqld.sock /tmp/db-backup
  +
</PRE>
  +
  +
===Ceph===
  +
  +
We have one controller with ceph monitor. There is no quorum, and ceph is not operational. We need remove 2 other monitors from down controllers.
  +
<BR>
  +
Stop ceph monitor:
  +
<PRE>
  +
$ stop ceph-mon-all
  +
</PRE>
  +
Dump monitor map:
  +
* <code> $ mkdir ./ceph-mon-dump </code>
  +
* <code> $ ceph-mon -i node-1 --extract-monmap ./ceph-mon-dump/monmap </code>
  +
* <code> $ cp ./ceph-mon-dump/monmap ./ceph-mon-dump/monmap.bak </code>
  +
Remove node–2 and node–3 from monitor-map
  +
* <code> $ monmaptool ./ceph-mon-dump/monmap --rm node-2 </code>
  +
* <code> $ monmaptool ./ceph-mon-dump/monmap --rm node-3 </code>
  +
Inject updated monitor map with one monitor:
  +
* <code> $ ceph-mon -i node-1 --inject-monmap ./ceph-mon-dump/monmap </code>
  +
Starting and checking ceph
  +
* <code> $ start ceph-mon-all </code>
  +
* <code> $ ceph -s </code>
  +
Restart ceph osd on ceph-osd nodes if necessary
  +
  +
===Neutron and nova services===
  +
  +
Now we have MySQL, RabbitMQ and ceph running.
  +
<BR>
  +
Check and restore pacemaker services on controller, and neutron and nova services on computes.
  +
* <code> $ pcs status </code>
  +
* <code> $ . openrc </code>
  +
* <code> $ nova service-list </code>
  +
* <code> $ neutron agent-list </code>
  +
* <code> $ neutron agent-list |grep -i xxx </code>
  +
After that cloud should be operational.
  +
  +
==Node–2 restoration==
  +
  +
Now we have:
  +
<BR>
  +
cloud – operational, up and running.
  +
  +
* node–1 – Single controller. up and running.
  +
* node–2 – down or malfunction controller. let’s start this node.
  +
* node–3 – down or malfunction controller
  +
<BR>
  +
Start node–2, if it down. <BR>
  +
For restoring node 2 we have the same sequence, except rabbitmq: adding second node to the rabbitMQ cluster by pacemaker causes rabbitMQ service interruption for a 1–5 min. Node–2 still unmanageable and marked as “maintenance” in the pacemaker.
  +
  +
===Restoring MySQL on node–2===
  +
  +
ssh to node–2.
  +
<BR>
  +
On node–2:
  +
<BR>
  +
Stop mysql by pacemaker script if running:
  +
* <code> $ export OCF_RESOURCE_INSTANCE=p_mysql </code>
  +
* <code> $ export OCF_ROOT=/usr/lib/ocf </code>
  +
* <code> $ export OCF_RESKEY_socket=/var/run/mysqld/mysqld.sock </code>
  +
* <code> $ /usr/lib/ocf/resource.d/fuel/mysql-wss stop </code>
  +
Ensure that mysql is down and no mysql process exist. This is mandatory. Kill the MySQL process if it still running or stucked.
  +
<BR>
  +
Clean up the MySQL directory and run MySQL by pacemaker.
  +
* <code> $ rm -R /var/lib/mysql/* </code>
  +
* <code> $ /usr/lib/ocf/resource.d/fuel/mysql-wss start </code>
  +
  +
In a 5 minutes or so replication started. <BR>
  +
In a some time depends of the data amount and network speed replication completed and mysql started. <BR>
  +
Check the /var/log/mysqld.log for details. Check mysql cluster status after mysql started as galera cluster member.
  +
<PRE>
  +
$ mysql -e "show status like 'wsrep_%';"
  +
</PRE>
  +
Return back to node–1.
  +
  +
===Adding node–2 to pacemaker cluster===
  +
  +
Now we need restore pacemaker cluster management for node–2.<BR>
  +
Please make sure that we will not start rabbitMQ on the node.
  +
  +
* <code> $ pcs resource ban p_rabbitmq-server node-2 </code>
  +
* <code> $ crm node ready node-2 </code>
  +
* <code> $ pcs status </code>
  +
  +
===Adding node–2 to the ceph map===
  +
  +
Stop ceph monitor on node–2
  +
<PRE>
  +
$ ssh node-2 stop ceph-mon-all </code>
  +
</PRE>
  +
re-add monitor on node-2 node.
  +
* <code> $ ceph mon remove node-2 </code>
  +
* <code> $ ceph mon add node-2 <ip addr of node-2 from management network> </code>
  +
* <code> $ ceph-deploy --overwrite-conf mon create node-2 </code>
  +
Check ceph status
  +
<PRE>$ ceph -s</PRE>
  +
As result you will set two monitors in the ceph cluster.
  +
  +
===Nova and neutron services.===
  +
  +
Check and restore (if necessary) nova and neutron services on node–2
  +
* <code> $ nova service-list </code>
  +
* <code> $ neutron agent-list </code>
  +
* <code> $ neutron agent-list |grep -i xxx </code>
  +
  +
==Node–3 restoration==
  +
  +
Now we have:<BR>
  +
cloud – operational, up and running.
  +
* node–1 – 1st controller. up and running.
  +
* node–2 – 2nd controller. up and running.
  +
* node–3 – down or malfunction controller.
  +
Start node–3, if it down.
  +
  +
Restoring node–3 is similar to restoting node–2, with the same exception for rabbitmq:<BR>
  +
adding second node to the rabbitMQ cluster by pacemaker causes rabbitMQ service interruption for a 1–5 min.
  +
<BR>
  +
Node–3 still is unmanageable and marked as “maintenance” in the pacemaker.
  +
Perform all steps similar to restore node-2
  +
  +
===Restoring quorum policy===
  +
  +
Restore quorum policy
  +
<PRE>
  +
$ crm configure property no-quorum-policy=stop
  +
</PRE>
  +
Check pacemaker config and status
  +
* <code>$ crm configure show </code>
  +
* <code>$ pcs cluster cib |grep no-quorum </code>
  +
* <code>$ pcs status </code>
  +
  +
==Post-recovering tasks.==
  +
  +
This is task assigned to the maintenances because some service interruption expected.
  +
  +
===Restoring mysql managing by pacemaker===
  +
  +
This step is not cause mysql restart. It is safe to do during business time, but we recommend shift this step to the maintenance.
  +
* <code> $ pcs resource clear p_mysql node-2 </code>
  +
* <code> $ pcs resource clear p_mysql node-3 </code>
  +
* <code> $ pcs resource enable clone_p_mysql </code>
  +
* <code> $ pcs status </code>
  +
  +
===Starting zabbix monitoring service.===
  +
  +
Start zabbix service
  +
* <code> $ pcs resource enable p_zabbix-server </code>
  +
In 2-5 minutes please check zabbix status in the pacemaker dashboard
  +
* <code> $ pcs status </code>
  +
  +
===Restoring rabbitmq cluster managing by pacemaker.===
  +
Service interruption for 1–5 min.
  +
  +
Return to pacemaker managing of rabbitmq
  +
* <code> $ pcs resource clear p_rabbitmq-server node-2 </code>
  +
* <code> $ pcs resource clear p_rabbitmq-server node-3 </code>
  +
In a 3-5 minutes check rabbitmq cluster status by rabbitmqctl:
  +
* <code> $ rabbitmqctl cluster_status </code>
  +
Check by pacemaker:
  +
* <code> $ pcs status p_rabbitmq-server </code>
  +
Ensure that cluster name is the same on master node from pacemaker prospective.

Текущая версия на 12:46, 8 февраля 2024

Восстановление Galera запущенной под PCS

Это заметка о том как восстанавливать разваленный кластер, что бы не потерять если еще раз понадобится.

На любом контроллере (в этом вом вопросе все контроллеры равнозначны): 

pcs resource disable clone_p_mysqld

Дождаться пока все процессы mysqld будут остановлены

Убедиться что ресурс clone_p_mysql остановлен на всех контроллерах: 

pcs status resources

На всех контроллерах зачистить (перенести или удалить) данные mysql: 

mv /var/lib/mysql/* /tmp/mysql/

Выбрать один контроллер как основной для восстановления (это может быть любой контроллер), далее называем его controller-x

Восстановить бекап (это конечно хорошо, если он есть) на controller-x: 

cp -R /ext-volume/mysql-backup/* /var/lib/mysql/

Поправить права на controller-x: 

chown -R mysql:mysql /var/lib/mysql

Для OCF скриптов нужны переменные окружения, их нужно установить перед запуском скрипта mysql-wss
После чего стартовать базу на выбранном контроллере controller-x.
Обратить внимание что первый (и только первый) контроллер запускается с параметром --wsrep-new-cluster 

export OCF_RESOURCE_INSTANCE=p_mysqld
export OCF_ROOT=/usr/lib/ocf
export OCF_RESKEY_socket=/var/run/mysqld/mysqld.sock
export OCF_RESKEY_master_timeout=10
export OCF_RESKEY_test_passwd=`crm_resource -r p_mysqld -g test_passwd`
export OCF_RESKEY_test_user=`crm_resource -r p_mysqld -g test_user`
export OCF_RESKEY_additional_parameters="--wsrep-new-cluster"
/usr/lib/ocf/resource.d/fuel/mysql-wss start

Запустить monitor operation на controller-x to update Galera GTID in Pacemaker cluster configuration,
Тоже нужны переменные окружения 

/usr/lib/ocf/resource.d/fuel/mysql-wss monitor

Запустить на остальных 2-х контроллерах: 

export OCF_RESOURCE_INSTANCE=p_mysqld
export OCF_ROOT=/usr/lib/ocf
export OCF_RESKEY_socket=/var/run/mysqld/mysqld.sock
export OCF_RESKEY_master_timeout=10
export OCF_RESKEY_test_passwd=`crm_resource -r p_mysqld -g test_passwd`
export OCF_RESKEY_test_user=`crm_resource -r p_mysqld -g test_user`
/usr/lib/ocf/resource.d/fuel/mysql-wss start

С любого контроллера включить ресурс mysql в Pacemaker: 

pcs resource enable clone_p_mysqld

Убедиться что ресурс стартовал на всех контроллерах: 

pcs status resources


https://clusterlabs.org/pacemaker/doc/2.1/Pacemaker_Explained/html/nodes.html#tracking-node-health

Что делать если все упало из-за занятого места на дисках

Как проверить?

Узнать что есть проблемы с диском с точки зрения Pacemaker можно из лога /var/log/pacemaker.log 

# grep 'health_disk.*value="red"' /var/log/pacemaker.log
May 10 02:53:21 [9160] node-X.default.ltd        cib:     info: cib_perform_op:     ++ /cib/status/node_state[@id='X']/transient_attributes[@id='X']/instance_attributes[@id='status-X']:  <nvpair id="status-X-#health_disk" name="#health_disk" value="red"/>

​After the health disk change you will see a health strategy: "migrate-on-red" 

May 10 02:53:21 [9164] node-X.default.ltd    pengine:     info: apply_system_health: 	Applying automated node health strategy: migrate-on-red

После чего какие-то ресурсы не могут стартовать 0 обращать внимание на

  • Applying automated node health strategy: migrate-on-red
  • Resource p_haproxy:0 cannot run anywhere

Как решать?

Проверить место 

# df -h / /var/log /var/lib/mysql

Обычно проблемные

  • /var/log
  • /var/lib/mysql

После того как место очищено - рестарт 

# service pacemaker stop 
# service pacemaker start

Проверка настроек

# crm configure show cib-bootstrap-options
property cib-bootstrap-options: \
	dc-version=1.1.12-561c4cf \
	cluster-infrastructure=corosync \
	no-quorum-policy=stop \
	cluster-recheck-interval=190s \
	stonith-enabled=false \
	start-failure-is-fatal=false \
	symmetric-cluster=false \
	last-lrm-refresh=1461091450 \
	node-health-strategy=migrate-on-red

Тут обратить внимание на node-health-strategy=migrate-on-red
есть примитив Pacemaker/Corosync который мониторит "живость" ноды, его можно посмотреть так: 

# crm configure show sysinfo_*
primitive sysinfo_node-1.default.tld ocf:pacemaker:SysInfo \
	op monitor interval=15s \
	params disks="/ /var/log /var/lib/mysql" min_disk_free=512M disk_unit=M
  • /
  • /var/log
  • /var/lib/mysql

Проверка статуса репликации Galera

SHOW GLOBAL STATUS LIKE 'wsrep_%';

Если кластер "не собирается" - поднять таймауты

ProblemSymptoms of the problem being solved, objectives of the procedure or additional details about the question being asked that summarize what the customer is experiencing. MySQL is not running on one of the controller nodes 

# pcs status
Clone Set: clone_p_mysql [p_mysql]
    Started: [ b05-39-controller.domain.tld b06-39-controller.domain.tld ]
    Stopped: [ b05-38-controller.domain.tld ]

Make sure MySQL is stopped on problematic node 

# ps -ef | grep mysql
root     14878  5566  0 14:33 pts/0    00:00:00 grep --color=auto mysql

Edit the default timeout for start operation 

# crm configure edit p_mysql

или 

# crm configure edit p_mysqld

Set the temporary timeout value for p_mysql-start-0 to be 1200 

# pcs resource show p_mysql
   Resource: p_mysql (class=ocf provider=fuel type=mysql-wss)
    Attributes: test_user=wsrep_sst test_passwd=??? socket=/var/run/mysqld/mysqld.sock 
    Operations: monitor interval=60 timeout=55 (p_mysql-monitor-60)
                start interval=0 timeout=1200 (p_mysql-start-0)
                stop interval=0 timeout=120 (p_mysql-stop-0)

Cleanup the p_mysql resource 

# crm resource cleanup p_mysql

Wait for 15-20 minutes for synchronization to be completed Run pcs status again to check mysql is back up and running on the problematic controller node 

# pcs status
Clone Set: clone_p_mysql [p_mysql]
    Started: [ b05-38-controller.domain.tld b05-39-controller.domain.tld b06-39-controller.domain.tld ]
Ensure that the cluster is synced again
| wsrep_local_state_comment  | Synced                                                |
| wsrep_cert_index_size      | 899                                                   |
| wsrep_causal_reads         | 0                                                     |
| wsrep_incoming_addresses   | 10.128.0.133:3307,10.128.0.132:3307,10.128.0.134:3307

Reset the timeout value for p_mysql-start-0 to default value.

Ходовые команды CRM/PCS

CRM

The most frequently used CRM commands
Action Command Example
Show cluster status crm status crm status
Show status of resource crm resource status crm resource status
Cleanup resource status crm resource cleanup <resource> crm resource cleanup p_neutron-dhcp-agent
Start a resource crm resource start <resource> crm resource start p_neutron-dhcp-agent
Stop a resource crm resource stop <resource> crm resource stop p_neutron-dhcp-agent
Restart a resource crm resource restart <resource> crm resource restart p_neutron-dhcp-agent
Put a resource into manage mode crm resource manage <resource> crm resource manage p_neutron-dhcp-agent
Put a resource into unmanage mode crm resource unmanage <resource> crm resource unmanage p_neutron-dhcp-agent
Migrate a resource to another node crm resource migrate <resource> [<node>] [<lifetime>] [force] crm resource migrate p_neutron-dhcp-agent

PCS

The frequently used PCS commands
Action Command Example
Show cluster status pcs status pcs status
Show status of resource pcs status resources pcs status resources
Cleanup resource status pcs resource cleanup resource_id pcs resource cleanup p_neutron-plugin-openvswitch-agent
Put a resource into manage mode pcs manage <resource id> ... [resource n]
Put a resource into unmanage mode pcs unmanage <resource id> ... [resource n]
Saves the raw xml from the CIB into a file pcs cluster cib pcs cluster cib
Display full cluster config pcs config pcs config
Stopping Cluster Services / Force stop
  • pcs cluster stop [--all] [node] [...]
  • pcs cluster kill
 pcs cluster stop node-1
Standby Mode

The specified node is no longer able to host resources

  • pcs cluster standby node | --all
  • pcs cluster unstandby node | --all
  • pcs cluster standby node-1
  • pcs cluster unstandby node-1
Enabling Cluster Resources pcs resource enable resource_id [--wait[=n]] pcs resource enable p_neutron-l3-agent
Disabling Cluster Resources pcs resource disable resource_id [--wait[=n]] pcs resource disable p_neutron-l3-agent
Prevent the resource id specified from running on the node pcs resource ban <resource id> [node] [--master] pcs resource ban p_neutron-plugin-openvswitch-agent node-1
Remove constraints created by move and/or ban on the specified resource pcs resource clear <resource id> [node] [--master] pcs resource clear p_neutron-plugin-openvswitch-agent node-1
Maintenance Mode tells the cluster to go to a "hands off" mode pcs property set maintenance-mode=[true|false]
  • pcs property set maintenance-mode=true
  • pcs property set maintenance-mode=false
Move resource off current node (and optionally onto destination node) pcs move <resource id> [destination node] [--master] pcs resource move p_neutron-dhcp-agent node-1
Force the specified resource to start on this node ignoring the cluster

recommendations and print the output from starting the resource

pcs debug-start <resource id> [--full] pcs resource debug-start p_ceilometer-agent-central
Show options specific resource pcs resource show <resource id> pcs resource show p_neutron-l3-agent
Show current failcount for specified resource pcs failcount show <resource id> [node] pcs resource failcount show p_neutron-l3-agent
Restart metadata-agent on a specific node pcs resource ban p_neutron-metadata-agent node-1.domain.tld pcs resource clear p_neutron-metadata-agent node-1.domain.tld

Еще один документ по восстановлению кластера

This is brief document like step-by-step how to for restoration in case 2 of 3 controllers are down (hardware malfunction, power outage, etc). There is no quorum on pacemaker. Cloud is down. current state:

  • 1 controller up or 1 controller healthy.
  • 2 controllers down or not available or not functional

Choose 1st controller – better will be controller with MySQL up and running. But on most cases, with such failures mysql is down on all nodes. node–1 – alive controller (mysql database should not be corrupted) we will back up this node node–2 – down or malfunction controller node–3 – down or malfunction controller Recover cloud on the one controller

For restoring node–1 as a single controller we need:

  • Fix pacemaker quorum policy
  • Fix MySQL server
  • * Fix RabbitMQ service

Fix other core services (Neutron, Nova, heat, etc)

Pacemaker

Switch node–2 and node–3 to maintenance/unmanage state. just for make sure they will not interrupt restoring process.

  • $ crm node maintenance node-2
  • $ crm node maintenance node-3

Check: 

$ pcs cluster cib |grep maint
<nvpair id="nodes-1-maintenance" name="maintenance" value="off"/>
<nvpair id="nodes-2-maintenance" name="maintenance" value="on"/>
<nvpair id="nodes-3-maintenance" name="maintenance" value="on"/>

$ pcs status

Disable mysql on node–1, if that node is not donor from pacemaker prospective we need manually start it as donor. Also we need disable zabbix (in case it installed). High load on MySQL can caused problem during replication or even break replication process.

  • $ pcs resource disable clone_p_mysql
  • $ pcs resource disable p_zabbix-server
  • $ pcs status

Switch no-quorum-policy to ignore. We need that for start all pacemaker’s services without quorum. Usually, this step is restoring RabbitMQ.

  • $ crm configure property no-quorum-policy=ignore
  • $ crm configure show
  • $ pcs cluster cib |grep no-quorum
  • $ pcs status

RabbitMQ

Restart rabbitmq messaging service. In some cases it is mandatory, for make sure that service is up, running and operational.
Stop p_rabbitmq-server on pacemaker 

$ crm resource  stop p_rabbitmq-server

Ensure that rabbitmq down and kill it if necessary. If you are using murano please do not kill murano rabbit: kill only rabbit with line like 'pluginsexpanddir "/var/lib/rabbitmq/mnesia/rabbit@node-1-plugins-expand". 

$ ps -ef|grep rabbit

Start p_rabbitmq-server by pacemaker

  • $ crm resource start p_rabbitmq-server
  • $ pcs status

Ensure that rabbitmq running

  • $ rabbitmqctl status
  • $ rabbitmqctl cluster_status

MySQL

Ensure that there is no MySQL is running and kill it if necessary. 

$ ps -ef|grep mysql

Start MySQL as donor for galera cluster. 

$ export OCF_RESOURCE_INSTANCE=p_mysql
$ export OCF_ROOT=/usr/lib/ocf
$ export OCF_RESKEY_socket=/var/run/mysqld/mysqld.sock
$ export OCF_RESKEY_additional_parameters="--wsrep-new-cluster"
$ /usr/lib/ocf/resource.d/fuel/mysql-wss start
</code>
Check mysql in a couple of minutes
<PRE>
$ mysql -e "show status like 'wsrep_%';"

Create mysql backup just in case (if necessary) 

$ mkdir /tmp/db-backup; \
 innobackupex --no-timestamp --socket=/var/run/mysqld/mysqld.sock /tmp/db-backup; \
 innobackupex --use-memory=1G --lock-wait-query-type=all --apply-log --socket=/var/run/mysqld/mysqld.sock /tmp/db-backup

You can specify just some special list of tables, Excluding zabbix. please refer to Percona Partial Backups. For example backup without zabbix we will have: 

$ mysql -e "SELECT CONCAT(table_schema,'.',table_name) FROM information_schema.tables WHERE table_schema not like 'zabbix'; " > db-tables.txt
$ innobackupex --tables-file=db-tables.txt --socket=/var/run/mysqld/mysqld.sock /tmp/db-backup

Ceph

We have one controller with ceph monitor. There is no quorum, and ceph is not operational. We need remove 2 other monitors from down controllers.
Stop ceph monitor: 

$ stop ceph-mon-all

Dump monitor map:

  • $ mkdir ./ceph-mon-dump
  • $ ceph-mon -i node-1 --extract-monmap ./ceph-mon-dump/monmap
  • $ cp ./ceph-mon-dump/monmap ./ceph-mon-dump/monmap.bak

Remove node–2 and node–3 from monitor-map

  • $ monmaptool ./ceph-mon-dump/monmap --rm node-2
  • $ monmaptool ./ceph-mon-dump/monmap --rm node-3

Inject updated monitor map with one monitor:

  • $ ceph-mon -i node-1 --inject-monmap ./ceph-mon-dump/monmap

Starting and checking ceph

  • $ start ceph-mon-all
  • $ ceph -s

Restart ceph osd on ceph-osd nodes if necessary

Neutron and nova services

Now we have MySQL, RabbitMQ and ceph running.
Check and restore pacemaker services on controller, and neutron and nova services on computes.

  • $ pcs status
  • $ . openrc
  • $ nova service-list
  • $ neutron agent-list
  • $ neutron agent-list |grep -i xxx

After that cloud should be operational.

Node–2 restoration

Now we have:
cloud – operational, up and running.

  • node–1 – Single controller. up and running.
  • node–2 – down or malfunction controller. let’s start this node.
  • node–3 – down or malfunction controller


Start node–2, if it down.
For restoring node 2 we have the same sequence, except rabbitmq: adding second node to the rabbitMQ cluster by pacemaker causes rabbitMQ service interruption for a 1–5 min. Node–2 still unmanageable and marked as “maintenance” in the pacemaker.

Restoring MySQL on node–2

ssh to node–2.
On node–2:
Stop mysql by pacemaker script if running:

  • $ export OCF_RESOURCE_INSTANCE=p_mysql
  • $ export OCF_ROOT=/usr/lib/ocf
  • $ export OCF_RESKEY_socket=/var/run/mysqld/mysqld.sock
  • $ /usr/lib/ocf/resource.d/fuel/mysql-wss stop

Ensure that mysql is down and no mysql process exist. This is mandatory. Kill the MySQL process if it still running or stucked.
Clean up the MySQL directory and run MySQL by pacemaker.

  • $ rm -R /var/lib/mysql/*
  • $ /usr/lib/ocf/resource.d/fuel/mysql-wss start

In a 5 minutes or so replication started.
In a some time depends of the data amount and network speed replication completed and mysql started.
Check the /var/log/mysqld.log for details. Check mysql cluster status after mysql started as galera cluster member. 

$ mysql -e "show status like 'wsrep_%';"

Return back to node–1.

Adding node–2 to pacemaker cluster

Now we need restore pacemaker cluster management for node–2.
Please make sure that we will not start rabbitMQ on the node.

  • $ pcs resource ban p_rabbitmq-server node-2
  • $ crm node ready node-2
  • $ pcs status

Adding node–2 to the ceph map

Stop ceph monitor on node–2 

$ ssh node-2 stop ceph-mon-all </code>

re-add monitor on node-2 node.

  • $ ceph mon remove node-2
  • $ ceph mon add node-2 <ip addr of node-2 from management network>
  • $ ceph-deploy --overwrite-conf mon create node-2

Check ceph status 

$ ceph -s

As result you will set two monitors in the ceph cluster.

Nova and neutron services.

Check and restore (if necessary) nova and neutron services on node–2

  • $ nova service-list
  • $ neutron agent-list
  • $ neutron agent-list |grep -i xxx

Node–3 restoration

Now we have:
cloud – operational, up and running.

  • node–1 – 1st controller. up and running.
  • node–2 – 2nd controller. up and running.
  • node–3 – down or malfunction controller.

Start node–3, if it down.

Restoring node–3 is similar to restoting node–2, with the same exception for rabbitmq:
adding second node to the rabbitMQ cluster by pacemaker causes rabbitMQ service interruption for a 1–5 min.
Node–3 still is unmanageable and marked as “maintenance” in the pacemaker. Perform all steps similar to restore node-2

Restoring quorum policy

Restore quorum policy 

$ crm configure property no-quorum-policy=stop

Check pacemaker config and status

  • $ crm configure show
  • $ pcs cluster cib |grep no-quorum
  • $ pcs status

Post-recovering tasks.

This is task assigned to the maintenances because some service interruption expected.

Restoring mysql managing by pacemaker

This step is not cause mysql restart. It is safe to do during business time, but we recommend shift this step to the maintenance.

  • $ pcs resource clear p_mysql node-2
  • $ pcs resource clear p_mysql node-3
  • $ pcs resource enable clone_p_mysql
  • $ pcs status

Starting zabbix monitoring service.

Start zabbix service

  • $ pcs resource enable p_zabbix-server

In 2-5 minutes please check zabbix status in the pacemaker dashboard

  • $ pcs status

Restoring rabbitmq cluster managing by pacemaker.

Service interruption for 1–5 min.

Return to pacemaker managing of rabbitmq

  • $ pcs resource clear p_rabbitmq-server node-2
  • $ pcs resource clear p_rabbitmq-server node-3

In a 3-5 minutes check rabbitmq cluster status by rabbitmqctl:

  • $ rabbitmqctl cluster_status

Check by pacemaker:

  • $ pcs status p_rabbitmq-server

Ensure that cluster name is the same on master node from pacemaker prospective.

Источник — https://noname.com.ua/mediawiki/index.php?title=Pacemaker_Corosync&oldid=12627