Pacemaker Corosync - История изменений

Sirmax: /* 3 */

2024-02-08T09:46:22Z

← Предыдущая		Версия 09:46, 8 февраля 2024
Строка 225:		Строка 225:
	</PRE>		</PRE>

		+	=Если кластер "не собирается" - поднять таймауты=
−	=3=
		+
−	Validated External
	ProblemSymptoms of the problem being solved, objectives of the procedure or additional details about the question being asked that summarize what the customer is experiencing.		ProblemSymptoms of the problem being solved, objectives of the procedure or additional details about the question being asked that summarize what the customer is experiencing.
	MySQL is not running on one of the controller nodes		MySQL is not running on one of the controller nodes

Sirmax: /* 3 */

2024-02-08T09:45:46Z

← Предыдущая		Версия 09:45, 8 февраля 2024
Строка 229:		Строка 229:
	ProblemSymptoms of the problem being solved, objectives of the procedure or additional details about the question being asked that summarize what the customer is experiencing.		ProblemSymptoms of the problem being solved, objectives of the procedure or additional details about the question being asked that summarize what the customer is experiencing.
	MySQL is not running on one of the controller nodes		MySQL is not running on one of the controller nodes
		+	<PRE>
	# pcs status		# pcs status
	Clone Set: clone_p_mysql [p_mysql]		Clone Set: clone_p_mysql [p_mysql]
	Started: [ b05-39-controller.domain.tld b06-39-controller.domain.tld ]		Started: [ b05-39-controller.domain.tld b06-39-controller.domain.tld ]
	Stopped: [ b05-38-controller.domain.tld ]		Stopped: [ b05-38-controller.domain.tld ]
		+	</PRE>
−	EnvironmentThe characteristics of the issue that are not part of the problem (Product/s, Release, Version/s, Operating System, Component/s)
−	Mirantis OpenStack 7.0 and higher
−	MySQL
−	Pacemaker
−	Galera
−	ResolutionNumbered series of steps about how to fix the issue
	Make sure MySQL is stopped on problematic node		Make sure MySQL is stopped on problematic node
		+	<PRE>
	# ps -ef \| grep mysql		# ps -ef \| grep mysql
	root 14878 5566 0 14:33 pts/0 00:00:00 grep --color=auto mysql		root 14878 5566 0 14:33 pts/0 00:00:00 grep --color=auto mysql
		+	</PRE>
	Edit the default timeout for start operation		Edit the default timeout for start operation
		+	<PRE>
	# crm configure edit p_mysql		# crm configure edit p_mysql
		+	</PRE>
−	Note: for Mirantis OpenStack 9.0 and higher, run the following command:
		+	или
		+	<PRE>
	# crm configure edit p_mysqld		# crm configure edit p_mysqld
		+	</PRE>
	Set the temporary timeout value for p_mysql-start-0 to be 1200		Set the temporary timeout value for p_mysql-start-0 to be 1200
		+	<PRE>
	# pcs resource show p_mysql		# pcs resource show p_mysql
	Resource: p_mysql (class=ocf provider=fuel type=mysql-wss)		Resource: p_mysql (class=ocf provider=fuel type=mysql-wss)
Строка 253:		Строка 256:
	start interval=0 timeout=1200 (p_mysql-start-0)		start interval=0 timeout=1200 (p_mysql-start-0)
	stop interval=0 timeout=120 (p_mysql-stop-0)		stop interval=0 timeout=120 (p_mysql-stop-0)
		+	</PRE>
	Cleanup the p_mysql resource		Cleanup the p_mysql resource
		+	<PRE>
	# crm resource cleanup p_mysql		# crm resource cleanup p_mysql
		+	</PRE>
	Wait for 15-20 minutes for synchronization to be completed		Wait for 15-20 minutes for synchronization to be completed
	Run pcs status again to check mysql is back up and running on the problematic controller node		Run pcs status again to check mysql is back up and running on the problematic controller node
		+	<PRE>
	# pcs status		# pcs status
	Clone Set: clone_p_mysql [p_mysql]		Clone Set: clone_p_mysql [p_mysql]
Строка 265:		Строка 272:
	\| wsrep_causal_reads \| 0 \|		\| wsrep_causal_reads \| 0 \|
	\| wsrep_incoming_addresses \| 10.128.0.133:3307,10.128.0.132:3307,10.128.0.134:3307		\| wsrep_incoming_addresses \| 10.128.0.133:3307,10.128.0.132:3307,10.128.0.134:3307
		+	</PRE>
−	8. Reset the timeout value for p_mysql-start-0 to default value.	+	Reset the timeout value for p_mysql-start-0 to default value.
−

	=Ходовые команды CRM/PCS=		=Ходовые команды CRM/PCS=

Sirmax: /* Restoring rabbitmq cluster managing by pacemaker. */

2024-02-08T09:43:32Z

Restoring rabbitmq cluster managing by pacemaker.

← Предыдущая		Версия 09:43, 8 февраля 2024
Строка 652:		Строка 652:

	Return to pacemaker managing of rabbitmq		Return to pacemaker managing of rabbitmq
−	$ pcs resource clear p_rabbitmq-server node-2 </code>	+	* <code> $ pcs resource clear p_rabbitmq-server node-2 </code>
−	$ pcs resource clear p_rabbitmq-server node-3 </code>	+	* <code> $ pcs resource clear p_rabbitmq-server node-3 </code>
	In a 3-5 minutes check rabbitmq cluster status by rabbitmqctl:		In a 3-5 minutes check rabbitmq cluster status by rabbitmqctl:
−	$ rabbitmqctl cluster_status </code>	+	* <code> $ rabbitmqctl cluster_status </code>
	Check by pacemaker:		Check by pacemaker:
−	$ pcs status p_rabbitmq-server </code>	+	* <code> $ pcs status p_rabbitmq-server </code>
	Ensure that cluster name is the same on master node from pacemaker prospective.		Ensure that cluster name is the same on master node from pacemaker prospective.

Sirmax: /* Starting zabbix monitoring service. */

2024-02-08T09:43:20Z

Starting zabbix monitoring service.

← Предыдущая		Версия 09:43, 8 февраля 2024
Строка 644:		Строка 644:

	Start zabbix service		Start zabbix service
−	$ pcs resource enable p_zabbix-server </code>	+	* <code> $ pcs resource enable p_zabbix-server </code>
	In 2-5 minutes please check zabbix status in the pacemaker dashboard		In 2-5 minutes please check zabbix status in the pacemaker dashboard
−	$ pcs status </code>	+	* <code> $ pcs status </code>

	===Restoring rabbitmq cluster managing by pacemaker.===		===Restoring rabbitmq cluster managing by pacemaker.===

Sirmax: /* Restoring mysql managing by pacemaker */

2024-02-08T09:43:11Z

Restoring mysql managing by pacemaker

← Предыдущая		Версия 09:43, 8 февраля 2024
Строка 636:		Строка 636:

	This step is not cause mysql restart. It is safe to do during business time, but we recommend shift this step to the maintenance.		This step is not cause mysql restart. It is safe to do during business time, but we recommend shift this step to the maintenance.
−	$ pcs resource clear p_mysql node-2 </code>	+	* <code> $ pcs resource clear p_mysql node-2 </code>
−	$ pcs resource clear p_mysql node-3 </code>	+	* <code> $ pcs resource clear p_mysql node-3 </code>
−	$ pcs resource enable clone_p_mysql </code>	+	* <code> $ pcs resource enable clone_p_mysql </code>
−	$ pcs status </code>	+	* <code> $ pcs status </code>

	===Starting zabbix monitoring service.===		===Starting zabbix monitoring service.===

Sirmax: /* Restoring rabbitmq cluster managing by pacemaker. */

2024-02-08T09:42:43Z

Restoring rabbitmq cluster managing by pacemaker.

← Предыдущая		Версия 09:42, 8 февраля 2024
Строка 652:		Строка 652:

	Return to pacemaker managing of rabbitmq		Return to pacemaker managing of rabbitmq
−	$ pcs resource clear p_rabbitmq-server node-2	+	$ pcs resource clear p_rabbitmq-server node-2 </code>
−	$ pcs resource clear p_rabbitmq-server node-3	+	$ pcs resource clear p_rabbitmq-server node-3 </code>
	In a 3-5 minutes check rabbitmq cluster status by rabbitmqctl:		In a 3-5 minutes check rabbitmq cluster status by rabbitmqctl:
−	$ rabbitmqctl cluster_status	+	$ rabbitmqctl cluster_status </code>
	Check by pacemaker:		Check by pacemaker:
−	$ pcs status p_rabbitmq-server	+	$ pcs status p_rabbitmq-server </code>
	Ensure that cluster name is the same on master node from pacemaker prospective.		Ensure that cluster name is the same on master node from pacemaker prospective.

Sirmax: /* Starting zabbix monitoring service. */

2024-02-08T09:42:33Z

Starting zabbix monitoring service.

← Предыдущая		Версия 09:42, 8 февраля 2024
Строка 644:		Строка 644:

	Start zabbix service		Start zabbix service
−	$ pcs resource enable p_zabbix-server	+	$ pcs resource enable p_zabbix-server </code>
	In 2-5 minutes please check zabbix status in the pacemaker dashboard		In 2-5 minutes please check zabbix status in the pacemaker dashboard
−	$ pcs status	+	$ pcs status </code>
−

	===Restoring rabbitmq cluster managing by pacemaker.===		===Restoring rabbitmq cluster managing by pacemaker.===

Sirmax: /* Restoring mysql managing by pacemaker */

2024-02-08T09:42:21Z

Restoring mysql managing by pacemaker

← Предыдущая		Версия 09:42, 8 февраля 2024
Строка 636:		Строка 636:

	This step is not cause mysql restart. It is safe to do during business time, but we recommend shift this step to the maintenance.		This step is not cause mysql restart. It is safe to do during business time, but we recommend shift this step to the maintenance.
−	$ pcs resource clear p_mysql node-2	+	$ pcs resource clear p_mysql node-2 </code>
−	$ pcs resource clear p_mysql node-3	+	$ pcs resource clear p_mysql node-3 </code>
−	$ pcs resource enable clone_p_mysql	+	$ pcs resource enable clone_p_mysql </code>
−	$ pcs status	+	$ pcs status </code>

	===Starting zabbix monitoring service.===		===Starting zabbix monitoring service.===

Sirmax: /* =Restoring quorum policy */

2024-02-08T09:42:03Z

=Restoring quorum policy

← Предыдущая		Версия 09:42, 8 февраля 2024
Строка 618:		Строка 618:
	Perform all steps similar to restore node-2		Perform all steps similar to restore node-2

−	===Restoring quorum policy==	+	===Restoring quorum policy===

	Restore quorum policy		Restore quorum policy

Sirmax: /* Node–3 restoration */

2024-02-08T09:41:50Z

Node–3 restoration

← Предыдущая		Версия 09:41, 8 февраля 2024
Строка 605:		Строка 605:
	==Node–3 restoration==		==Node–3 restoration==

−	Now we have:	+	Now we have:<BR>
	cloud – operational, up and running.		cloud – operational, up and running.
−	node–1 – 1st controller. up and running.	+	* node–1 – 1st controller. up and running.
−	node–2 – 2nd controller. up and running.	+	* node–2 – 2nd controller. up and running.
−	node–3 – down or malfunction controller.	+	* node–3 – down or malfunction controller.
	Start node–3, if it down.		Start node–3, if it down.
		+
⚫	~~Restoring node–3 is similar to restoting node–2, with the same exception for rabbitmq:~~ adding second node to the rabbitMQ cluster by pacemaker causes rabbitMQ service interruption for a 1–5 min.
		+	Restoring node–3 is similar to restoting node–2, with the same exception for rabbitmq:<BR>
		⚫	adding second node to the rabbitMQ cluster by pacemaker causes rabbitMQ service interruption for a 1–5 min.
		+	<BR>
	Node–3 still is unmanageable and marked as “maintenance” in the pacemaker.		Node–3 still is unmanageable and marked as “maintenance” in the pacemaker.
	Perform all steps similar to restore node-2		Perform all steps similar to restore node-2
		+
−	Restoring quorum policy	+	===Restoring quorum policy==

	Restore quorum policy		Restore quorum policy
		+	<PRE>
	$ crm configure property no-quorum-policy=stop		$ crm configure property no-quorum-policy=stop
		+	</PRE>
	Check pacemaker config and status		Check pacemaker config and status
−	$ crm configure show	+	* <code>$ crm configure show </code>
−	$ pcs cluster cib \|grep no-quorum	+	* <code>$ pcs cluster cib \|grep no-quorum </code>
−	$ pcs status	+	* <code>$ pcs status </code>
		+
	==Post-recovering tasks.==		==Post-recovering tasks.==