K8s Notes: различия между версиями

Материал из noname.com.ua
Перейти к навигацииПерейти к поиску
(Новая страница: «Категория:k8s =Заметки= PolicyDesruptionBudget: https://cloud.google.com/anthos/clusters/docs/bare-metal/latest/troubleshooting/remove-node-i...»)
 
 
(не показаны 2 промежуточные версии этого же участника)
Строка 1: Строка 1:
 
[[Категория:k8s]]
  
[[Категория:k8s]]
  +
[[Категория:MCC]]
  +
[[Категория:MOSK]]
  +
[[Категория:MIRANTIS]]
 
=Заметки=
  
=Заметки=
  +
Это заметки о том что могу забыть что бы не искать долго
⚫
PolicyDesruptionBudget: https://cloud.google.com/anthos/clusters/docs/bare-metal/latest/troubleshooting/remove-node-issues
  
  +
<BR>Касаются в основном продуктов Mirantis (MCC/MOS/MOSK)
  +
=PDB=
  +
Проблема удаления ноды (drain): <BR>
  +
Иногда upgrade может зависнутьи висеть бесконечно долго. Эту проблему вызывает <code>PolicyDesruptionBudget</code><BR>
  +
  ⚫
* PolicyDesruptionBudget: https://cloud.google.com/anthos/clusters/docs/bare-metal/latest/troubleshooting/remove-node-issues
  +
В логах это выглядит так (на менеджмент кластере или региональном кластере если он есть)
  +
<code>kubectl -n kaas logs lcm-lcm-controller-67b54d6855-68mcl --since=50m | grep -iE "evict|pdb|budget|drain" --color</code>
  +
<PRE>
  +
{"level":"info","ts":"2023-03-24T13:25:18Z","logger":"lcmcluster-controller","msg":"holding off machine update","machine":"workload-xzk4w","needCordonDrain":true}
  +
{"level":"info","ts":"2023-03-24T13:25:18Z","logger":"lcmcluster-controller","msg":"holding off machine update","machine":"kng-kubecondemo-auto-upg-node-bfcbf","needCordonDrain":true}
  +
{"level":"info","ts":"2023-03-24T13:25:18Z","logger":"lcmcluster-controller","msg":"holding off machine update","machine":"kubeflow-nodes-tq2q7","needCordonDrain":true}
  +
evicting pod knative-serving/webhook-6975ccdbf4-wd6fm
  +
error when evicting pods/"webhook-6975ccdbf4-wd6fm" -n "knative-serving" (will retry after 5s): Cannot evict pod as it would violate the pod's disruption budget.
  +
evicting pod knative-serving/webhook-6975ccdbf4-wd6fm
  +
error when evicting pods/"webhook-6975ccdbf4-wd6fm" -n "knative-serving" (will retry after 5s): Cannot evict pod as it would violate the pod's disruption budget.
  +
evicting pod knative-serving/webhook-6975ccdbf4-wd6fm
  +
error when evicting pods/"webhook-6975ccdbf4-wd6fm" -n "knative-serving" (will retry after 5s): Cannot evict pod as it would violate the pod's disruption budget.
  +
{"level":"info","ts":"2023-03-24T13:25:31Z","logger":"lcmcluster-controller","msg":"holding off machine update","machine":"workload-xzk4w","needCordonDrain":true}
  +
{"level":"info","ts":"2023-03-24T13:25:31Z","logger":"lcmcluster-controller","msg":"holding off machine update","machine":"kng-kubecondemo-auto-upg-node-bfcbf","needCordonDrain":true}
  +
</PRE>
  +
Под который вызывает "затык" - <code>knative-serving/webhook-6975ccdbf4-wd6fm</code> <BR>
  +
Проблема решается или вручную удалением POD или редактированием disruptionBudget

Текущая версия на 15:34, 24 марта 2023

Заметки

Это заметки о том что могу забыть что бы не искать долго
Касаются в основном продуктов Mirantis (MCC/MOS/MOSK)

PDB

Проблема удаления ноды (drain):
Иногда upgrade может зависнутьи висеть бесконечно долго. Эту проблему вызывает PolicyDesruptionBudget

В логах это выглядит так (на менеджмент кластере или региональном кластере если он есть) kubectl -n kaas logs lcm-lcm-controller-67b54d6855-68mcl --since=50m | grep -iE "evict|pdb|budget|drain" --color 

{"level":"info","ts":"2023-03-24T13:25:18Z","logger":"lcmcluster-controller","msg":"holding off machine update","machine":"workload-xzk4w","needCordonDrain":true}
{"level":"info","ts":"2023-03-24T13:25:18Z","logger":"lcmcluster-controller","msg":"holding off machine update","machine":"kng-kubecondemo-auto-upg-node-bfcbf","needCordonDrain":true}
{"level":"info","ts":"2023-03-24T13:25:18Z","logger":"lcmcluster-controller","msg":"holding off machine update","machine":"kubeflow-nodes-tq2q7","needCordonDrain":true}
evicting pod knative-serving/webhook-6975ccdbf4-wd6fm
error when evicting pods/"webhook-6975ccdbf4-wd6fm" -n "knative-serving" (will retry after 5s): Cannot evict pod as it would violate the pod's disruption budget.
evicting pod knative-serving/webhook-6975ccdbf4-wd6fm
error when evicting pods/"webhook-6975ccdbf4-wd6fm" -n "knative-serving" (will retry after 5s): Cannot evict pod as it would violate the pod's disruption budget.
evicting pod knative-serving/webhook-6975ccdbf4-wd6fm
error when evicting pods/"webhook-6975ccdbf4-wd6fm" -n "knative-serving" (will retry after 5s): Cannot evict pod as it would violate the pod's disruption budget.
{"level":"info","ts":"2023-03-24T13:25:31Z","logger":"lcmcluster-controller","msg":"holding off machine update","machine":"workload-xzk4w","needCordonDrain":true}
{"level":"info","ts":"2023-03-24T13:25:31Z","logger":"lcmcluster-controller","msg":"holding off machine update","machine":"kng-kubecondemo-auto-upg-node-bfcbf","needCordonDrain":true}

Под который вызывает "затык" - knative-serving/webhook-6975ccdbf4-wd6fm
Проблема решается или вручную удалением POD или редактированием disruptionBudget

Источник — https://noname.com.ua/mediawiki/index.php?title=K8s_Notes&oldid=11065