K8s Notes

Заметки

Это заметки о том что могу забыть что бы не искать долго
Касаются в основном продуктов Mirantis (MCC/MOS/MOSK)

PDB

Проблема удаления ноды (drain):
Иногда upgrade может зависнутьи висеть бесконечно долго. Эту проблему вызывает PolicyDesruptionBudget

• PolicyDesruptionBudget: https://cloud.google.com/anthos/clusters/docs/bare-metal/latest/troubleshooting/remove-node-issues

В логах это выглядит так (на менеджмент кластере или региональном кластере если он есть) kubectl -n kaas logs lcm-lcm-controller-67b54d6855-68mcl --since=50m | grep -iE "evict|pdb|budget|drain" --color

{"level":"info","ts":"2023-03-24T13:25:18Z","logger":"lcmcluster-controller","msg":"holding off machine update","machine":"workload-xzk4w","needCordonDrain":true}
{"level":"info","ts":"2023-03-24T13:25:18Z","logger":"lcmcluster-controller","msg":"holding off machine update","machine":"kng-kubecondemo-auto-upg-node-bfcbf","needCordonDrain":true}
{"level":"info","ts":"2023-03-24T13:25:18Z","logger":"lcmcluster-controller","msg":"holding off machine update","machine":"kubeflow-nodes-tq2q7","needCordonDrain":true}
evicting pod knative-serving/webhook-6975ccdbf4-wd6fm
error when evicting pods/"webhook-6975ccdbf4-wd6fm" -n "knative-serving" (will retry after 5s): Cannot evict pod as it would violate the pod's disruption budget.
evicting pod knative-serving/webhook-6975ccdbf4-wd6fm
error when evicting pods/"webhook-6975ccdbf4-wd6fm" -n "knative-serving" (will retry after 5s): Cannot evict pod as it would violate the pod's disruption budget.
evicting pod knative-serving/webhook-6975ccdbf4-wd6fm
error when evicting pods/"webhook-6975ccdbf4-wd6fm" -n "knative-serving" (will retry after 5s): Cannot evict pod as it would violate the pod's disruption budget.
{"level":"info","ts":"2023-03-24T13:25:31Z","logger":"lcmcluster-controller","msg":"holding off machine update","machine":"workload-xzk4w","needCordonDrain":true}
{"level":"info","ts":"2023-03-24T13:25:31Z","logger":"lcmcluster-controller","msg":"holding off machine update","machine":"kng-kubecondemo-auto-upg-node-bfcbf","needCordonDrain":true}

Под который вызывает "затык" - knative-serving/webhook-6975ccdbf4-wd6fm
Проблема решается или вручную удалением POD или редактированием disruptionBudget