最近在 Kubernetes 上部署 pod 時，

時常遇到 CPU 或記憶體不足的問題，導致部署失敗。

如果用  kubectl describe node 指令的話，是可以看到每個 node 的預期使用率，

只是要一個個 node 看，不是很方便…

$ kubectl describe node
......

Allocated resources:
  (Total limits may be over 100 percent, i.e., overcommitted.)
  Resource           Requests           Limits
  --------           --------           ------
  cpu                7731m (48%)        9851m (61%)
  memory             15574884608 (26%)  15851708672 (27%)
  ephemeral-storage  0 (0%)             0 (0%)
  hugepages-1Gi      0 (0%)             0 (0%)
  hugepages-2Mi      0 (0%)             0 (0%)

 

在網路上找了一下，發現了 Krew 這個用來管理 K8S 套件的工具。

按照指示，在我的 Linux 機器上執行下面的指令來安裝：

(
   set -x; cd "$(mktemp -d)" &&
   OS="$(uname | tr '[:upper:]' '[:lower:]')" &&
   ARCH="$(uname -m | sed -e 's/x86_64/amd64/' -e 's/\(arm\)\(64\)\?.*/\1\2/' -e 's/aarch64$/arm64/')" &&
   KREW="krew-${OS}_${ARCH}" &&
   curl -fsSLO "https://github.com/kubernetes-sigs/krew/releases/latest/download/${KREW}.tar.gz" &&
   tar zxvf "${KREW}.tar.gz" &&
   ./"${KREW}" install krew
)

 

安裝好之後，在 ~/.bashrc 裡面加上下面這行：

export PATH="${KREW_ROOT:-$HOME/.krew}/bin:$PATH"

 

重新開啟 shell 後，已經可以使用 Krew 的命令了：

$ kubectl krew
krew is the kubectl plugin manager.
You can invoke krew through kubectl: "kubectl krew [command]..."

Usage:
  kubectl krew [command]

Available Commands:
  completion  generate the autocompletion script for the specified shell
  help        Help about any command
  index       Manage custom plugin indexes
  info        Show information about an available plugin
  install     Install kubectl plugins
  list        List installed kubectl plugins
  search      Discover kubectl plugins
  uninstall   Uninstall plugins
  update      Update the local copy of the plugin index
  upgrade     Upgrade installed plugins to newer versions
  version     Show krew version and diagnostics

Flags:
  -h, --help      help for krew
  -v, --v Level   number for the log level verbosity

Use "kubectl krew [command] --help" for more information about a command.

 

這時使用 Krew 來安裝 resource-capacity 這個套件：

kubectl krew install resource-capacity

 

安裝好的套件可以用  kubectl krew list 來查看：

$ kubectl krew list

PLUGIN             VERSION
krew               v0.4.3
resource-capacity  v0.7.0

 

裝好之後，就可以來查看所有 node 的資源使用量了，

預設使用表格方式顯示，可以很容易看到各個 node 上 CPU/Memory 的 requests/limits 的數值：

$ kubectl resource-capacity --sort mem.limit

NODE                       CPU REQUESTS   CPU LIMITS      MEMORY REQUESTS         MEMORY LIMITS
*                          67426m (46%)   113405m (77%)   137083Mi (27%)          176770Mi (34%)
k8s-linuxpool-37104297-5   9432m (58%)    11902m (74%)    26009Mi (46%)           27729Mi (49%)
k8s-linuxpool-37104297-1   6661m (41%)    16401m (102%)   11898Mi (21%)           23802Mi (42%)
k8s-linuxpool-37104297-7   8013m (50%)    14665m (91%)    18868459008000m (32%)   22724458240000m (39%)
k8s-linuxpool-37104297-2   6610m (41%)    12650m (79%)    10963Mi (19%)           19521Mi (35%)
k8s-linuxpool-37104297-6   8560m (53%)    12600m (78%)    15840Mi (28%)           18440Mi (33%)
k8s-linuxpool-37104297-3   6960m (43%)    10500m (65%)    15412Mi (27%)           17004Mi (30%)
k8s-linuxpool-37104297-4   7084m (44%)    9324m (58%)     16082651648000m (27%)   17728915968000m (30%)
k8s-linuxpool-37104297-0   6930m (43%)    9550m (59%)     12454Mi (22%)           13742Mi (24%)
k8s-linuxpool-37104297-8   6781m (42%)    9763m (61%)     10719Mi (19%)           12545Mi (22%)
k8s-master-37104297-0      395m (19%)     6050m (302%)    458Mi (7%)              5410Mi (87%)

 

可以再加上  --util 參數，來查看目前真正的使用率：

$ kubectl resource-capacity --util --sort mem.limit
NODE                       CPU REQUESTS   CPU LIMITS      CPU UTIL     MEMORY REQUESTS         MEMORY LIMITS           MEMORY UTIL
*                          67426m (46%)   113405m (77%)   3640m (2%)   137083Mi (27%)          176770Mi (34%)          62655Mi (12%)
k8s-linuxpool-37104297-5   9432m (58%)    11902m (74%)    231m (1%)    26009Mi (46%)           27729Mi (49%)           4540Mi (8%)
k8s-linuxpool-37104297-1   6661m (41%)    16401m (102%)   383m (2%)    11898Mi (21%)           23802Mi (42%)           7880Mi (14%)
k8s-linuxpool-37104297-7   8013m (50%)    14665m (91%)    297m (1%)    18868459008000m (32%)   22724458240000m (39%)   9632Mi (17%)
k8s-linuxpool-37104297-2   6610m (41%)    12650m (79%)    351m (2%)    10963Mi (19%)           19521Mi (35%)           4852Mi (8%)
k8s-linuxpool-37104297-6   8560m (53%)    12600m (78%)    383m (2%)    15840Mi (28%)           18440Mi (33%)           6137Mi (11%)
k8s-linuxpool-37104297-3   6960m (43%)    10500m (65%)    320m (2%)    15412Mi (27%)           17004Mi (30%)           8671Mi (15%)
k8s-linuxpool-37104297-4   7084m (44%)    9324m (58%)     386m (2%)    16082651648000m (27%)   17728915968000m (30%)   4908Mi (8%)
k8s-linuxpool-37104297-0   6930m (43%)    9550m (59%)     190m (1%)    12454Mi (22%)           13742Mi (24%)           6746Mi (12%)
k8s-linuxpool-37104297-8   6781m (42%)    9763m (61%)     698m (4%)    10719Mi (19%)           12545Mi (22%)           5720Mi (10%)
k8s-master-37104297-0      395m (19%)     6050m (302%)    405m (20%)   458Mi (7%)              5410Mi (87%)            3573Mi (57%)

 

而可以用  --sort 參數來排序的欄位有這些：

• cpu.util
• cpu.request
• cpu.limit
• mem.util
• mem.request
• mem.limit
• name

 

有了這個 resource-capacity 套件，

就可以很容易查看 node 上的 CPU/Memory 使用率，

也可以更好地規劃 Kubernetes 叢集應該要配置多少的 CPU 與記憶體囉～

 

參考資料：Get CPU and Memory Usage of NODES and PODS – Kubectl