最近在測試新建的 Ceph 叢集的效能，

把搜尋到的指令都記錄一下…

首先是在 cephadm shell 指令模式下，

可以執行 rados bench 指令來測試磁碟效能，

像是下面的指令是測試 10 秒內寫入的速度：

# rados bench -p testbench 10 write --no-cleanup

hints = 1
Maintaining 16 concurrent writes of 4194304 bytes to objects of size 4194304 for up to 10 seconds or 0 objects
Object prefix: benchmark_data_test-ceph-vm1_1372
  sec Cur ops   started  finished  avg MB/s  cur MB/s last lat(s)  avg lat(s)
    0       0         0         0         0         0           -           0
    1      16       313       297   1187.95      1188   0.0414735   0.0516698
    2      16       573       557   1113.93      1040     0.05781   0.0547785
    3      16       847       831   1107.91      1096   0.0779656   0.0571364
    4      16      1120      1104   1103.91      1092   0.0732392   0.0564813
    5      16      1391      1375   1099.89      1084     0.11295   0.0578266
    6      16      1659      1643   1095.22      1072   0.0371295   0.0564791
    7      16      1941      1925   1099.89      1128   0.0131482   0.0519181
    8      16      2185      2169   1084.39       976  0.00961795   0.0479551
    9      16      2454      2438   1083.38      1076   0.0383834    0.058801
   10      16      2727      2711   1084.23      1092   0.0227965   0.0584466
   11       4      2727      2723   990.027        48   0.0427452   0.0583888
   12       4      2727      2723   907.529         0           -   0.0583888
Total time run:         12.3263
Total writes made:      2727
Write size:             4194304
Object size:            4194304
Bandwidth (MB/sec):     884.94
Stddev Bandwidth:       415.83
Max bandwidth (MB/sec): 1188
Min bandwidth (MB/sec): 0
Average IOPS:           221
Stddev IOPS:            103.958
Max IOPS:               297
Min IOPS:               0
Average Latency(s):     0.0621596
Stddev Latency(s):      0.183044
Max latency(s):         3.10595
Min latency(s):         0.00657373

以上面的輸出來看，寫入可以達到 884.94 MB/秒，

IOPS 平均是 221，但因為 IOPS 每秒平均 I/O 次數，

因此和單次讀取/寫入的大小會有關係，像上例中預設是單次寫入 4MB 左右。

加上 --object-size 來指定要寫入的大小為 1MB 後，

寫入達到 1070.92 MB/秒，IOPS 也上升到 1070：

# rados bench -p testbench 10 write --no-cleanup --object-size $((1024*1024))

hints = 1
Maintaining 16 concurrent writes of 1048576 bytes to objects of size 1048576 for up to 10 seconds or 0 objects
Object prefix: benchmark_data_test-ceph-vm1_1942
  sec Cur ops   started  finished  avg MB/s  cur MB/s last lat(s)  avg lat(s)
    0       0         0         0         0         0           -           0
    1      16      1237      1221   1220.92      1221   0.0155967   0.0130126
    2      16      2297      2281   1140.41      1060   0.0148864   0.0139774
    3      16      3328      3312   1103.91      1031   0.0123301   0.0144233
    4      16      4390      4374    1093.4      1062   0.0107925   0.0146018
    5      16      5457      5441    1088.1      1067   0.0138464   0.0146763
    6      16      6504      6488   1081.24      1047  0.00574564   0.0147792
    7      16      7536      7520   1074.19      1032   0.0200246   0.0148717
    8      16      8617      8601   1075.03      1081   0.0135501   0.0148632
    9      16      9682      9666    1073.9      1065   0.0119147   0.0148804
   10      16     10717     10701      1070      1035   0.0140663   0.0149382
Total time run:         10.0073
Total writes made:      10717
Write size:             1048576
Object size:            1048576
Bandwidth (MB/sec):     1070.92
Stddev Bandwidth:       55.6166
Max bandwidth (MB/sec): 1221
Min bandwidth (MB/sec): 1031
Average IOPS:           1070
Stddev IOPS:            55.6166
Max IOPS:               1221
Min IOPS:               1031
Average Latency(s):     0.0149344
Stddev Latency(s):      0.00457351
Max latency(s):         0.0354361
Min latency(s):         0.00351768

另一個可以測試的是看 OSD (Object Storage Daemon) 的效能，

像是下面的指令是叫 osd.0 全部寫入 12MB，切成 4KB block 大小，

每個物件大小是 4MB，總共寫入 100 次：

# ceph tell osd.0 bench 12288000 4096 4194304 100

{
    "bytes_written": 12288000,
    "blocksize": 4096,
    "elapsed_sec": 0.436031583,
    "bytes_per_sec": 28181444.828963228,
    "iops": 6880.2355539461005
}

也可以直接用 fio 指令測試硬碟的效能，

像是下面的指令使用 fio 對磁碟裝置 /dev/nvme0n1 進行隨機寫入測試：

• -ioengine=libaio：使用 libaio 非同步 I/O 引擎
• -fsync=1：每次 I/O 操作後立即同步寫入磁碟
• -direct=1：直接對磁碟進行寫入，而不經過作業系統快取，
• -bs=4k：區塊大小為 4 KB
• -iodepth=1：每次同時只能有一個 I/O 動作
• -rw=randwrite：進行隨機寫入操作
• -runtime=15：測試時間為 15 秒

$ sudo fio -ioengine=libaio -name=test -filename=/dev/nvme0n1 -fsync=1 -direct=1 -bs=4k -iodepth=1 -rw=randwrite -runtime=15

test: (g=0): rw=randwrite, bs=(R) 4096B-4096B, (W) 4096B-4096B, (T) 4096B-4096B, ioengine=libaio, iodepth=1
fio-3.36
Starting 1 process
Jobs: 1 (f=1): [w(1)][100.0%][w=36.8MiB/s][w=9413 IOPS][eta 00m:00s]
test: (groupid=0, jobs=1): err= 0: pid=73520: Thu Sep 26 08:08:35 2024
  write: IOPS=23.4k, BW=91.4MiB/s (95.9MB/s)(1372MiB/15001msec); 0 zone resets
    slat (nsec): min=1164, max=943220, avg=30805.63, stdev=48108.29
    clat (nsec): min=394, max=370116, avg=10917.42, stdev=6877.66
     lat (usec): min=10, max=944, avg=41.72, stdev=41.79
    clat percentiles (nsec):
     |  1.00th=[  410],  5.00th=[  418], 10.00th=[  422], 20.00th=[  434],
     | 30.00th=[13376], 40.00th=[14144], 50.00th=[14400], 60.00th=[14656],
     | 70.00th=[14784], 80.00th=[15040], 90.00th=[15424], 95.00th=[16192],
     | 99.00th=[21632], 99.50th=[29568], 99.90th=[32384], 99.95th=[33024],
     | 99.99th=[42752]
   bw (  KiB/s): min=35848, max=233824, per=100.00%, avg=95587.59, stdev=86644.16, samples=29
   iops        : min= 8962, max=58456, avg=23896.90, stdev=21661.04, samples=29
  lat (nsec)   : 500=26.41%, 750=1.22%, 1000=0.29%
  lat (usec)   : 2=0.18%, 4=0.03%, 10=0.04%, 20=70.31%, 50=1.50%
  lat (usec)   : 100=0.01%, 250=0.01%, 500=0.01%
  fsync/fdatasync/sync_file_range:
    sync (nsec): min=960, max=375193, avg=11578.64, stdev=6944.77
    sync percentiles (nsec):
     |  1.00th=[  988],  5.00th=[ 1004], 10.00th=[ 1012], 20.00th=[ 1048],
     | 30.00th=[13888], 40.00th=[14784], 50.00th=[15040], 60.00th=[15296],
     | 70.00th=[15424], 80.00th=[15680], 90.00th=[16064], 95.00th=[17024],
     | 99.00th=[22656], 99.50th=[31616], 99.90th=[35072], 99.95th=[35584],
     | 99.99th=[48384]
  cpu          : usr=3.13%, sys=77.41%, ctx=252229, majf=0, minf=12
  IO depths    : 1=200.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     issued rwts: total=0,351132,0,351131 short=0,0,0,0 dropped=0,0,0,0
     latency   : target=0, window=0, percentile=100.00%, depth=1

Run status group 0 (all jobs):
  WRITE: bw=91.4MiB/s (95.9MB/s), 91.4MiB/s-91.4MiB/s (95.9MB/s-95.9MB/s), io=1372MiB (1438MB), run=15001-15001msec

參考資料：Chapter 7. Ceph performance benchmark