[Ceph] 使用 rados、ceph、和 fio 指令測試 Ceph 效能
最近在測試新建的 Ceph 叢集的效能，
把搜尋到的指令都記錄一下…
首先是在 cephadm shell 指令模式下，
可以執行 rados bench 指令來測試磁碟效能，
像是下面的指令是測試 10 秒內寫入的速度：
# rados bench -p testbench 10 write --no-cleanup
hints = 1
Maintaining 16 concurrent writes of 4194304 bytes to objects of size 4194304 for up to 10 seconds or 0 objects
Object prefix: benchmark_data_test-ceph-vm1_1372
sec Cur ops started finished avg MB/s cur MB/s last lat(s) avg lat(s)
0 0 0 0 0 0 - 0
1 16 313 297 1187.95 1188 0.0414735 0.0516698
2 16 573 557 1113.93 1040 0.05781 0.0547785
3 16 847 831 1107.91 1096 0.0779656 0.0571364
4 16 1120 1104 1103.91 1092 0.0732392 0.0564813
5 16 1391 1375 1099.89 1084 0.11295 0.0578266
6 16 1659 1643 1095.22 1072 0.0371295 0.0564791
7 16 1941 1925 1099.89 1128 0.0131482 0.0519181
8 16 2185 2169 1084.39 976 0.00961795 0.0479551
9 16 2454 2438 1083.38 1076 0.0383834 0.058801
10 16 2727 2711 1084.23 1092 0.0227965 0.0584466
11 4 2727 2723 990.027 48 0.0427452 0.0583888
12 4 2727 2723 907.529 0 - 0.0583888
Total time run: 12.3263
Total writes made: 2727
Write size: 4194304
Object size: 4194304
Bandwidth (MB/sec): 884.94
Stddev Bandwidth: 415.83
Max bandwidth (MB/sec): 1188
Min bandwidth (MB/sec): 0
Average IOPS: 221
Stddev IOPS: 103.958
Max IOPS: 297
Min IOPS: 0
Average Latency(s): 0.0621596
Stddev Latency(s): 0.183044
Max latency(s): 3.10595
Min latency(s): 0.00657373
以上面的輸出來看，寫入可以達到 884.94 MB/秒，
IOPS 平均是 221，但因為 IOPS 每秒平均 I/O 次數，
因此和單次讀取/寫入的大小會有關係，像上例中預設是單次寫入 4MB 左右。
加上 --object-size 來指定要寫入的大小為 1MB 後，
寫入達到 1070.92 MB/秒，IOPS 也上升到 1070：
# rados bench -p testbench 10 write --no-cleanup --object-size $((1024*1024))
hints = 1
Maintaining 16 concurrent writes of 1048576 bytes to objects of size 1048576 for up to 10 seconds or 0 objects
Object prefix: benchmark_data_test-ceph-vm1_1942
sec Cur ops started finished avg MB/s cur MB/s last lat(s) avg lat(s)
0 0 0 0 0 0 - 0
1 16 1237 1221 1220.92 1221 0.0155967 0.0130126
2 16 2297 2281 1140.41 1060 0.0148864 0.0139774
3 16 3328 3312 1103.91 1031 0.0123301 0.0144233
4 16 4390 4374 1093.4 1062 0.0107925 0.0146018
5 16 5457 5441 1088.1 1067 0.0138464 0.0146763
6 16 6504 6488 1081.24 1047 0.00574564 0.0147792
7 16 7536 7520 1074.19 1032 0.0200246 0.0148717
8 16 8617 8601 1075.03 1081 0.0135501 0.0148632
9 16 9682 9666 1073.9 1065 0.0119147 0.0148804
10 16 10717 10701 1070 1035 0.0140663 0.0149382
Total time run: 10.0073
Total writes made: 10717
Write size: 1048576
Object size: 1048576
Bandwidth (MB/sec): 1070.92
Stddev Bandwidth: 55.6166
Max bandwidth (MB/sec): 1221
Min bandwidth (MB/sec): 1031
Average IOPS: 1070
Stddev IOPS: 55.6166
Max IOPS: 1221
Min IOPS: 1031
Average Latency(s): 0.0149344
Stddev Latency(s): 0.00457351
Max latency(s): 0.0354361
Min latency(s): 0.00351768
另一個可以測試的是看 OSD (Object Storage Daemon) 的效能，
像是下面的指令是叫 osd.0 全部寫入 12MB，切成 4KB block 大小，
每個物件大小是 4MB，總共寫入 100 次：
# ceph tell osd.0 bench 12288000 4096 4194304 100
{
"bytes_written": 12288000,
"blocksize": 4096,
"elapsed_sec": 0.436031583,
"bytes_per_sec": 28181444.828963228,
"iops": 6880.2355539461005
}
也可以直接用 fio 指令測試硬碟的效能，
像是下面的指令使用 fio 對磁碟裝置 /dev/nvme0n1 進行隨機寫入測試：
- -ioengine=libaio：使用 libaio 非同步 I/O 引擎
- -fsync=1：每次 I/O 操作後立即同步寫入磁碟
- -direct=1：直接對磁碟進行寫入，而不經過作業系統快取，
- -bs=4k：區塊大小為 4 KB
- -iodepth=1：每次同時只能有一個 I/O 動作
- -rw=randwrite：進行隨機寫入操作
- -runtime=15：測試時間為 15 秒
$ sudo fio -ioengine=libaio -name=test -filename=/dev/nvme0n1 -fsync=1 -direct=1 -bs=4k -iodepth=1 -rw=randwrite -runtime=15
test: (g=0): rw=randwrite, bs=(R) 4096B-4096B, (W) 4096B-4096B, (T) 4096B-4096B, ioengine=libaio, iodepth=1
fio-3.36
Starting 1 process
Jobs: 1 (f=1): [w(1)][100.0%][w=36.8MiB/s][w=9413 IOPS][eta 00m:00s]
test: (groupid=0, jobs=1): err= 0: pid=73520: Thu Sep 26 08:08:35 2024
write: IOPS=23.4k, BW=91.4MiB/s (95.9MB/s)(1372MiB/15001msec); 0 zone resets
slat (nsec): min=1164, max=943220, avg=30805.63, stdev=48108.29
clat (nsec): min=394, max=370116, avg=10917.42, stdev=6877.66
lat (usec): min=10, max=944, avg=41.72, stdev=41.79
clat percentiles (nsec):
| 1.00th=[ 410], 5.00th=[ 418], 10.00th=[ 422], 20.00th=[ 434],
| 30.00th=[13376], 40.00th=[14144], 50.00th=[14400], 60.00th=[14656],
| 70.00th=[14784], 80.00th=[15040], 90.00th=[15424], 95.00th=[16192],
| 99.00th=[21632], 99.50th=[29568], 99.90th=[32384], 99.95th=[33024],
| 99.99th=[42752]
bw ( KiB/s): min=35848, max=233824, per=100.00%, avg=95587.59, stdev=86644.16, samples=29
iops : min= 8962, max=58456, avg=23896.90, stdev=21661.04, samples=29
lat (nsec) : 500=26.41%, 750=1.22%, 1000=0.29%
lat (usec) : 2=0.18%, 4=0.03%, 10=0.04%, 20=70.31%, 50=1.50%
lat (usec) : 100=0.01%, 250=0.01%, 500=0.01%
fsync/fdatasync/sync_file_range:
sync (nsec): min=960, max=375193, avg=11578.64, stdev=6944.77
sync percentiles (nsec):
| 1.00th=[ 988], 5.00th=[ 1004], 10.00th=[ 1012], 20.00th=[ 1048],
| 30.00th=[13888], 40.00th=[14784], 50.00th=[15040], 60.00th=[15296],
| 70.00th=[15424], 80.00th=[15680], 90.00th=[16064], 95.00th=[17024],
| 99.00th=[22656], 99.50th=[31616], 99.90th=[35072], 99.95th=[35584],
| 99.99th=[48384]
cpu : usr=3.13%, sys=77.41%, ctx=252229, majf=0, minf=12
IO depths : 1=200.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0%
submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
issued rwts: total=0,351132,0,351131 short=0,0,0,0 dropped=0,0,0,0
latency : target=0, window=0, percentile=100.00%, depth=1
Run status group 0 (all jobs):
WRITE: bw=91.4MiB/s (95.9MB/s), 91.4MiB/s-91.4MiB/s (95.9MB/s-95.9MB/s), io=1372MiB (1438MB), run=15001-15001msec
參考資料：Chapter 7. Ceph performance benchmark
(本頁面已被瀏覽過 4 次)