[Ceph] 使用 rados、ceph、和 fio 指令測試 Ceph 效能
最近在測試新建的 Ceph 叢集的效能,
把搜尋到的指令都記錄一下…
首先是在 cephadm shell 指令模式下,
可以執行 rados bench
指令來測試磁碟效能,
像是下面的指令是測試 10 秒內寫入的速度:
# rados bench -p testbench 10 write --no-cleanup hints = 1 Maintaining 16 concurrent writes of 4194304 bytes to objects of size 4194304 for up to 10 seconds or 0 objects Object prefix: benchmark_data_test-ceph-vm1_1372 sec Cur ops started finished avg MB/s cur MB/s last lat(s) avg lat(s) 0 0 0 0 0 0 - 0 1 16 313 297 1187.95 1188 0.0414735 0.0516698 2 16 573 557 1113.93 1040 0.05781 0.0547785 3 16 847 831 1107.91 1096 0.0779656 0.0571364 4 16 1120 1104 1103.91 1092 0.0732392 0.0564813 5 16 1391 1375 1099.89 1084 0.11295 0.0578266 6 16 1659 1643 1095.22 1072 0.0371295 0.0564791 7 16 1941 1925 1099.89 1128 0.0131482 0.0519181 8 16 2185 2169 1084.39 976 0.00961795 0.0479551 9 16 2454 2438 1083.38 1076 0.0383834 0.058801 10 16 2727 2711 1084.23 1092 0.0227965 0.0584466 11 4 2727 2723 990.027 48 0.0427452 0.0583888 12 4 2727 2723 907.529 0 - 0.0583888 Total time run: 12.3263 Total writes made: 2727 Write size: 4194304 Object size: 4194304 Bandwidth (MB/sec): 884.94 Stddev Bandwidth: 415.83 Max bandwidth (MB/sec): 1188 Min bandwidth (MB/sec): 0 Average IOPS: 221 Stddev IOPS: 103.958 Max IOPS: 297 Min IOPS: 0 Average Latency(s): 0.0621596 Stddev Latency(s): 0.183044 Max latency(s): 3.10595 Min latency(s): 0.00657373
以上面的輸出來看,寫入可以達到 884.94 MB/秒,
IOPS 平均是 221,但因為 IOPS 每秒平均 I/O 次數,
因此和單次讀取/寫入的大小會有關係,像上例中預設是單次寫入 4MB 左右。
加上 --object-size
來指定要寫入的大小為 1MB 後,
寫入達到 1070.92 MB/秒,IOPS 也上升到 1070:
# rados bench -p testbench 10 write --no-cleanup --object-size $((1024*1024)) hints = 1 Maintaining 16 concurrent writes of 1048576 bytes to objects of size 1048576 for up to 10 seconds or 0 objects Object prefix: benchmark_data_test-ceph-vm1_1942 sec Cur ops started finished avg MB/s cur MB/s last lat(s) avg lat(s) 0 0 0 0 0 0 - 0 1 16 1237 1221 1220.92 1221 0.0155967 0.0130126 2 16 2297 2281 1140.41 1060 0.0148864 0.0139774 3 16 3328 3312 1103.91 1031 0.0123301 0.0144233 4 16 4390 4374 1093.4 1062 0.0107925 0.0146018 5 16 5457 5441 1088.1 1067 0.0138464 0.0146763 6 16 6504 6488 1081.24 1047 0.00574564 0.0147792 7 16 7536 7520 1074.19 1032 0.0200246 0.0148717 8 16 8617 8601 1075.03 1081 0.0135501 0.0148632 9 16 9682 9666 1073.9 1065 0.0119147 0.0148804 10 16 10717 10701 1070 1035 0.0140663 0.0149382 Total time run: 10.0073 Total writes made: 10717 Write size: 1048576 Object size: 1048576 Bandwidth (MB/sec): 1070.92 Stddev Bandwidth: 55.6166 Max bandwidth (MB/sec): 1221 Min bandwidth (MB/sec): 1031 Average IOPS: 1070 Stddev IOPS: 55.6166 Max IOPS: 1221 Min IOPS: 1031 Average Latency(s): 0.0149344 Stddev Latency(s): 0.00457351 Max latency(s): 0.0354361 Min latency(s): 0.00351768
另一個可以測試的是看 OSD (Object Storage Daemon) 的效能,
像是下面的指令是叫 osd.0 全部寫入 12MB,切成 4KB block 大小,
每個物件大小是 4MB,總共寫入 100 次:
# ceph tell osd.0 bench 12288000 4096 4194304 100 { "bytes_written": 12288000, "blocksize": 4096, "elapsed_sec": 0.436031583, "bytes_per_sec": 28181444.828963228, "iops": 6880.2355539461005 }
也可以直接用 fio 指令測試硬碟的效能,
像是下面的指令使用 fio 對磁碟裝置 /dev/nvme0n1 進行隨機寫入測試:
- -ioengine=libaio:使用 libaio 非同步 I/O 引擎
- -fsync=1:每次 I/O 操作後立即同步寫入磁碟
- -direct=1:直接對磁碟進行寫入,而不經過作業系統快取,
- -bs=4k:區塊大小為 4 KB
- -iodepth=1:每次同時只能有一個 I/O 動作
- -rw=randwrite:進行隨機寫入操作
- -runtime=15:測試時間為 15 秒
$ sudo fio -ioengine=libaio -name=test -filename=/dev/nvme0n1 -fsync=1 -direct=1 -bs=4k -iodepth=1 -rw=randwrite -runtime=15 test: (g=0): rw=randwrite, bs=(R) 4096B-4096B, (W) 4096B-4096B, (T) 4096B-4096B, ioengine=libaio, iodepth=1 fio-3.36 Starting 1 process Jobs: 1 (f=1): [w(1)][100.0%][w=36.8MiB/s][w=9413 IOPS][eta 00m:00s] test: (groupid=0, jobs=1): err= 0: pid=73520: Thu Sep 26 08:08:35 2024 write: IOPS=23.4k, BW=91.4MiB/s (95.9MB/s)(1372MiB/15001msec); 0 zone resets slat (nsec): min=1164, max=943220, avg=30805.63, stdev=48108.29 clat (nsec): min=394, max=370116, avg=10917.42, stdev=6877.66 lat (usec): min=10, max=944, avg=41.72, stdev=41.79 clat percentiles (nsec): | 1.00th=[ 410], 5.00th=[ 418], 10.00th=[ 422], 20.00th=[ 434], | 30.00th=[13376], 40.00th=[14144], 50.00th=[14400], 60.00th=[14656], | 70.00th=[14784], 80.00th=[15040], 90.00th=[15424], 95.00th=[16192], | 99.00th=[21632], 99.50th=[29568], 99.90th=[32384], 99.95th=[33024], | 99.99th=[42752] bw ( KiB/s): min=35848, max=233824, per=100.00%, avg=95587.59, stdev=86644.16, samples=29 iops : min= 8962, max=58456, avg=23896.90, stdev=21661.04, samples=29 lat (nsec) : 500=26.41%, 750=1.22%, 1000=0.29% lat (usec) : 2=0.18%, 4=0.03%, 10=0.04%, 20=70.31%, 50=1.50% lat (usec) : 100=0.01%, 250=0.01%, 500=0.01% fsync/fdatasync/sync_file_range: sync (nsec): min=960, max=375193, avg=11578.64, stdev=6944.77 sync percentiles (nsec): | 1.00th=[ 988], 5.00th=[ 1004], 10.00th=[ 1012], 20.00th=[ 1048], | 30.00th=[13888], 40.00th=[14784], 50.00th=[15040], 60.00th=[15296], | 70.00th=[15424], 80.00th=[15680], 90.00th=[16064], 95.00th=[17024], | 99.00th=[22656], 99.50th=[31616], 99.90th=[35072], 99.95th=[35584], | 99.99th=[48384] cpu : usr=3.13%, sys=77.41%, ctx=252229, majf=0, minf=12 IO depths : 1=200.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=0,351132,0,351131 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=1 Run status group 0 (all jobs): WRITE: bw=91.4MiB/s (95.9MB/s), 91.4MiB/s-91.4MiB/s (95.9MB/s-95.9MB/s), io=1372MiB (1438MB), run=15001-15001msec
參考資料:Chapter 7. Ceph performance benchmark
(本頁面已被瀏覽過 29 次)