PERFORMANCE COMPARISON AMONG dm-cache, bcache and EnhanceIO

This article is copied from https://github.com/stec-inc/EnhanceIO/wiki/PERFORMANCE-COMPARISON-AMONG-dm-cache,-bcache-and-EnhanceIO

I found it's really good. Thanks

The fio config and setup information follows:

HDD              : 100GB
SSD              :  20GB
Mode             : write through / write back
Cache block_size : 4KB for bcache, EnhanceIO and 256KB for dm-cache

The other options have been left to their default values.

Note: 1. In case of dm-cache, we used 2 partitions of same SSD with 1GB partition as metadata and 20GB partition as caching device. This has been done so as to ensure a fair comparison as EnhanceIO and bcache do not have a separate metadata device. 2. In order to ensure proper cache warm up, We have turned off sequential bypass in bcache. This does not impact our performance results as they are taken for random workload.

For each test, we first performed a warm up run using the following fio options:

fio --direct=1 --size=90% --filesize=20G --blocksize=4k --ioengine=libaio --rw=rw --rwmixread=100
--rwmixwrite=0 --iodepth=8 ...

Then, we performed our actual run with the following fio options:

fio --direct=1 --size=100% --filesize=20G --blocksize=4k --ioengine=libaio --rw=randrw --rwmixread=90
--rwmixwrite=10 --iodepth=8 --numjobs=4 --random_distribution=zipf:1.2 ...

WRITE THROUGH MODE TEST RESULTS

EnhanceIO-mode-wt

Iteration         Read Latency ms        Write Latency ms    Read MB/s    Write MB/s
1            1.58            16.21            32.86        3.65
2            1.58            16.35            32.7        3.63
3            1.58            16.54            32.56        3.61
4            1.59            16.6            32.48        3.6
5            1.58            16.58            32.57        3.61
Standard deviation    0.004472136        0.1694992625        0.1489295135    0.02
Mean            1.582            16.456            32.634        3.62

dm-cache-HDD-100G-SSD-20G-mode-wt-io-4K-read-90-write-10-hits-90-ram-48G

Iteration         Read Latency ms        Write Latency ms    Read MB/s    Write MB/s
1            0.86            31.04            23.96        2.65
2            0.31            28.01            29.58        3.28
3            0.39            28.67            28.48        3.16
4            0.39            29.15            29.46        3.27
5            0.28            27.94            29.72        3.3
Standard deviation    0.2364952431        1.2638710377        2.4418435658    0.2748999818
Mean            0.446            28.962            28.24        3.132

bcache-HDD-100G-SSD-20G-mode-wt-io-4K-read-90-write-10-hits-90-ram-48G

Iteration         Read Latency ms        Write Latency ms    Read MB/s    Write MB/s
1            0.56            31.08            27.03        3
2            0.61            30            30.7        3.4
3            0.57            29.36            30.72        3.41
4            0.6            30.02            29.79        3.3
5            0.93            35.65            25.55        2.83
Standard Deviation    0.1556598856        2.5511017228        2.343239211    0.2599423013
Mean            0.654            31.222            28.758        3.188

WRITE BACK MODE TEST RESULTS

EnhanceIO-mode-wb

Iteration         Read Latency ms        Write Latency ms    Read MB/s    Write MB/s
1            0.33            4.69            145.11        16.14
2            0.33            4.64            146.06        16.23
3            0.33            4.62            146.72        16.3
4            0.33            4.65            145.99        16.2
5            0.33            4.72            144.25        16.02
Standard Deviation    0            0.0403732585        0.9588169794    0.1054514106
Mean            0.33            4.664            145.626        16.178

dm-cache-mode-wb

Iteration         Read Latency ms        Write Latency ms    Read MB/s    Write MB/s
1            1.02            2.42            94.49        10.5
2            1.69            4.97            53.89        5.98
3            1.11            2.72            86.2        9.56
4            0.53            0.52            208.54        23.16
5            1.6            4.11            59.47        6.61
Standard Deviation    0.4714339827        1.7076797124        62.7877437881    6.9737091996
Mean            1.19            2.948            100.518        11.162

bcache-mode-wb

Iteration         Read Latency ms        Write Latency ms    Read MB/s    Write MB/s
1            0.95            1.67            108.08        12
2            0.97            1.63            106.77        11.86
3            0.97            1.61            106.99        11.88
4            1            1.5            104.93        11.66
5            0.94            1.67            107.47        11.95
Standard Deviation    0.0230217289        0.0698569968        1.1842381517    0.13
Mean            0.966            1.616            106.848        11.87

Note:The scripts used for these tests can be found at: https://github.com/stec-inc/EnhanceIO/tree/master/performance_test