In TCP performance tuning for 10G NIC on linux, the tests were done on physical servers, can the same experience be used on virtual machine? a VM disk server?

More and more storage vendors providing VM server products. like DDN SCALAR products, directly providing file system to enduser. In most cases, they are well tuned and end user won't get chance to access their embeded VM servers.

However, on some products, they allow you to install, maintain and install your own storage applications on embeded servers, then tuning the server becomes your own task.

Here is my experience:

The test bed was a dual controller system, each controller actually is a linux server hosts VM servers which users can manage, dedicated 10GE NIC for each VM.

Each VM was assigned 6 CPUs (3 cores, hyper-threading enabled) and 32 GB of memory.
The kernel and driver versions were:
Scientific Linux 6.4, kernel 2.6.32-431.29.2.el6.x86_64
Mellanox 10GE NIC, driver 1.5.10, SR-IOV enabled (essential for performance), MTU 9000

Below is the sysctl tuning on the VM server, it yield nearly as same as physical machine's performance though it has only 3 cores.

net.core.netdev_max_backlog = 250000
net.core.rmem_max = 16777216
net.core.wmem_max = 16777216
net.ipv4.tcp_rmem = 4096 87380 524288
net.ipv4.tcp_wmem = 4096 65536 524288
# VM Settings
vm.min_free_kbytes = 262144

Note: SR-IOV feature is essensial for the tuning, the 10GE NIC driver accordingly used version 1.5.10 with SR-IOV support.

more info about SR-IOV tuning for performance, see

Effect of SR-IOV Support in Red Hat KVM on Network Performance in Virtualized Environments


Comments powered by CComment