KVM环境下vCPU绑定到物理CPU

在KVM环境中测试虚拟系统性能时，如果宿主机是有两个CPU socket的硬件，会碰到由于vCPU在不同物理CPU上浮动导致测试RFC2544时出现少量丢包的现象，测试结果非常不稳定。可以将vCPU绑定到物理CPU来优化这个问题(不能完全避免)，本章仅介绍单vCPU绑定到单物理CPU上的情况。绑定到单物理CPU上可以暂时规避少量丢包的问题，但是也存在宿主机上单个物理CPU满载后实际虚拟系统未到达性能上限的情况，本章也暂不深入。

1、查看当前运行的虚拟系统名称

[root@localhost ~]# virsh list
setlocale: No such file or directory
Id Name State
----------------------------------------------------
4 vOS2 running

2、查看虚拟机的配置

[root@localhost ~]# virsh dumpxml vOS2
setlocale: No such file or directory
<domain type='kvm' id='4'>
<name>vOS2</name>
<uuid>9c31cd50-f719-4d35-90b2-3044dfdc95bb</uuid>
<memory unit='KiB'>4194304</memory>
<currentMemory unit='KiB'>4194304</currentMemory>
<vcpu placement='static'>2</vcpu>
<resource>
<partition>/machine</partition>
</resource>
<os>
<type arch='x86_64' machine='pc-i440fx-rhel7.0.0'>hvm</type>
<boot dev='hd'/>
</os>
<features>
<acpi/>
<apic/>
</features>
<cpu mode='host-model'>
<model fallback='allow'/>
<topology sockets='1' cores='2' threads='1'/>
</cpu>
<clock offset='utc'>
<timer name='rtc' tickpolicy='catchup'/>
<timer name='pit' tickpolicy='delay'/>
<timer name='hpet' present='no'/>
</clock>
<on_poweroff>destroy</on_poweroff>
<on_reboot>restart</on_reboot>
<on_crash>restart</on_crash>
<pm>
<suspend-to-mem enabled='no'/>
<suspend-to-disk enabled='no'/>
</pm>

...省略大量信息

3、查看qemu进程号

[root@localhost ~]# ps -ef |grep vOS2
qemu 8480 1 51 16:43 ? 00:25:54 /usr/libexec/qemu-kvm -name vOS2 -S -machine pc-i440fx-rhel7.0.0,accel=kvm,usb=off -cpu SandyBridge,+vme,+ds,+acpi,+ss,+ht,+tm,+pbe,+dtes64,+monitor,+ds_cpl,+vmx,+smx,+est,+tm2,+xtpr,+pdcm,+pcid,+dca,+osxsave,+f16c,+rdrand,+arat,+fsgsbase,+smep,+erms,+xsaveopt,+pdpe1gb -m 4096 -realtime mlock=off -smp 2,sockets=1,cores=2,threads=1 -uuid 9c31cd50-f719-4d35-90b2-3044dfdc95bb -no-user-config -nodefaults -chardev socket,id=charmonitor,path=/var/lib/libvirt/qemu/domain-4-vNGFW2/monitor.sock,server,nowait -mon chardev=charmonitor,id=monitor,mode=control -rtc base=utc,driftfix=slew -global kvm-pit.lost_tick_policy=discard -no-hpet -no-shutdown -global PIIX4_PM.disable_s3=1 -global PIIX4_PM.disable_s4=1 -boot strict=on -device ich9-usb-ehci1,id=usb,bus=pci.0,addr=0x6.0x7 -device ich9-usb-uhci1,masterbus=usb.0,firstport=0,bus=pci.0,multifunction=on,addr=0x6 -device ich9-usb-uhci2,masterbus=usb.0,firstport=2,bus=pci.0,addr=0x6.0x1 -device ich9-usb-uhci3,masterbus=usb.0,firstport=4,bus=pci.0,addr=0x6.0x2 -device virtio-serial-pci,id=virtio-serial0,bus=pci.0,addr=0x5 -drive file=/var/lib/libvirt/images/vNGFW2.qcow2,format=qcow2,if=none,id=drive-ide0-0-0 -device ide-hd,bus=ide.0,unit=0,drive=drive-ide0-0-0,id=ide0-0-0,bootindex=1 -drive if=none,id=drive-ide0-0-1,readonly=on -device ide-cd,bus=ide.0,unit=1,drive=drive-ide0-0-1,id=ide0-0-1 -netdev tap,fd=27,id=hostnet0,vhost=on,vhostfd=29 -device virtio-net-pci,netdev=hostnet0,id=net0,mac=52:54:00:28:21:fe,bus=pci.0,addr=0x3 -chardev pty,id=charserial0 -device isa-serial,chardev=charserial0,id=serial0 -chardev spicevmc,id=charchannel0,name=vdagent -device virtserialport,bus=virtio-serial0.0,nr=1,chardev=charchannel0,id=channel0,name=com.redhat.spice.0 -spice port=5900,addr=127.0.0.1,disable-ticketing,image-compression=off,seamless-migration=on -vga qxl -global qxl-vga.ram_size=67108864 -global qxl-vga.vram_size=67108864 -global qxl-vga.vgamem_mb=16 -device intel-hda,id=sound0,bus=pci.0,addr=0x4 -device hda-duplex,id=sound0-codec0,bus=sound0.0,cad=0 -chardev spicevmc,id=charredir0,name=usbredir -device usb-redir,chardev=charredir0,id=redir0,bus=usb.0,port=1 -chardev spicevmc,id=charredir1,name=usbredir -device usb-redir,chardev=charredir1,id=redir1,bus=usb.0,port=2 -device vfio-pci,host=01:00.0,id=hostdev0,bus=pci.0,addr=0x8 -device vfio-pci,host=01:00.1,id=hostdev1,bus=pci.0,addr=0x9 -device virtio-balloon-pci,id=balloon0,bus=pci.0,addr=0x7 -msg timestamp=on
root 10036 7775 0 17:34 pts/3 00:00:00 grep --color=auto vOS2

4、反复查看vcpu信息，最后一列是cpu的编号，可以看到qemu进程是在不同的CPU上不停的漂移的

[root@localhost ~]# ps -eLo ruser,pid,ppid,lwp,psr|grep 8480 
qemu 8480 1 8480 3
qemu 8480 1 8491 2
qemu 8480 1 8494 4
qemu 8480 1 8505 9
[root@localhost ~]# 
[root@localhost ~]# 
[root@localhost ~]# ps -eLo ruser,pid,ppid,lwp,psr|grep 8480
qemu 8480 1 8480 6
qemu 8480 1 8491 2
qemu 8480 1 8494 4
qemu 8480 1 8505 9
[root@localhost ~]# 
[root@localhost ~]# ps -eLo ruser,pid,ppid,lwp,psr|grep 8480
qemu 8480 1 8480 6
qemu 8480 1 8491 2
qemu 8480 1 8494 4
qemu 8480 1 8505 9
qemu 8480 1 10056 11

5、通过绑定进程到cpu比较麻烦，可以直接绑定vCPU到物理CPU。反复通过virsh命令查看vCPU对应的物理CPU，可以看到也是浮动的

[root@localhost ~]# virsh vcpuinfo vOS2
setlocale: No such file or directory
VCPU: 0
CPU: 2
State: running
CPU time: 107.3s
CPU Affinity: yyyyyyyyyyyyyyyyyyyy   //这里看到是宿主机所有物理CPU 核心，Y代表有可以使用，基于CPU时间片来回切换

VCPU: 1
CPU: 0
State: running
CPU time: 116.3s
CPU Affinity: yyyyyyyyyyyyyyyyyyyy

[root@localhost ~]# 
[root@localhost ~]# virsh vcpuinfo vOS2
setlocale: No such file or directory
VCPU: 0
CPU: 3
State: running
CPU time: 107.8s
CPU Affinity: yyyyyyyyyyyyyyyyyyyy

VCPU: 1
CPU: 8
State: running
CPU time: 116.8s
CPU Affinity: yyyyyyyyyyyyyyyyyyyy

[root@localhost ~]# 
[root@localhost ~]# 
[root@localhost ~]# 
[root@localhost ~]# 
[root@localhost ~]# virsh vcpuinfo vOS2
setlocale: No such file or directory
VCPU: 0
CPU: 0
State: running
CPU time: 108.0s
CPU Affinity: yyyyyyyyyyyyyyyyyyyy

VCPU: 1
CPU: 9
State: running
CPU time: 117.1s
CPU Affinity: yyyyyyyyyyyyyyyyyyyy

6、绑定vCPU到物理CPU，这里将vCPU绑定到主CPU上

[root@localhost ~]# virsh vcpupin vOS2 0 2
setlocale: No such file or directory

[root@localhost ~]# virsh vcpupin vOS2 1 4
setlocale: No such file or directory

查看绑定结果，可以看到已经将vCPU0绑定到物理CPU2，vCPU1绑定到物理CPU4上

[root@localhost ~]# virsh vcpuinfo vOS2
setlocale: No such file or directory
VCPU: 0
CPU: 2
State: running
CPU time: 630.1s
CPU Affinity: --y-----------------

VCPU: 1
CPU: 4
State: running
CPU time: 880.7s
CPU Affinity: ----y---------------

[root@localhost ~]# virsh vcpuinfo vOS2
setlocale: No such file or directory
VCPU: 0
CPU: 2
State: running
CPU time: 693.0s
CPU Affinity: --y-----------------

VCPU: 1
CPU: 4
State: running
CPU time: 993.7s
CPU Affinity: ----y---------------

[root@localhost ~]# virsh dumpxml vOS2
setlocale: No such file or directory
<domain type='kvm' id='4'>
<name>vOS2</name>
<uuid>9c31cd50-f719-4d35-90b2-3044dfdc95bb</uuid>
<memory unit='KiB'>4194304</memory>
<currentMemory unit='KiB'>4194304</currentMemory>
<vcpu placement='static'>2</vcpu>
<cputune>
<vcpupin vcpu='0' cpuset='2'/>
<vcpupin vcpu='1' cpuset='4'/>
</cputune>
<resource>
<partition>/machine</partition>
</resource>
<os>
<type arch='x86_64' machine='pc-i440fx-rhel7.0.0'>hvm</type>
<boot dev='hd'/>
</os>
<features>
<acpi/>
<apic/>
</features>
<cpu mode='host-model'>
<model fallback='allow'/>
<topology sockets='1' cores='2' threads='1'/>
</cpu>
<clock offset='utc'>
<timer name='rtc' tickpolicy='catchup'/>
<timer name='pit' tickpolicy='delay'/>
<timer name='hpet' present='no'/>
</clock>
<on_poweroff>destroy</on_poweroff>
<on_reboot>restart</on_reboot>
<on_crash>restart</on_crash>
<pm>

7、再次测试RFC2544性能，对比绑定前后结果：

未绑定CPU，连续测试64,512,1518字节时，测试到1518字节的时候会出现非常不稳定的丢包现象，需要单独测试1518字节才能得到一个测试结果

绑定CPU，多次测试，结果基本稳定