NUMA and vNUMA

NUMA and vNUMA

posted by szamosattila on march 04, 2012
Tutorial, Virtualization

With the spread of SMP (Symmetric Multi-Processing) systems a new scalability issue came up: CPU-memory communication became a bottleneck (again.) In spite of the newest and fastest multicore CPUs, bandwidth of Northbridge wasn’t enough to take advantage of new developments.

This is where NUMA came to the stage. NUMA (Non-Uniform Memory Access) introduced local memory and remote memory. DIMMs have no longer been connected to the IOH, but to the CPU itself. With the apperance of Nehalem processors Intel departed the traditional Front-Side Bus architecture and started to build integrated memory controllers into its processors with two or three memory channels. This new architecture meant that processors were able to read/write from/to local DIMMs faster than DIMMs attached to other CPUs.

With the appearance of NUMA, operating systems and applications had to be prepared for taking advantage of this technology, so a lot of popular OSs and ESX(i) became NUMA aware. The scheduling of NUMA is fully transparent to administrators, ESXi deals with placing vCPUs onto physical CPU cores. If a VM has equal or less vCPUs than the number of cores that a NUMA node has, then the scheduler of ESXi assigns these vCPUs to cores within the same NUMA node. This results in fast memory access speed, because guest memory will be allocated to the virtual machine from the same NUMA node. What if we have VMs with a greater number of vCPUs than the number of cores a NUMA node has? Then so called wide virtual machines are going to be created. In this case a virtual machine cannot fit into a single NUMA node, so this VM has local and remote memory at the same time. This can cause performance degradation in specific systems. If a virtual machine has more vCPUs than the number of cores a NUMA node has, but less virtual processors (because of hyperthreading), there is an option to tell the scheduler to use hyperthreaded logical processors instead of physical cores in another physical CPU. This results in the usage of only local memory modules, which could be useful for memory intensive applications. You can set this option per VM basis with the following flag:

numa.vcpu.preferHT

So basically that is NUMA all about. But what about vNUMA? NUMA and vNUMA are the same technology, the only difference is that while NUMA is for ESXi, vNUMA is for the guest OS.

vNUMA is a new feature of vSphere 5.0. By default, vNUMA is only enabled for virtual machines with more than 8 vCPUs. If vNUMA is enabled for a VM, the guest OS running inside can take advantage of NUMA technology. Homogeneousity of host servers is very important when using vNUMA. A vNUMA-enabled virtual machine uses a NUMA topology that the underlying physical server has when the VM is powered-on. If the VM is vMotioned to another host, the new host’s NUMA topology might be different from the previous one, which could result in performance degradation.

- See more at: http://blog.szamosattila.hu/index.php/2012/03/numa-and-vnuma/#sthash.2gQCkYjg.dpuf