esxtop 指标%RDY，NUMA,Wide-VMs

参考文章：http://www.boche.net/blog/index.php/2010/10/21/cpu-ready-to-rdy-conversion/

http://kb.vmware.com/selfservice/microsites/search.do?language=en_US&cmd=displayKC&externalId=1026063

好文：http://frankdenneman.nl/2010/09/13/esx-4-1-numa-scheduling/

http://bbs.vmanager.cn/thread-3939-1-1.html

http://kb.vmware.com/selfservice/microsites/search.do?language=en_US&cmd=displayKC&externalId=1026063

http://wahlnetwork.com/2011/09/22/numa-woes-battling-high-cpu-rdy-on-vsphere-4/

http://frankdenneman.nl/2010/02/03/sizing-vms-and-numa-nodes/

http://frankdenneman.nl/2010/12/28/node-interleaving-enable-or-disable/

如果是每个node有6core，vm最好分配2,3,6个vcpu

缺省是vm超过8个vcpu会启用vNUMA,但是如果VM分配的vcpu不超过8个，但是仍然大于node中的内核总数，可以对这些vm启用vNUMA

有一些多核处理器的NUMA节点大小不同于每socket的核心数目。例如有些12核心cpu具有2个6核心的NUMA节点。

Some multi-core processors have NUMA node sizes that are different than the number of cores
per socket. For example, some 12-core processors have two six-core NUMA nodes per processor.

First and most importantly, try and size your VMs as a multiple of your physical server’s NUMA node size. For instance, if your physical server has 6 cores per NUMA node, size your VMs as either 2, 3, or 6 way.  This KB article explains the potential impact of mismatched VM sizes to NUMA node sizes in more detail http://kb.vmware.com/kb/1026063

Virtual NUMA

vSphere 5.0 and later support for exposing virtual NUMA topology to guest operating systems, which can improve performance by facilitating guest operating system and application NUMA optimizations.

virtual machines so they align with physical NUMA boundaries. For example, you have a host system with six cores per NUMA node. In this case, size your virtual machines with a multiple of six vCPUs (that is, 6 vCPUs, 12 vCPUs, 18vCPUs, 24 vCPUs, and so on).

By default, vNUMA is enabled only for virtual machines (hardware versions: 8) with more than eight vCPUs. This feature can be enabled for smaller virtual machines, however, by adding to the .vmx file the following line:numa.vcpu.maxPerVirtualNode = X (where X is the number of vCPUs per vNUMA node).

使用虚拟 NUMA

vSphere 5.0 及更高版本包括对向客户机操作系统公开虚拟 NUMA 拓扑的支持，这样便于客户机操作系统和应用程序 NUMA 优化，从而可提高性能。

虚拟 NUMA 拓扑可用于硬件版本 8 的虚拟机，且默认情况下在虚拟 CPU 的数目大于 8 时启用。也可以使用高级配置选项手动影响虚拟 NUMA 拓扑。

可以通过 vSphere Client 中的两种设置来影响虚拟 NUMA 拓扑：虚拟机的虚拟插槽数及每个插槽的内核数。如果每个插槽的内核数 (cpuid.coresPerSocket) 大于 1，且虚拟机中的虚拟内核数大于 8，则虚拟 NUMA 节点大小与虚拟插槽大小相匹配。如果每个插槽的内核数小于或等于 1，则会创建虚拟 NUMA 节点以匹配首个打开虚拟机电源的物理主机的拓扑。

当虚拟 CPU 数目和使用的内存量按比例增长时，可以使用默认值。对于占用不成比例的大量内存的虚拟机，可通过以下方式之一替代默认值：

■

增加虚拟 CPU 的数目，即使未使用此数目的虚拟 CPU 也是如此。请参见更改虚拟 CPU 的数目。

■

使用高级选项通过物理 NUMA 拓扑来控制虚拟 NUMA 拓扑及其映射。请参见虚拟 NUMA 控制。

Figure 2: NON-NUMA vCPU placement

Figure 3: NUMA vCPU placement

Figure 4: esxtop memory totals

如图：上图服务器有64GB内存，2个4核CPU，每个NUMA分配有32GB内存。

第一个NUMA node有13988 free,第二个NUMA node有372 MB free.

在第二个节点上的vm看起来会没有多少内存可以使用了，但是幸运的是可以远程使用node1上的内存。

N%L指示使用本地NUMA node节点上的内存的百分比。少于一定比例会将vm迁移到令一个node以便提高本地内存存取性能。

METRIC	EXPLANATION
NHN	Current Home Node for virtual machine
NMIG	Number of NUMA migrations between two snapshots. It includes balance migration, inter-mode VM swaps performed for locality balancing and load balancing
NRMEM (MB)	Current amount of remote memory being accessed by VM
NLMEM (MB)	Current amount of local memory being accessed by VM
N%L	Current percentage memory being accessed by VM that is local
GST_NDx (MB)	The guest memory being allocated for VM on NUMA node x. “x” is the node number
OVD_NDx (MB)	The VMM overhead memory being allocated for VM on NUMA node x

Transparent page sharing and memory locality.

------------------------------------------------------------------------------------

当考察一个物理主机性能的时候，一个重要的衡量指标是CPU使用率。

而考察一个虚拟机的性能的时候，另一个重要的指标是Ready time.

Ready time 告诉我们guest vm在获得主机分配的CPU资源前等待了多久时间，以 % format的格式

%RDY曾经用来发现CPU性能瓶颈

vSphere Client中也可以监控vm性能指标，可以发现CPU ready这项，以微秒为统计

两种格式需要转换。

NUMA
NUMA stands for Non-Uniform Memory Access, which translates into a variance of memory access latencies. Both AMD Opteron and Intel Nehalem are NUMA architectures. A processor and memory form a NUMA node. Access to memory within the same NUMA node is considered local access, access to the memory belonging to the other NUMA node is considered remote access.

在激活了NUMA的ESXi主机的虚拟机上执行esxtop显示%RDY值过高

[复制链接]

leader.wu 升级  100%

电梯直达 1#  发表于 2011-3-16 13:48:09 | 只看该作者  分享到： 故障状态 1、ESXi主机上的物理CPU数量相对于为其上配置的vCPUs的数量而言，不足； 2、NUMA节点之间的CPU较高使用率失衡； 故障分析 这个问题是由于总的物理CPUs数量和总的vCPUs之间的匹配失衡； 例如： 在4 core/socket的机器上分配了3vCPU虚拟机 在6 core/socket的机器上分配了4/5 vCPU虚拟机 在8 core/socket的机器上分配了5/6/7 vCPU虚拟机 解决方案 计算公式应该是在N核心的机器上为VM分配N/2个vCPU； 禁止掉缺省的RoundRobin初始化放置策略，降低%RDY参数值，命令如下： #esxcfg-advcfg -s 0 /Numa/RoundRobin 复制代码 不要在4 core/socket的机器上分配3vCPU虚拟机 不要在6 core/socket的机器上分配4/5 vCPU虚拟机 不要在8 core/socket的机器上分配5/6/7 vCPU虚拟机 备注：Numa.RoundRobin选项是隐藏的，因此无法通过vSphere Client登录系统查看到，只能利用命令行；

Node interleaving disabled equals NUMA

、、