Mirantis OpenStack 7.0: NFVI Deployment Guide — NUMA/CPU pinning
https://www.mirantis.com/blog/mirantis-openstack-7-0-nfvi-deployment-guide-numacpu-pinning/
Compute hosts configuration
To enable CPU Pinning, perform the following steps on every compute host where you want CPU pinning to be enabled.
- Upgrade QEMU to 2.4 to use NUMA CPU pinning (see the Appendix A1 “Installing qemu 2.1”).
- Get the NUMA topology for the node:
# lscpu | grep NUMA
NUMA node(s): 2
NUMA node0 CPU(s): 0-5,12-17
NUMA node1 CPU(s): 6-11,18-23 - 在/etc/default/grub添加以下来告诉系统哪些cores只能被虚拟机使用,而不能被host os使用:
GRUB_CMDLINE_LINUX="$GRUB_CMDLINE_LINUX isolcpus=1-5,7-23”
- 同时把相同的list添加到 vcpu_pin_set in /etc/nova/nova.conf:
vcpu_pin_set=1-5,7-23
In this example we ensured that cores 0 and 6 will be dedicated to the host system. Virtual machines will use cores 1-5 and 12-17 on NUMA cell 1, and cores 7-11 and 18-23 on NUMA cell 2.
- Update boot record and reboot compute node:
update-grub
reboot
Nova configuration
- 创建 aggregates for instances with and without cpu pinning:
# nova aggregate-create performance
# nova aggregate-set-metadata performance pinned=true# nova aggregate-create normal
# nova aggregate-set-metadata normal pinned=false - Add one or more hosts to the new aggregates:
# nova aggregate-add-host performance node-9.domain.tld
# nova aggregate-add-host normal node-10.domain.tld - Create a new flavor for VMs that require CPU pinning:
# nova flavor-create m1.small.performance auto 2048 20 2
# nova flavor-key m1.small.performance set hw:cpu_policy=dedicated
# nova flavor-key m1.small.performance set aggregate_instance_extra_specs:pinned=true - To be thorough, you should update all other flavours so they will start only on hosts without CPU pinning:
# openstack flavor list -f csv|grep -v performance |cut -f1 -d,| \
tail -n +2| xargs -I% -n 1 nova flavor-key % set aggregate_instance_extra_specs:pinned=false - On every controller add values AggregateInstanceExtraSpecFilter and NUMATopologyFilter to the scheduler_default_filters parameter in /etc/nova/nova.conf:
scheduler_default_filters=RetryFilter,AvailabilityZoneFilter,RamFilter,DiskFilter,ComputeFilter,ComputeCapabilitiesFilter,ImagePropertiesFilter,ServerGroupAntiAffinityFilter,ServerGroupAffinityFilter,NUMATopologyFilter,AggregateInstanceExtraSpecsFilter
- Restart nova scheduler service on all controllers:
restart nova-scheduler
Using CPU pinning
Once you’ve done this configuration, using CPU Pinning is straightforward. Follow these steps:
- Start a new VM with a flavor that requires pinning …
# nova boot --image TestVM --nic net-id=`openstack network show net04 -f value | head -n1` --flavor m1.small.performance test1
… and check its vcpu configuration:
# hypervisor=`nova show test1 | grep OS-EXT-SRV-ATTR:host | cut -d\| -f3`
# instance=`nova show test1 | grep OS-EXT-SRV-ATTR:instance_name | cut -d\| -f3`
# ssh $hypervisor virsh dumpxml $instance |awk '/vcpu placement/ {p=1}; p; /\/numatune/ {p=0}’<vcpu placement='static'>2</vcpu>
<cputune>
<shares>2048</shares>
<vcpupin vcpu='0' cpuset='16'/>
<vcpupin vcpu='1' cpuset='4'/>
<emulatorpin cpuset='4,16'/>
</cputune>
<numatune>
<memory mode='strict' nodeset='0'/>
<memnode cellid='0' mode='strict' nodeset='0'/>
</numatune>You should see that each vCPU is pinned to a dedicated CPU core, which is not used by the host operating system, and that these cores are inside the same host NUMA cell (in our example it’s cores 4 and 16 in NUMA cell 1).
- Repeat the test for the instance with two NUMA cells:
# nova flavor-create m1.small.performance-2 auto 2048 20 2
# nova flavor-key m1.small.performance-2 set hw:cpu_policy=dedicated
# nova flavor-key m1.small.performance-2 set aggregate_instance_extra_specs:pinned=true
# nova flavor-key m1.small.performance-2 set hw:numa_nodes=2 把instance pin到2个NUMA cell上
# nova boot --image TestVM --nic net-id=`openstack network show net04 -f value | head -n1` --flavor m1.small.performance-2 test2
# hypervisor=`nova show test2 | grep OS-EXT-SRV-ATTR:host | cut -d\| -f3`
# instance=`nova show test2 | grep OS-EXT-SRV-ATTR:instance_name | cut -d\| -f3`
# ssh $hypervisor virsh dumpxml $instance |awk '/vcpu placement/ {p=1}; p; /\/numatune/ {p=0}’<vcpu placement='static'>2</vcpu>
<cputune>
<shares>2048</shares>
<vcpupin vcpu='0' cpuset='2'/>
<vcpupin vcpu='1' cpuset='10'/>
<emulatorpin cpuset='2,10'/>
</cputune>
<numatune>
<memory mode='strict' nodeset='0-1'/>
<memnode cellid='0' mode='strict' nodeset='0'/>
<memnode cellid='1' mode='strict' nodeset='1'/>
</numatune>
You should see that each vCPU is pinned to a dedicated CPU core, which is not used by the host operating system, and that these cores are inside another host NUMA cell. In our example it’s core 2 in NUMA cell 1 and core 10 in NUMA cell 2. As you may remember in our configuration, cores 1-5 and 12-17 from cell 1 and cores 7-11 and 18-23 from cell 2 are available to virtual machines.
Troubleshooting
You might run into the following errors:
internal error: No PCI buses available in /etc/nova/nova.conf
In this case, you’ve specified the wrong hw_machine_type in /etc/nova/nova.conf
libvirtError: unsupported configuration
Per-node memory binding is not supported with this version of QEMU. You may have an older version of qemu, or a stale libvirt cache.
- http://docs.openstack.org/developer/nova/testing/libvirt-numa.html
Mirantis OpenStack 7.0: NFVI Deployment Guide — NUMA/CPU pinning的更多相关文章
- Mirantis OpenStack 8.0 版本大概性分析
作为 OpenStack 领域标杆性企业之一的 Mirantis 在2016年3月初发布了最新的 MOS 8.0 版本.本文试着基于公开资料进行一些归纳分析. 1. 版本概况 1.1 概况 社区版本: ...
- Mirantis OpenStack 8.0 版本
作为 OpenStack 领域标杆性企业之一的 Mirantis 在2016年3月初发布了最新的 MOS 8.0 版本.本文试着基于公开资料进行一些归纳分析. 1. 版本概况 1.1 概况 社区版本: ...
- Mirantis MCP 1.0:OpenStack 和 Kubernetes 整合的第一步
1.前言 Mirantis 公司在2014年9月14日宣布收购 TCPCloud,然后宣布在2017年第一季度会推出全新的私有云产品.从那时候开始,我就一直满怀期待.终于,今年4月19日,Mirant ...
- Carrier-Grade Mirantis OpenStack (the Mirantis NFV Initiative), Part 1: Single Root I/O Virtualization (SR-IOV)
The Mirantis NFV initiative aims to create an NFV ecosystem for OpenStack, with validated hardware ...
- RedHat 和 Mirantis OpenStack 产品的版本和功能汇总和对比(持续更新)
Mirantis 和 Red Hat 作为 OpenStack 商业化产品领域的两大领军企业,在行业内有重要的地位.因此,研究其产品版本发布周期和所支持的功能,对制定 OpenStack 产品的版本和 ...
- Zabbix3.0.4监控Windows的CPU使用百分比并在CPU使用率超过90%触发报警
Zabbix3.0.4监控Windows的CPU使用百分比 Zabbix 自带的模块没有 CPU 使用率(百分比)这个监控项,我们可以通过添加计数器的方式实现 CPU 百分比的监控. 1.在Zabbi ...
- windows下在virtualbox中的Fuel Openstack 9.0 安装过程
一.材料: 1.软件: virtualbox xshell(或putty,winscp) bootstrap.zip(580MB) mirrors(3.01GB) MirantisOpenStack- ...
- 学习OpenStack之 (0):基础知识
vi 方向键出现字母问题解决方法 执行命令 sudo apt-get remove vim-common 执行命令 sudo apt-get install vim 鼠标被virtualbox捕获无法 ...
- Mirantis OpenStack HA
Mysql使用Galera做Active/Active集群,同时使用Pacemaker,因为Galera mysql用到了领导机选举机制quorum,所以控制节点至少三个 RabbitMQ使用mirr ...
随机推荐
- 前台传递给后台的JSON字符串中的引号 “” 在JAVA后台被转义为 "
前台传递给后台的JSON字符串中的引号 "" 在JAVA后台被转义为 " 1.问题: 前台数据,JSON字符串带有引号 "" ,数据被传递到后台 ...
- sqlserver 脚本和批处理指令小结
一.脚本基础 1.USE语句 设置当前数据库. 2.声明变量 语法:DECLARE @变量名 变量类型 在声明变量后,给变量赋值之 ...
- 巨蟒python全栈开发linux之centos6 第二篇
1. .nginx负载均衡的实现 .准备三台机器,准备3台虚拟机,或者和俩同桌交流一下 192.168.226.128 是nginx资源服务器,返回页面的 192.168.226.129 用作ngin ...
- Linux中的欢迎信息
本地终端欢迎信息 /etc/issue \d 显示当前系统日期 \s 显示操作系统名称 \l 显示终端的终端号,这个比较常用 \m 显示硬件体系结构,如i386.i68 ...
- 转载:阮一峰 RESTful API设计指南
阮一峰先生的文章对我理解启发很大,转载到我的博客 http://www.ruanyifeng.com/blog/2014/05/restful_api.html 网络应用程序,分为前端和后端两个部分. ...
- .net全部版本的官方下载地址
https://technet.microsoft.com/zh-cn/5a4x27ek
- hadoop学习(一)概念理解
1.概念 1.1什么是hadoop? hadoop 是大数据存储和处理的框架,主要组成为文件存储系统hdfs和分布式计算框架mapreduce. 1.2能做什么,擅长做什么,不擅长做什么? 1.2.1 ...
- pyhton3 hashlib模块
hashlib模块提供一下常量属性 hashlib.algorithms_guaranteed 获取保证在所有平台上此模块支持的哈希算法名称的集合 hashlib.algorithms_availab ...
- Array排序方法sort()中的大坑
sort() 方法用于对数组的元素进行排序. 但是排序结果就有点坑了,都不按常规出牌的: // 看上去正常的结果: ['Google', 'Apple', 'Microsoft'].sort(); / ...
- 【HackerRank】 Find Digits
Find Digits Problem Statement Given a number you have to print how many digits in that number exactl ...