https://www.mirantis.com/blog/mirantis-openstack-7-0-nfvi-deployment-guide-numacpu-pinning/

Compute hosts configuration

To enable CPU Pinning, perform the following steps on every compute host where you want CPU pinning to be enabled.

  1. Upgrade QEMU to 2.4 to use NUMA CPU pinning (see the Appendix A1 “Installing qemu 2.1”).
  2. Get the NUMA topology for the node:
    # lscpu  | grep NUMA
    NUMA node(s):          2
    NUMA node0 CPU(s):     0-5,12-17
    NUMA node1 CPU(s):     6-11,18-23
  3. 在/etc/default/grub添加以下来告诉系统哪些cores只能被虚拟机使用,而不能被host os使用:
    GRUB_CMDLINE_LINUX="$GRUB_CMDLINE_LINUX isolcpus=1-5,7-23”
  4. 同时把相同的list添加到 vcpu_pin_set in /etc/nova/nova.conf:
    vcpu_pin_set=1-5,7-23

    In this example we ensured that cores 0 and 6 will be dedicated to the host system. Virtual machines will use cores 1-5 and 12-17 on NUMA cell 1, and cores 7-11 and 18-23 on NUMA cell 2.

  5. Update boot record and reboot compute node:
    update-grub
    reboot

Nova configuration

  1. 创建 aggregates for instances with and without cpu pinning:

    # nova aggregate-create performance
    # nova aggregate-set-metadata performance pinned=true
    # nova aggregate-create normal
    # nova aggregate-set-metadata normal pinned=false
  2. Add one or more hosts to the new aggregates:
    # nova aggregate-add-host performance node-9.domain.tld
    # nova aggregate-add-host normal node-10.domain.tld
  3. Create a new flavor for VMs that require CPU pinning:
    # nova flavor-create m1.small.performance auto 2048 20 2
    # nova flavor-key m1.small.performance set hw:cpu_policy=dedicated
    # nova flavor-key m1.small.performance set aggregate_instance_extra_specs:pinned=true
  4. To be thorough, you should update all other flavours so they will start only on hosts without CPU pinning:
    # openstack flavor list -f csv|grep -v performance |cut -f1 -d,| \
    tail -n +2| xargs -I% -n 1 nova flavor-key % set aggregate_instance_extra_specs:pinned=false
  5. On every controller add values AggregateInstanceExtraSpecFilter and NUMATopologyFilter to the scheduler_default_filters parameter in /etc/nova/nova.conf:
    scheduler_default_filters=RetryFilter,AvailabilityZoneFilter,RamFilter,DiskFilter,ComputeFilter,ComputeCapabilitiesFilter,ImagePropertiesFilter,ServerGroupAntiAffinityFilter,ServerGroupAffinityFilter,NUMATopologyFilter,AggregateInstanceExtraSpecsFilter
  6. Restart nova scheduler service on all controllers:
    restart nova-scheduler

Using CPU pinning

Once you’ve done this configuration, using CPU Pinning is straightforward.  Follow these steps:

  1. Start a new VM with a flavor that requires pinning …

    # nova boot --image TestVM --nic net-id=`openstack network show net04 -f value | head -n1` --flavor m1.small.performance test1

    … and check its vcpu configuration:

    # hypervisor=`nova show test1 | grep OS-EXT-SRV-ATTR:host | cut -d\| -f3`
    # instance=`nova show test1 | grep OS-EXT-SRV-ATTR:instance_name | cut -d\| -f3`
    # ssh $hypervisor virsh dumpxml $instance |awk '/vcpu placement/ {p=1}; p; /\/numatune/ {p=0}’
      <vcpu placement='static'>2</vcpu>
      <cputune>
        <shares>2048</shares>
        <vcpupin vcpu='0' cpuset='16'/>
        <vcpupin vcpu='1' cpuset='4'/>
       <emulatorpin cpuset='4,16'/>
      </cputune>
      <numatune>
        <memory mode='strict' nodeset='0'/>
        <memnode cellid='0' mode='strict' nodeset='0'/>
      </numatune>

    You should see that each vCPU is pinned to a dedicated CPU core, which is not used by the host operating system, and that these cores are inside the same host NUMA cell (in our example it’s cores 4 and 16 in NUMA cell 1).

  2. Repeat the test for the instance with two NUMA cells:
    # nova flavor-create m1.small.performance-2 auto 2048 20 2
    # nova flavor-key m1.small.performance-2 set hw:cpu_policy=dedicated
    # nova flavor-key m1.small.performance-2 set aggregate_instance_extra_specs:pinned=true
    # nova flavor-key m1.small.performance-2 set hw:numa_nodes=2 把instance pin到2个NUMA cell上
    # nova boot --image TestVM --nic net-id=`openstack network show net04 -f value | head -n1` --flavor m1.small.performance-2 test2
    # hypervisor=`nova show test2 | grep OS-EXT-SRV-ATTR:host | cut -d\| -f3`
    # instance=`nova show test2 | grep OS-EXT-SRV-ATTR:instance_name | cut -d\| -f3`
    # ssh $hypervisor virsh dumpxml $instance |awk '/vcpu placement/ {p=1}; p; /\/numatune/ {p=0}’
      <vcpu placement='static'>2</vcpu>
      <cputune>
        <shares>2048</shares>
        <vcpupin vcpu='0' cpuset='2'/>
        <vcpupin vcpu='1' cpuset='10'/>
        <emulatorpin cpuset='2,10'/>
      </cputune>
      <numatune>
        <memory mode='strict' nodeset='0-1'/>
        <memnode cellid='0' mode='strict' nodeset='0'/>
        <memnode cellid='1' mode='strict' nodeset='1'/>
      </numatune>

You should see that each vCPU is pinned to a dedicated CPU core, which is not used by the host operating system, and that these cores are inside another host NUMA cell.  In our example it’s core 2 in NUMA cell 1 and core 10 in NUMA cell 2. As you may remember in our configuration, cores 1-5 and 12-17 from cell 1 and cores 7-11 and 18-23 from cell 2 are available to virtual machines.

Troubleshooting

You might run into the following errors:

  • internal error: No PCI buses available in /etc/nova/nova.conf

    In this case, you’ve specified the wrong hw_machine_type in /etc/nova/nova.conf

  • libvirtError: unsupported configuration

    Per-node memory binding is not supported with this version of QEMU.  You may have an older version of qemu, or a stale libvirt cache.

  • http://docs.openstack.org/developer/nova/testing/libvirt-numa.html

Mirantis OpenStack 7.0: NFVI Deployment Guide — NUMA/CPU pinning的更多相关文章

  1. Mirantis OpenStack 8.0 版本大概性分析

    作为 OpenStack 领域标杆性企业之一的 Mirantis 在2016年3月初发布了最新的 MOS 8.0 版本.本文试着基于公开资料进行一些归纳分析. 1. 版本概况 1.1 概况 社区版本: ...

  2. Mirantis OpenStack 8.0 版本

    作为 OpenStack 领域标杆性企业之一的 Mirantis 在2016年3月初发布了最新的 MOS 8.0 版本.本文试着基于公开资料进行一些归纳分析. 1. 版本概况 1.1 概况 社区版本: ...

  3. Mirantis MCP 1.0:OpenStack 和 Kubernetes 整合的第一步

    1.前言 Mirantis 公司在2014年9月14日宣布收购 TCPCloud,然后宣布在2017年第一季度会推出全新的私有云产品.从那时候开始,我就一直满怀期待.终于,今年4月19日,Mirant ...

  4. Carrier-Grade Mirantis OpenStack (the Mirantis NFV Initiative), Part 1: Single Root I/O Virtualization (SR-IOV)

    The Mirantis NFV initiative aims to create an NFV ecosystem for OpenStack, with validated  hardware ...

  5. RedHat 和 Mirantis OpenStack 产品的版本和功能汇总和对比(持续更新)

    Mirantis 和 Red Hat 作为 OpenStack 商业化产品领域的两大领军企业,在行业内有重要的地位.因此,研究其产品版本发布周期和所支持的功能,对制定 OpenStack 产品的版本和 ...

  6. Zabbix3.0.4监控Windows的CPU使用百分比并在CPU使用率超过90%触发报警

    Zabbix3.0.4监控Windows的CPU使用百分比 Zabbix 自带的模块没有 CPU 使用率(百分比)这个监控项,我们可以通过添加计数器的方式实现 CPU 百分比的监控. 1.在Zabbi ...

  7. windows下在virtualbox中的Fuel Openstack 9.0 安装过程

    一.材料: 1.软件: virtualbox xshell(或putty,winscp) bootstrap.zip(580MB) mirrors(3.01GB) MirantisOpenStack- ...

  8. 学习OpenStack之 (0):基础知识

    vi 方向键出现字母问题解决方法 执行命令 sudo apt-get remove vim-common 执行命令 sudo apt-get install vim 鼠标被virtualbox捕获无法 ...

  9. Mirantis OpenStack HA

    Mysql使用Galera做Active/Active集群,同时使用Pacemaker,因为Galera mysql用到了领导机选举机制quorum,所以控制节点至少三个 RabbitMQ使用mirr ...

随机推荐

  1. 前台传递给后台的JSON字符串中的引号 “” 在JAVA后台被转义为 "

    前台传递给后台的JSON字符串中的引号 "" 在JAVA后台被转义为 &quot 1.问题: 前台数据,JSON字符串带有引号 "" ,数据被传递到后台 ...

  2. sqlserver 脚本和批处理指令小结

    一.脚本基础     1.USE语句          设置当前数据库.     2.声明变量          语法:DECLARE @变量名 变量类型          在声明变量后,给变量赋值之 ...

  3. 巨蟒python全栈开发linux之centos6 第二篇

    1. .nginx负载均衡的实现 .准备三台机器,准备3台虚拟机,或者和俩同桌交流一下 192.168.226.128 是nginx资源服务器,返回页面的 192.168.226.129 用作ngin ...

  4. Linux中的欢迎信息

    本地终端欢迎信息 /etc/issue \d     显示当前系统日期 \s     显示操作系统名称 \l      显示终端的终端号,这个比较常用 \m    显示硬件体系结构,如i386.i68 ...

  5. 转载:阮一峰 RESTful API设计指南

    阮一峰先生的文章对我理解启发很大,转载到我的博客 http://www.ruanyifeng.com/blog/2014/05/restful_api.html 网络应用程序,分为前端和后端两个部分. ...

  6. .net全部版本的官方下载地址

    https://technet.microsoft.com/zh-cn/5a4x27ek

  7. hadoop学习(一)概念理解

    1.概念 1.1什么是hadoop? hadoop 是大数据存储和处理的框架,主要组成为文件存储系统hdfs和分布式计算框架mapreduce. 1.2能做什么,擅长做什么,不擅长做什么? 1.2.1 ...

  8. pyhton3 hashlib模块

    hashlib模块提供一下常量属性 hashlib.algorithms_guaranteed 获取保证在所有平台上此模块支持的哈希算法名称的集合 hashlib.algorithms_availab ...

  9. Array排序方法sort()中的大坑

    sort() 方法用于对数组的元素进行排序. 但是排序结果就有点坑了,都不按常规出牌的: // 看上去正常的结果: ['Google', 'Apple', 'Microsoft'].sort(); / ...

  10. 【HackerRank】 Find Digits

    Find Digits Problem Statement Given a number you have to print how many digits in that number exactl ...