openstack PCI透传(GPU)

描述

kolla-ansible部署openstack的GPU透传方法

一、gpu物理服务器配置

在gpu服务器上主启用IOMMU

确认内核⽀支持iommu

$ cat /proc/cmdline | grep iommu
输出以下信息：BOOT_IMAGE=/vmlinuz-3.10.0-1062.el7.x86_64 root=/dev/mapper/centos-root ro crashkernel=auto rd.lvm.lv=centos/root rhgb quiet intel_iommu=on
如果没有输出, 则需要修改kernel启动参数,接着下面操作。

对于Intel cpu：GRUB_CMDLINE_LINUX_DEFAULT="intel_iommu=on"

对于AMD cpu：GRUB_CMDLINE_LINUX_DEFAULT="iommu=pt iommu=1"

下面以intel cpu 为修改例子

$ vim /etc/default/grub
GRUB_TIMEOUT=5
GRUB_DISTRIBUTOR="$(sed 's, release .*$,,g' /etc/system-release)"
GRUB_DEFAULT=saved
GRUB_DISABLE_SUBMENU=true
GRUB_TERMINAL_OUTPUT="console"
GRUB_CMDLINE_LINUX="crashkernel=auto rd.lvm.lv=centos/root rhgb quiet intel_iommu=on"
GRUB_DISABLE_RECOVERY="true"

重新生成grub

• 系统采用EFI启动生成方式：grub2-mkconfig -o /boot/efi/EFI/centos/grub.cfg
• 系统采用非EFI启动生成方式： grub2-mkconfig -o /boot/grub2/grub.cfg

将下列内容加入到blacklist中以避免被gpu服务器占用

$ vim  /etc/modprobe.d/blacklist.conf
blacklist snd_hda_intel
blacklist amd76x_edac
blacklist vga16fb
blacklist nouveau
blacklist rivafb
blacklist nvidiafb
blacklist rivatv
blacklist nvidia

重新生成initramfs

$ mv /boot/initramfs-$(uname -r).img /boot/initramfs-$(uname -r).img.bak
$ dracut -v /boot/initramfs-$(uname -r).img $(uname -r)

重启系统

$ reboot

二、配置openstack

确认pci设备驱动信息

$ lspci -vnn | grep -i nvi
0c:00.0 VGA compatible controller [0300]: NVIDIA Corporation Device [10de:2204] (rev a1) (prog-if 00 [VGA controller])
	Subsystem: NVIDIA Corporation Device [10de:1454]
0c:00.1 Audio device [0403]: NVIDIA Corporation Device [10de:1aef] (rev a1)
	Subsystem: NVIDIA Corporation Device [10de:1454]

其中[10de:2204]的10de为NVIDIA pci设备的vendor id, 2204为product id, 0c:00.0为设备地址

增加 PciPassthroughFilter 和 available_filters = nova.scheduler.filters.all_filters

$ vim /etc/kolla/config/nova.conf
*******
[filter_scheduler]
enabled_filters = RetryFilter,AvailabilityZoneFilter,AggregateInstanceExtraSpecsFilter,ComputeFilter,ComputeCapabilitiesFilter,NUMATopologyFilter,PciPassthroughFilter,ImagePropertiesFilter,ServerGroupAntiAffinityFilter,ServerGroupAffinityFilter,AggregateCoreFilter,DifferentHostFilter,SameHostFilter
available_filters = nova.scheduler.filters.all_filters

添加pci设备显卡信息

$ vim /etc/kolla/config/nova.conf
*******
[pci]
alias = { "vendor_id":"10de", "product_id":"2204", "device_type":"type-PCI","name":"Nvidia3090a" }
passthrough_whitelist = { "vendor_id": "10de", "product_id": "2204", "address": "0000:0c:00.0" }

reconfigure nova组件

$ kolla-ansible -i /etc/ansible/hosts -t nova deploy

配置flavor并用这个flavor创建虚拟机测试

$ openstack flavor set GPU-16C32G-e --property pci_passthrough:alias='Nvidia3090a:1'

参考文献

自动化kolla-ansible部署openstack+GPU透传方法