openstack 开发step-by-step

Set up your Open Stack

There are several ways to deploy openstack, Devstack is easily for developer to deploy Open Stack.

Please follow this URL to set up your Open Stack.

http://docs.openstack.org/developer/devstack/

nested KVM support:

Kernel
- First check if your system has already configured nested kvm:

root@minicloud-allinone-controller0-i7bnc6baumzl:~# dpkg -S /etc/modprobe.d/qemu-system-x86.conf

qemu-system-x86: /etc/modprobe.d/qemu-system-x86.conf

root@minicloud-allinone-controller0-i7bnc6baumzl:~# cat /etc/modprobe.d/qemu-system-x86.conf

options kvm_intel nested=1

If not, run:

echo “options kvm-intel nested=1″ | sudo tee /etc/modprobe.d/kvm-intel.conf

then reload kvm_intel.ko

rmmod kvm_intel

modprobe kvm_intel

Nova
- Optional: Modify nova.conf and add host-passthrough. Please note it may impact migration.

libvirt_cpu_mode = host-passthrough

Restart nova services.

PCI PassThrough Support

https://access.redhat.com/documentation/en-US/Red_Hat_Enterprise_Linux/5/html/Virtualization/chap-Virtualization-PCI_passthrough.html

There are two ways to support PCI PassThrough.

legacy UIO framework(PCI Assign);
VFIO, it is recommended http://lwn.net/Articles/509153/

For PCI SR-IOV device we have a specially way to setup.

Preparation for PCI PassThrough

(1) Enable the Intel VT-d extensions http://www.linux-kvm.org/page/How_to_assign_devices_with_VT-d_in_KVM

Setup in your bios. Check you have enable it.

$sudo dmesg | grep -e DMAR -e IOMMU

(2) Activate Intel VT-d in the kernel http://www.zerg.sc/technote/?p=241

$ sudo echo 1 > /sys/module/kvm/parameters/allow_unsafe_assigned_interrupts

update the the kernel cmdline.

https://www.gnu.org/software/grub/manual/html_node/Simple-configuration.html

$ sudo su

$ sudo sed -i -e 's/\(^GRUB_CMDLINE_LINUX_DEFAULT.*\)"$/\1 intel_iommu=on"/' /etc/default/grub

$ sudo update-grub

$ sudo grep intel_iommu=on /boot/grub/grub.cfg

Use legacy UIO framework instead of VFIO

Why we still need the legacy UIO framework, for there are some thing wrong with vfio and igb in some PC.

http://www.linux-kvm.org/page/How_to_assign_devices_with_VT-d_in_KVM

Set parameter KVM allow_unsafe_assigned_interrupts

Transient configure (take effect Immediately)

$ sudo echo 1 > /sys/module/kvm/parameters/allow_unsafe_assigned_interrupts

Or persistent

$ sudo echo "options kvm allow_unsafe_assigned_interrupts=1" >>  /etc/modprobe.d/kvm_iommu.conf

Search the PCI device

$ lspci -n

01:00.0 0200: 8086:10fb (rev 01)

01:00.1 0200: 8086:10fb (rev 01)

$ lspci

01:00.0 Ethernet controller: Intel Corporation 82599ES 10-Gigabit SFI/SFP+ Network Connection (rev 01)

01:00.1 Ethernet controller: Intel Corporation 82599ES 10-Gigabit SFI/SFP+ Network Connection (rev 01)

Bind pci_stub driver to PCI device

$ sudo modprobe pci_stub

$ sudo su(take effect Immediately)

$ echo "8086 10fb" > /sys/bus/pci/drivers/pci-stub/new_id

$ echo 0000:01:00.0   > /sys/bus/pci/devices/0000\:01\:00.0/driver/unbind

$ echo 0000:01:00.0  > /sys/bus/pci/drivers/pci-stub/bind

$ echo 0000:01:00.1   > /sys/bus/pci/devices/0000\:01\:00.1/driver/unbind

$ echo 0000:01:00.1  > /sys/bus/pci/drivers/pci-stub/bind

Start attach PCI device to guest and start

in this way, if you want to use virsh to start your guest, you must assign the driver name, or you will failed for libvirt will use vfio driver as default.
name='kvm' means: the legacy pci device assignment handled directly by the KVM kernel module

$ echo "

<hostdev mode='subsystem' type='pci' managed='yes'>

  <driver name='kvm'/>

  <source>

    <address domain='0x0000' bus='0x01' slot='0x00' function='0x1'/>

  </source>

</hostdev>" > new-pci.xml

$ virsh attach-device guest-name --config new-pci.xml

$ virsh start guest-name

Use VFIO

There are tons of tutorials on how to get VT-d working with KVM/libvirt/Qemu/etc. It’s still a fairly new and unused feature, which means that it is getting updated pretty frequently. Unfortunately, the Internet doesn’t update as quickly, so most of the documentation out there is not accurate or the best approach. readlink You can refer this when any trouble.

http://hinish.com/?p=338

We need this command to load the kernel modules at boot time

$ sudo echo "vfio

vfio_iommu_type1

vfio_pci" >> /etc/modules

Set vfio_iommu_type1 allow_unsafe_interrupts parameters If you're overriding the ivrs table to enable intremap you dont need this line Persistent (need reboot.)

$ echo "options vfio_iommu_type1 allow_unsafe_interrupts=1" > /etc/modprobe.d/vfio_iommu_type1.conf

Or transient (take effect Immediately)

$ sudo echo 1 >  /sys/module/vfio_iommu_type1/parameters/allow_unsafe_interrupts

Disable AppArmor for libvirtd

Actually, we do not need to disable Apparmor, this is a bug on ubuntu, we need to work around it. (It may can not works)

https://bugs.launchpad.net/ubuntu/+source/libvirt/+bug/1276719

http://libvirt.org/drvqemu.html#securitysvirtaa

$ apt-get install apparmor-utils

$ sudo aa-complain /usr/sbin/libvirtd

$ sudo su

$ for x in /etc/apparmor.d/libvirt/libvirt-*[0-9a-b];do aa-complain $x; done

Or (It may can not works)

$ sudo ln -s  /etc/apparmor.d/usr.sbin.libvirtd /etc/apparmor.d/disable/

$ sudo ln -s /etc/apparmor.d/usr.lib.libvirt.virt-aa-helper /etc/apparmor.d/disable/

$ sudo apparmor_parser -R /etc/apparmor.d/usr.sbin.libvirtd

$ sudo apparmor_parser -R /etc/apparmor.d/usr.lib.libvirt.virt-aa-helper

$ sudo service libvirt-bin restart

Or remove it (the best way)

http://www.techytalk.info/disable-and-remove-apparmor-on-ubuntu-based-linux-distributions/

$ sudo /etc/init.d/apparmor stop

$ sudo update-rc.d -f apparmor remove

$ sudo apt-get --purge remove apparmor apparmor-utils libapparmor-perl libapparmor1

It may cause the ubuntu-desktop crash, please reinstall it.

$ sudo apt-get install ubuntu-desktop

Find the all the pci devices in sam iommu group

We must binding all devices in same group to the vfio-pci driver creates.

$ readlink /sys/bus/pci/devices/0000\:01\:00.0/iommu_group

../../../../kernel/iommu_groups/1

$ ls -l /sys/bus/pci/devices/0000\:01\:00.0/iommu_group/devices

Bind vfio driver to PCI device

$ sudo su

$ echo "8086 10fb" > /sys/bus/pci/drivers/vfio-pci/new_id

$ echo "0000:01:00.1" > /sys/bus/pci/devices/0000\:01\:00.1/driver/unbind

$ echo "0000:01:00.1" > /sys/bus/pci/drivers/vfio-pci/bind

$ echo "0000:01:00.0" > /sys/bus/pci/devices/0000\:01\:00.0/driver/unbind

$ echo "0000:01:00.0" > /sys/bus/pci/drivers/vfio-pci/bind

Start attach PCI device to guest and start

$ echo "

<hostdev mode='subsystem' type='pci' managed='yes'>

  <driver name='vfio'/>

  <source>

    <address domain='0x0000' bus='0x01' slot='0x00' function='0x1'/>

  </source>

</hostdev>" > new-pci.xml

$ virsh attach-device guest-name --config new-pci.xml

$ virsh start guest-name

Configure for igb module for SR-IOV support.

https://access.redhat.com/documentation/en-US/Red_Hat_Enterprise_Linux/6/html/Virtualization_Host_Configuration_and_Guest_Installation_Guide/sect-Virtualization_Host_Configuration_and_Guest_Installation_Guide-SR_IOV-How_SR_IOV_Libvirt_Works.html

Persistent configure for igb module.

https://communities.intel.com/community/wired/blog/2011/08/31/using-sr-iov-with-intel-ethernet-and-red-hat-enterprise-linux

$ sudo echo 'options igb max_vfs=7' > /etc/modprobe.d/igb.conf

$ sudo echo "igb" >> /etc/modules

$ sudo update-initramfs -k all -t -u

Transient configure for igb module.

$ sudo su

$ sudo rmmod igb && sudo modprobe igb max_vfs=7

It maill failed as follow on PC:

Jan 10 04:42:18 localhost kernel: [ 8441.914658] ixgbe 0000:01:00.1: SR-IOV: bus number out of range

Jan 10 04:42:18 localhost kernel: [ 8441.914662] ixgbe 0000:01:00.1 0000:01:00.1 (uninitialized): Failed to enable PCI sriov: -12

This is intel official document, it is some different from the above.

http://www.intel.com/content/www/us/en/network-adapters/10-gigabit-network-adapters/ethernet-x520-sr-iov-red-hat-tech-brief.html

Work around if pci-passthrough can not work.

Upgrade your kernel.

Download the latest stable kernel. https://www.kernel.org/

$ wget https://www.kernel.org/pub/linux/kernel/v3.x/linux-3.18.1.tar.xz

$ tar -xvf linux-3.18.1.tar.xz

$ cd linux-3.18.1

$ cp /boot/config-3.13.0-43-generic ./.config

$ echo "" | make oldconfig

Please enable CONFIG_IP_NF_NAT the config, kernel 3.17 change the macro define for nat.

$ sed -i -e "s/# CONFIG_IP_NF_NAT.*/CONFIG_IP_NF_NAT=m/" ./.config

$ make -j8

$ make modules_install

$ make install

$ sudo reboot

compile your owen qemu

$ git clone git://git.qemu.org/qemu.git

$ apt-get install libglib2.0-dev zlib1g-dev libspice-server-dev libspice-protocol-dev

$ ./configure '--target-list=x86_64-softmmu' '--enable-debug' '--enable-kvm' '--enable-spice' '--prefix=/home/shhfeng/qemu/'

$ make

re make you initrd

$ gunzip # uncompress your initrd-*.img file

$ mkdir tmp

$ cd tmp/

$ cpio -i --make-directories < "your uncompress  initrd direction."

$ tree -C .

# tree: option -C color show，-n shutdown color show.

$ cd tmp/

$ gzip -dc ../sth.gz |cpio -id

# edit your initrd，

$ cd tmp/

# add or remove *.ko

$ find ./ | cpio -H newc -o > /boot/initrd-2.6.22.6.img-ahci

$ gzip /boot/initrd-2.6.22.6.img-ahci

$ mv initrd-2.6.22.6.img-ahci.gz /boot/initrd-2.6.22.6.img-ahci

Build up your skills to develop Open stack

Doc In Repository

Please read the docs in /opt/stack/nova/doc/source/devref first. Then you will get some ideas about Open Stack. Read up these files, you can ignore them if you already know them.

http://docs.openstack.org/api/quick-start/content/index.html#Compute_API_Quick_Start

gerrit.rst

It introduce the gerrit work flow.

development.environment.rst

It introduce how to set up development environment.

architecture.rst

Introduce the profile of the Open Stack.

rpc.rst

Open Stack rpc is different with the traditional SUN IPC, it base on a AMQP broker. http://www.rabbitmq.com/getstarted.html

Learn more about implementation of rpc. Please seen the Open Stack Common Library.

$ git clone git://git.openstack.org/openstack/oslo.git

Also learn more about kombu:

http://kombu.readthedocs.org/en/latest/

threading.rst

All OpenStack services use *green thread* model of threading, If your know coroutine, you will learn greenthread quickly.

All the above files also can be access by this url link.

http://docs.openstack.org/developer/nova/devref/rpc.html

import component

You had better do some exercise about some technologies that Open Stack uses. This is helpful for you to know Open Stack.

Rabbitmq python tutorials

Tutorial:

https://github.com/rabbitmq/rabbitmq-tutorials/

http://wenku.baidu.com/view/800285ea0975f46527d3e19b.html (a Chinese Document for Rabbitmq)

http://www.openstack.cn/p136.html (a Chinese Simple Document for Rabbitmq)

Tutorial Source Code:

https://github.com/rabbitmq/rabbitmq-tutorials/

sqlalchemy (http://www.sqlalchemy.org/)

Tutorial Source Code:

http://docs.sqlalchemy.org/en/rel_0_9/orm/tutorial.html

WSGI And eventlet

Read the Red hat REST API Guide. It is useful to understand REST API. https://access.redhat.com/documentation/en-US/Red_Hat_Enterprise_Virtualization/3.0/html-single/REST_API_Guide/index.html#sect-REST_API_Guide-Common_Features-Resources-Creating_Resources

Join The Open Stack Community to do Contributions

Join the IRC

IRC: https://wiki.openstack.org/wiki/IRC

Subscribe the mail-list

Dev: http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

General: http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack

To submit the patch

Workflow: https://wiki.openstack.org/wiki/Gerrit_Workflow

Play with Open Stack

DashBoard

http://docs.openstack.org/user-guide/content/log_in_dashboard.html

Try to access http://localhost/

CURL

http://docs.openstack.org/api/quick-start/content/index.html#Compute_API_Quick_Start

I have write a small script to get token ID, as follow, you can name it as “get-token”:

#!/bin/bash

return=`curl -i 'http://127.0.0.1:5000/v2.0/tokens' -X POST -H "Content-Type: application/json" -H "Accept: application/json"  -d '{"auth": {"tenantName": "admin", "passwordCredentials": {"username": "admin", "password": "123"}}}'`

echo $return

echo "-----------------------------------------------------------------------------"

ID=""

for x in $return; do

    if [ "x$ID" = "xtokenID" ] ;then

        id=`echo $x | tr -d '[",]'`

        export TOKEN=$id

        echo "TOKEN=$id"

        ID=""

        break

    fi

    if [ "x$x" = 'x"id":' ] ;then

        ID="tokenID"

    fi

done

echo ""

This script just is a sample, you can improve it.

Run it by

$ . get-token

Then you can run other commands use the TOKEN environment value, such as get all the tenents, as follow:

$ curl -i -X GET http://127.0.0.1:35357/v2.0/tenants -H "User-Agent: python-keystoneclient" -H "X-Auth-Token: $TOKEN"

Open Stack Client

http://docs.openstack.org/api/quick-start/content/index.html#Compute_API_Quick_Start

You should export the OS_USERNAME and OS_TENANT_NAME environment by the follow command firstly.

$ cd devstack/

$ . openrc OS_USERNAME OS_TENANT_NAME

http://docs.openstack.org/juno/config-reference/content/section_compute-scheduler.html

update your /etc/nova/nova.conf

Add the follow option to “[DEFAULT]” section.

$ grep -e DEFAULT -e pci_passthrough -e pci_alias -e scheduler_available_filters /etc/nova/nova.conf

[DEFAULT]

pci_passthrough_whitelist={"product_id":"10fb", "vendor_id":"8086", "deviceids":"0000:01:00.0, 0000:01:00.1"}

pci_passthrough_whitelist={"product_id":"8c20", "vendor_id":"8086", "deviceids":"0000:00:1b.0"}

pci_alias = {"vendor_id":"8086", "product_id":"10fb", "name":"intelnic"}

scheduler_available_filters = nova.scheduler.filters.pci_passthrough_filter.PciPassthroughFilter

Create a instance with pci-passthrough device.

https://wiki.openstack.org/wiki/Pci_passthrough

$ nova flavor-list

$ nova flavor-key m1.tiny set pci_passthrough:alias=intelnic:1

$ nova boot  pci  --flavor m1.tiny  --image=cirros-0.3.2-x86_64-uec

Debug with Open Stack

I usually use “print” for debugging. The “print” is on the stack screen terminal.

$ cd devstack/

$ ./rejoin-stack.sh

$ screen –ls

There is a screen on:

	13846.stack

$ screen -r stack

Some useful operation:

run a command by “Ctrl + A + :”,  you can input help command to show more info.

Swith to any window by “Ctrl + A + '  ”, input the windows number you want to switch.

Swith to any window by “Ctrl + A + "  ”, select the windows you want to switch.

write testcase

https://wiki.openstack.org/wiki/SmallTestingGuide