How to build and run ARM Linux on QEMU from scratch

This blog shows how to run ARM Linux on QEMU! This can be used as a base for later projects using this as an environment for kernel development or study!

1. We need to build qemu, at the time of this writing, the latest is qemu 1.6.0, so grab the qemu-1.6.0.tar.bz2. Then extract, configure, build and install qemu.

# wget http://wiki.qemu-project.org/download/qemu-1.6.0.tar.bz2
# tar xvjf qemu-1.6..tar.bz2
# cd qemu-1.6.
# ./configure --enable-sdl --enable-kvm --audio-drv-list=alsa
# make
# sudo make install

Note that the above configuration for qemu "--audio-drv-list=alsa" is required, otherwise you will receive errors such as "oss: Could not initialize DAC" when running the kernel, since the default audio driver is using OSS but on my system it is ALSA.

2. We need to build our own cross compiler for ARM, in my case, I want to play with ARM Versatile Express with the A15 x4 core chip. There is a very good cross compiler making tool, crosstool-ng (I still remember the hard times when I was trying to build my own cross compiler back in the year 2004, when there was no such one-stop tool script thus I had to get packages one by one and build round and round, fixing errors here and there...).

# cd ~
# wget http://crosstool-ng.org/download/crosstool-ng/crosstool-ng-1.18.0.tar.bz2
# tar xf crosstool-ng-1.18..tar.bz2
# cd crosstool-ng-1.18.
# ./configure --prefix=`pwd`/crosstool
# make && make install
# export PATH="${PATH}:`pwd`/crosstool/bin"

After installing crosstool-ng, we can then actually build our own cross compilier toolchain, but first we need to check what defaults the crosstool-ng provides:

# cd ~
# mkdir crosstool
# cd crosstool
# ct-ng list-samples

Here is the full list:

Status Sample name
LN config
MKDIR config.gen
IN config.gen/arch.in
IN config.gen/kernel.in
IN config.gen/cc.in
IN config.gen/binutils.in
IN config.gen/libc.in
IN config.gen/debug.in
[G.X] alphaev56-unknown-linux-gnu
[G.X] alphaev67-unknown-linux-gnu
[G.X] arm-bare_newlib_cortex_m3_nommu-eabi
[G.X] arm-cortex_a15-linux-gnueabi
[G..] arm-cortex_a8-linux-gnueabi
[G..] arm-davinci-linux-gnueabi
[G..] armeb-unknown-eabi
[G.X] armeb-unknown-linux-gnueabi
[G.X] armeb-unknown-linux-uclibcgnueabi
[G..] arm-unknown-eabi
[G..] arm-unknown-linux-gnueabi
[G.X] arm-unknown-linux-uclibcgnueabi
[G.X] armv6-rpi-linux-gnueabi
[G.X] avr32-unknown-none
[G..] bfin-unknown-linux-uclibc
[G..] i586-geode-linux-uclibc
[G.X] i586-mingw32msvc,i686-none-linux-gnu
[G.X] i686-nptl-linux-gnu
[G.X] i686-unknown-mingw32
[G.X] m68k-unknown-elf
[G.X] m68k-unknown-uclinux-uclibc
[G.X] mips64el-n32-linux-uclibc
[G.X] mips64el-n64-linux-uclibc
[G.X] mips-ar2315-linux-gnu
[G..] mipsel-sde-elf
[G..] mipsel-unknown-linux-gnu
[G.X] mips-malta-linux-gnu
[G..] mips-unknown-elf
[G.X] mips-unknown-linux-uclibc
[G..] powerpc-405-linux-gnu
[G.X] powerpc64-unknown-linux-gnu
[G..] powerpc-860-linux-gnu
[G.X] powerpc-e300c3-linux-gnu
[G.X] powerpc-e500v2-linux-gnuspe
[G..] powerpc-unknown-linux-gnu
[G..] powerpc-unknown-linux-uclibc
[G..] powerpc-unknown_nofpu-linux-gnu
[G.X] s390-ibm-linux-gnu
[G.X] s390x-ibm-linux-gnu
[G..] sh4-unknown-linux-gnu
[G..] x86_64-unknown-linux-gnu
[G..] x86_64-unknown-linux-uclibc
[G.X] x86_64-unknown-mingw32
L (Local) : sample was found in current directory
G (Global) : sample was installed with crosstool-NG
X (EXPERIMENTAL): sample may use EXPERIMENTAL features
B (BROKEN) : sample is currently broken

Great, there is a line saying "[G.X] arm-cortex_a15-linux-gnueabi", which indicates ARM Cortex A15 is supported as a sample configuration, we can just use it, but let's check what is the actual confiruation it is:

# ct-ng show-arm-cortex_a15-linux-gnueabi

IN config.gen/arch.in
IN config.gen/kernel.in
IN config.gen/cc.in
IN config.gen/binutils.in
IN config.gen/libc.in
[G.X] arm-cortex_a15-linux-gnueabi
OS : linux-2.6.38.8
Companion libs : gmp-5.0.1 mpfr-3.0.1 ppl-0.11.2 cloog-ppl-0.15.11 mpc-0.9 libelf-0.8.13
binutils : binutils-2.21.1a
C compiler : gcc-linaro-4.7-2013.01 (C,C++,Fortran)
C library : glibc-2.12.1 (threads: nptl)
Tools : dmalloc-5.5.2 duma-2_5_15 gdb-linaro-7.2-2011.05-0 ltrace-0.5.3 strace-4.5.19

Now, its time to actually build the cross compiler

# ct-ng arm-cortex_a15-linux-gnueabi

CONF config/config.in
#
# configuration saved
#

***********************************************************

Initially reported by: dsreed on freenode/#crosstool-ng
URL:

Comment:
Enables experimental support for VFP/NEON hardware floating point.

***********************************************************

WARNING! This sample may enable experimental features.
Please be sure to review the configuration prior
to building and using your toolchain!
Now, you have been warned!

***********************************************************

Now configured for "arm-cortex_a15-linux-gnueabi"

# mkdir ~/src
# mkdir ~/x-tools

Then in "Target options -> Architecture level ->" you will see:

(cortex-a15) Emit assembly for CPU
(cortex-a15) Tune for CPU
(neon-vfpv4) Use specific FPU

Then, build it, which will take a lot of time (I left for a late night sleep)!

# ct-ng build

The next morning you will find the cross compiler is there:

# ls ~/x-tools/arm-cortex_a15-linux-gnueabi/

arm-cortex_a15-linux-gnueabi bin build.log.bz2 include lib libexec share test-suite

# PATH=~/x-tools/arm-cortex_a15-linux-gnueabi/bin:$PATH\
# arm-cortex_a15-linux-gnueabi-gcc -v

Using built-in specs.
COLLECT_GCC=arm-cortex_a15-linux-gnueabi-gcc
COLLECT_LTO_WRAPPER=/home/coryxie/x-tools/arm-cortex_a15-linux-gnueabi/libexec/gcc/arm-cortex_a15-linux-gnueabi/4.7.3/lto-wrapper
Target: arm-cortex_a15-linux-gnueabi
Configured with: /media/coryxie/WorkSpace/qemu/crosstool-ng-1.18.0/.build/src/gcc-linaro-4.7-2013.01/configure --build=x86_64-build_unknown-linux-gnu --host=x86_64-build_unknown-linux-gnu --target=arm-cortex_a15-linux-gnueabi --prefix=/home/coryxie/x-tools/arm-cortex_a15-linux-gnueabi --with-sysroot=/home/coryxie/x-tools/arm-cortex_a15-linux-gnueabi/arm-cortex_a15-linux-gnueabi/sysroot --enable-languages=c,c++,fortran --with-cpu=cortex-a15 --with-tune=cortex-a15 --with-fpu=neon-vfpv4 --with-float=hard --with-pkgversion='crosstool-NG 1.18.0' --enable-__cxa_atexit --disable-libmudflap --disable-libgomp --disable-libssp --disable-libquadmath --disable-libquadmath-support --with-gmp=/media/coryxie/WorkSpace/qemu/crosstool-ng-1.18.0/.build/arm-cortex_a15-linux-gnueabi/buildtools --with-mpfr=/media/coryxie/WorkSpace/qemu/crosstool-ng-1.18.0/.build/arm-cortex_a15-linux-gnueabi/buildtools --with-mpc=/media/coryxie/WorkSpace/qemu/crosstool-ng-1.18.0/.build/arm-cortex_a15-linux-gnueabi/buildtools --with-ppl=/media/coryxie/WorkSpace/qemu/crosstool-ng-1.18.0/.build/arm-cortex_a15-linux-gnueabi/buildtools --with-cloog=/media/coryxie/WorkSpace/qemu/crosstool-ng-1.18.0/.build/arm-cortex_a15-linux-gnueabi/buildtools --with-libelf=/media/coryxie/WorkSpace/qemu/crosstool-ng-1.18.0/.build/arm-cortex_a15-linux-gnueabi/buildtools --with-host-libstdcxx='-static-libgcc -Wl,-Bstatic,-lstdc++,-Bdynamic -lm -L/media/coryxie/WorkSpace/qemu/crosstool-ng-1.18.0/.build/arm-cortex_a15-linux-gnueabi/buildtools/lib -lpwl' --enable-threads=posix --enable-target-optspace --enable-gold --disable-nls --disable-multilib --with-local-prefix=/home/coryxie/x-tools/arm-cortex_a15-linux-gnueabi/arm-cortex_a15-linux-gnueabi/sysroot --enable-c99 --enable-long-long --with-float=hard
Thread model: posix
gcc version 4.7.3 20130102 (prerelease) (crosstool-NG 1.18.0)

3. We then need to use the cross compiler to build our own kernel, at the time of this writing, the latest kernel is linux-3.11.tar.xz, which you can get from kernel.org directly, then extract it.

# cd linux-3.11
# make ARCH=arm CROSS_COMPILE=arm-cortex_a15-linux-gnueabi- vexpress_defconfig
# make ARCH=arm CROSS_COMPILE=arm-cortex_a15-linux-gnueabi- zImage modules

Note that we also need DTS tree from the Linux ARM project:

# git clone git://linux-arm.org/arm-dts.git
# ./scripts/dtc/dtc -O dtb -o rtsm_ve-cortex_a15x4.dtb arm-dts/fast_models/rtsm_ve-cortex_a15x4.dts

4. We can now run the kernel with qemu, without any init program, no rootfs, so you can just see the kernel runs, but will finally crash due to no root!

# qemu-system-arm -kernel arch/arm/boot/zImage -dtb rtsm_ve-cortex_a15x4.dtb -m  -M vexpress-a15 -serial stdio -append "console=ttyAMA0"

Uncompressing Linux... done, booting the kernel.
Booting Linux on physical CPU 0x0
Initializing cgroup subsys cpuset

.....

VFS: Cannot open root device "(null)" or unknown-block(0,0): error -6
Please append a correct "root=" boot option; here are the available partitions:
Kernel panic - not syncing: VFS: Unable to mount root fs on unknown-block(0,0)
CPU: 0 PID: 1 Comm: swapper/0 Not tainted 3.11.0 #1
[<800153b8>] (unwind_backtrace+0x0/0xf8) from [<80011b0c>] (show_stack+0x10/0x14)
[<80011b0c>] (show_stack+0x10/0x14) from [<8035f84c>] (dump_stack+0x70/0x8c)
[<8035f84c>] (dump_stack+0x70/0x8c) from [<8035d150>] (panic+0x94/0x1e8)
[<8035d150>] (panic+0x94/0x1e8) from [<80484fe4>] (mount_block_root+0x1a4/0x258)
[<80484fe4>] (mount_block_root+0x1a4/0x258) from [<8048518c>] (mount_root+0xf4/0x114)
[<8048518c>] (mount_root+0xf4/0x114) from [<804852d8>] (prepare_namespace+0x12c/0x184)
[<804852d8>] (prepare_namespace+0x12c/0x184) from [<80484c60>] (kernel_init_freeable+0x1e4/0x230)
[<80484c60>] (kernel_init_freeable+0x1e4/0x230) from [<8035ba68>] (kernel_init+0xc/0xe4)
[<8035ba68>] (kernel_init+0xc/0xe4) from [<8000e3f8>] (ret_from_fork+0x14/0x3c)

Bomn~~! That's expected!

5. Remember that the first program that Linux tries to execute is “/init“, we can actually create our own init program, which just does some "Hello-World" job!

We can use a ramdisk as the first filesystem that Linux uses as root, using the “initramfs” scheme. More information about ramdisks can be found in the kernel source tree, in the file “Documentation/early-userspace/README“. Our simple init program looks like below:

#include <stdio.h>

void main() {
printf("Hello World!\n");
while();
}

To build it as initramfs, we can use the following commands:

# arm-cortex_a15-linux-gnueabi-gcc -static init.c -o init
# echo init | cpio -o --format=newc > initramfs

We can the modify our qemu command line to add "-initrd initramfs" in it, then it looks like below:

# qemu-system-arm -kernel arch/arm/boot/zImage -dtb rtsm_ve-cortex_a15x4.dtb -m  -M vexpress-a15 -serial stdio -append "console=ttyAMA0" -initrd ../init/initramfs

Uncompressing Linux... done, booting the kernel.
Booting Linux on physical CPU 0x0

......

Freeing unused kernel memory: 196K (80484000 - 804b5000)
Hello World!
input: AT Raw Set 2 keyboard as /devices/smb.1/motherboard.2/iofpga.3/1c060000.kmi/serio0/input/input0
input: ImExPS/2 Generic Explorer Mouse as /devices/smb.1/motherboard.2/iofpga.3/1c070000.kmi/serio1/input/input1

6. Of course in the above step we are trying to create the init program by our own, which is to reinvent the wheel! We all know the famous BusyBox, don't we? Let's use it as our init!

# wget http://busybox.net/downloads/busybox-1.21.1.tar.bz2
# tar xvjf busybox-1.21..tar.bz2
# cd busybox-1.21.
# make ARCH=arm CROSS_COMPILE=arm-cortex_a15-linux-gnueabi- defconfig
# make ARCH=arm CROSS_COMPILE=arm-cortex_a15-linux-gnueabi- install

The last step will actuall install the BusyBox under busybox-1.21.1/_install.

# cd _install
# mkdir proc sys dev etc etc/init.d
# touch etc/init.d/rcS
# vi etc/init.d/rcS

Then add the fllowing:

#!/bin/sh
mount -t proc none /proc
mount -t sysfs none /sys
/sbin/mdev -s

# chmod +x etc/init.d/rcS
# find . | cpio -o --format=newc > ../rootfs.img
# cd ..
# gzip -c rootfs.img > rootfs.img.gz

Now, it's time to use BusyBox as the root!

# qemu-system-arm -kernel arch/arm/boot/zImage -dtb rtsm_ve-cortex_a15x4.dtb -m  -M vexpress-a15 -serial stdio -append "console=ttyAMA0 root=/dev/ram rdinit=/bin/sh" -initrd ../busybox-1.21./rootfs.img.gz

Uncompressing Linux... done, booting the kernel.
Booting Linux on physical CPU 0x0
Initializing cgroup subsys cpuset
Linux version 3.11.0 (coryxie@coryxie-desktop) (gcc version 4.7.3 20130102 (prerelease) (crosstool-NG 1.18.0) ) #1 SMP Wed Sep 18 21:39:58 CST 2013
CPU: ARMv7 Processor [412fc0f1] revision 1 (ARMv7), cr=10c53c7d
CPU: PIPT / VIPT nonaliasing data cache, PIPT instruction cache
Machine: ARM-Versatile Express, model: RTSM_VE_Cortex_A15x4
Memory policy: ECC disabled, Data cache writealloc
PERCPU: Embedded 7 pages/cpu @8090a000 s7296 r8192 d13184 u32768
Built 1 zonelists in Zone order, mobility grouping on. Total pages: 130048
Kernel command line: console=ttyAMA0 root=/dev/ram rdinit=/bin/sh
PID hash table entries: 2048 (order: 1, 8192 bytes)
Dentry cache hash table entries: 65536 (order: 6, 262144 bytes)
Inode-cache hash table entries: 32768 (order: 5, 131072 bytes)
Memory: 512456K/524288K available (3571K kernel code, 166K rwdata, 1016K rodata, 199K init, 138K bss, 11832K reserved)
Virtual kernel memory layout:
vector : 0xffff0000 - 0xffff1000 ( 4 kB)
fixmap : 0xfff00000 - 0xfffe0000 ( 896 kB)
vmalloc : 0xa0800000 - 0xff000000 (1512 MB)
lowmem : 0x80000000 - 0xa0000000 ( 512 MB)
modules : 0x7f000000 - 0x80000000 ( 16 MB)
.text : 0x80008000 - 0x8048318c (4589 kB)
.init : 0x80484000 - 0x804b5c80 ( 200 kB)
.data : 0x804b6000 - 0x804dfa60 ( 167 kB)
.bss : 0x804dfa60 - 0x80502550 ( 139 kB)
SLUB: HWalign=64, Order=0-3, MinObjects=0, CPUs=4, Nodes=1
Hierarchical RCU implementation.
NR_IRQS:16 nr_irqs:16 16
GIC CPU mask not found - kernel will fail to boot.
GIC CPU mask not found - kernel will fail to boot.
sched_clock: 32 bits at 1000kHz, resolution 1000ns, wraps every 4294967ms
sched_clock: 32 bits at 24MHz, resolution 41ns, wraps every 178956ms
Console: colour dummy device 80x30
Calibrating delay loop... 904.39 BogoMIPS (lpj=4521984)
pid_max: default: 32768 minimum: 301
Mount-cache hash table entries: 512
CPU: Testing write buffer coherency: ok
/cpus/cpu@0 missing clock-frequency property
/cpus/cpu@1 missing clock-frequency property
/cpus/cpu@2 missing clock-frequency property
/cpus/cpu@3 missing clock-frequency property
CPU0: thread -1, cpu 0, socket 0, mpidr 80000000
Setting up static identity map for 0x803639d0 - 0x80363a28
CPU1: failed to boot: -38
CPU2: failed to boot: -38
CPU3: failed to boot: -38
Brought up 1 CPUs
SMP: Total of 1 processors activated (904.39 BogoMIPS).
CPU: All CPU(s) started in SVC mode.
NET: Registered protocol family 16
DMA: preallocated 256 KiB pool for atomic coherent allocations
hw-breakpoint: debug architecture 0x0 unsupported.
Serial: AMBA PL011 UART driver
1c090000.uart: ttyAMA0 at MMIO 0x1c090000 (irq = 37) is a PL011 rev1
console [ttyAMA0] enabled
1c0a0000.uart: ttyAMA1 at MMIO 0x1c0a0000 (irq = 38) is a PL011 rev1
1c0b0000.uart: ttyAMA2 at MMIO 0x1c0b0000 (irq = 39) is a PL011 rev1
1c0c0000.uart: ttyAMA3 at MMIO 0x1c0c0000 (irq = 40) is a PL011 rev1
bio: create slab <bio-0> at 0
SCSI subsystem initialized
usbcore: registered new interface driver usbfs
usbcore: registered new interface driver hub
usbcore: registered new device driver usb
Advanced Linux Sound Architecture Driver Initialized.
Switched to clocksource arm,sp804
NET: Registered protocol family 2
TCP established hash table entries: 4096 (order: 3, 32768 bytes)
TCP bind hash table entries: 4096 (order: 3, 32768 bytes)
TCP: Hash tables configured (established 4096 bind 4096)
TCP: reno registered
UDP hash table entries: 256 (order: 1, 8192 bytes)
UDP-Lite hash table entries: 256 (order: 1, 8192 bytes)
NET: Registered protocol family 1
RPC: Registered named UNIX socket transport module.
RPC: Registered udp transport module.
RPC: Registered tcp transport module.
RPC: Registered tcp NFSv4.1 backchannel transport module.
Unpacking initramfs...
Freeing initrd memory: 2012K (88000000 - 881f7000)
jffs2: version 2.2. (NAND) © 2001-2006 Red Hat, Inc.
msgmni has been set to 1004
io scheduler noop registered (default)
clcd-pl11x: probe of 1c1f0000.clcd failed with error -22
usbcore: registered new interface driver usb-storage
mousedev: PS/2 mouse device common for all mice
rtc-pl031 1c170000.rtc: rtc core: registered pl031 as rtc0
mmci-pl18x 1c050000.mmci: mmc0: PL181 manf 41 rev0 at 0x1c050000 irq 41,42 (pio)
usbcore: registered new interface driver usbhid
usbhid: USB HID core driver
aaci-pl041 1c040000.aaci: ARM AC'97 Interface PL041 rev0 at 0x1c040000, irq 43
aaci-pl041 1c040000.aaci: FIFO 512 entries
oprofile: no performance counters
oprofile: using timer interrupt.
TCP: cubic registered
NET: Registered protocol family 17
VFP support v0.3: implementor 41 architecture 4 part 30 variant f rev 0
rtc-pl031 1c170000.rtc: setting system clock to 2013-09-18 16:34:11 UTC (1379522051)
ALSA device list:
#0: ARM AC'97 Interface PL041 rev0 at 0x1c040000, irq 43
Freeing unused kernel memory: 196K (80484000 - 804b5000)
input: AT Raw Set 2 keyboard as /devices/smb.1/motherboard.2/iofpga.3/1c060000.kmi/serio0/input/input0
/bin/sh: can't access tty; job control turned off
/ # input: ImExPS/2 Generic Explorer Mouse as /devices/smb.1/motherboard.2/iofpga.3/1c070000.kmi/serio1/input/input1
/ # ls
bin dev etc linuxrc proc root sbin sys usr
/ # ls /bin/sh
/bin/sh

7. References:

1. http://shyuanliang.blogspot.com/2013/08/crosstool-ng-toolchain.html
2. http://briolidz.wordpress.com/2012/02/07/building-embedded-arm-systems-with-crosstool-ng/
3. http://siryogi.wordpress.com/2013/05/05/busybox-run-on-qemu/
4. http://www.spinics.net/lists/arm-kernel/msg191186.html
5. http://balau82.wordpress.com/2012/03/31/compile-linux-kernel-3-2-for-arm-and-emulate-with-qemu/