A series of posts about QEMU internals

Introduction

This is a series of posts about QEMU internals. It won’t cover everything about QEMU, but should help you understand how it works and foremost how to hack into it for fun and profit.

We won’t explain usage and other things that can be found in the official documentation. The following topics will be addressed:

• Creating a new machine
• Controlling memory regions
• Creating a new device
• Interrupts controller
• Timers
• PCI controller
• PCI devices
• Options
• Execution loop
• Breakpoints handling
• VM running states
• TCG internals part 1, part 2 and part 3
• Snapshots

The official code and documentation can be found here:

• https://github.com/qemu/qemu
• https://www.qemu.org/documentation/

Terminology

Host and target

The host is the plaform and architecture which QEMU is running on. Usually an x86 machine.

The target is the architecture which is emulated by QEMU. You can choose at build time which one you want:

./configure --target-list=ppc-softmmu ...

As such, in the source code organisation you will find all supported architectures in the target/ directory:

(qemu-git) ll target
drwxrwxr-x 2 xxx xxx 4.0K alpha
drwxrwxr-x 2 xxx xxx 4.0K arm
drwxrwxr-x 2 xxx xxx 4.0K cris
drwxrwxr-x 2 xxx xxx 4.0K hppa
drwxrwxr-x 3 xxx xxx 4.0K i386
drwxrwxr-x 2 xxx xxx 4.0K lm32
drwxrwxr-x 2 xxx xxx 4.0K m68k
drwxrwxr-x 2 xxx xxx 4.0K microblaze
drwxrwxr-x 2 xxx xxx 4.0K mips
drwxrwxr-x 2 xxx xxx 4.0K moxie
drwxrwxr-x 2 xxx xxx 4.0K nios2
drwxrwxr-x 2 xxx xxx 4.0K openrisc
drwxrwxr-x 3 xxx xxx 4.0K ppc
drwxr-xr-x 3 xxx xxx 4.0K riscv
drwxrwxr-x 2 xxx xxx 4.0K s390x
drwxrwxr-x 2 xxx xxx 4.0K sh4
drwxrwxr-x 2 xxx xxx 4.0K sparc
drwxrwxr-x 2 xxx xxx 4.0K tilegx
drwxrwxr-x 2 xxx xxx 4.0K tricore
drwxrwxr-x 2 xxx xxx 4.0K unicore32
drwxrwxr-x 9 xxx xxx 4.0K xtensa

The qemu-system-<target> binaries are built into their respective <target>-softmmu directory:

(qemu-git) ls -ld *-softmmu
drwxr-xr-x  9 xxx xxx 4096 i386-softmmu
drwxrwxr-x 11 xxx xxx 4096 ppc-softmmu
drwxr-xr-x  9 xxx xxx 4096 x86_64-softmmu

System and user modes

QEMU is a system emulator. It offers emulation of a lot of architectures and can be run on a lot of architectures.

It is able to emulate a full system (cpu, devices, kernel and apps) through the qemu-system-<target> command line tool. This is the mode we will dive into.

It also provides a userland emulation mode through the qemu-<target> command line tool.

This allows to directly run <target> architecture Linux binaries on a Linux host. It mainly emulates <target> instructions set and forward system calls to the host Linux kernel. The emulation is only related to user level cpu instructions, not system ones, no device nore low level memory handling.

We won’t cover qemu user mode in this blog post series.

Emulation, JIT and virtualization

Initially QEMU was an emulation engine, with a Just-In-Time compiler (TCG). The TCG is here to dynamically translate target instruction set architecture (ISA) to host ISA.

We will later see that in the context of the TCG, the tcg-target becomes the architecture to which the TCG has to generate final assembly code to run on (which is host ISA). Obvious !

There exists scenario where target and host architectures are the same. This is typically the case in classical virtualization environment (VMware, VirtualBox, …) when a user wants to run Windows on Linux for instance. The terminology is usually Host and Guest (target).

Nowadays, QEMU offers virtualization through different accelerators. Virtualization is considered an accelerator because it prevents unneeded emulation of instructions when host and target share the same architecture. Only system level (aka supervisor/ring0) instructions might be emulated/intercepted.

Of course, the QEMU virtualization capabilities are tied to the host OS and architecture. The x86 architecture offers hardware virtualization extensions (Intel VMX/AMD SVM). But the host operating system must allow QEMU to take benefit of them.

Under an x86-64 Linux host, we found the following accelerators:

$ qemu-system-x86_64 -accel ?
Possible accelerators: kvm, xen, hax, tcg

While on an x86-64 MacOS host:

$ qemu-system-x86_64 -accel ?
Possible accelerators: tcg, hax, hvf

The supported accelerators can be found in qemu_init_vcpu():

void qemu_init_vcpu(CPUState *cpu)
{
...
    if (kvm_enabled()) {
        qemu_kvm_start_vcpu(cpu);
    } else if (hax_enabled()) {
        qemu_hax_start_vcpu(cpu);
    } else if (hvf_enabled()) {
        qemu_hvf_start_vcpu(cpu);
    } else if (tcg_enabled()) {
        qemu_tcg_init_vcpu(cpu);
    } else if (whpx_enabled()) {
        qemu_whpx_start_vcpu(cpu);
    } else {
        qemu_dummy_start_vcpu(cpu);
    }
...
}

To make it short:

• kvm is the Linux Kernel-based Virtual Machine accelerator;
• hvf is the MacOS Hypervisor.framework accelerator;
• hax is the cross-platform Intel HAXM accelerator;
• whp is the Windows Hypervisor Platform accelerator.

You can take benefit of the speed of x86 hardware virtualization under the three major operating systems. Notice that the TCG is also considered an accelerator. We can enter a long debate about terminology here …

QEMU APIs

There exists a lot of APIs in QEMU, some are obsolete and not well documented. Reading the source code still remains your best option. There is a good overview available.

The posts series will mainly address QOM, qdev and VMState. The QOM is the more abstract one. While QEMU is developped in C language, the developpers chose to implement the QEMU Object Model to provide a framework for registering user creatable types and instantiating objects from those types: device, machine, cpu, … People used to OOP concepts will find their mark in the QOM.

We will briefly illustrate how to make use of it, but won’t detail its underlying implementation. Stay pragmatic !

The interested reader can have a look at include/qom/object.h.

Disclaimer

It shall be noted that Airbus does not commit itself on the exhaustiveness and completeness regarding this blog post series. The information presented here results from the author knowledge and understandings as of QEMU v4.2.0.