Developing bare-metal hypervisor for aarch64

In this article I will talk about virtualization and virtualization concepts in aarch64. After, I will talk about how we can build our very basic hypervisor for aarch64 architecture.

What Is Virtualization

Small motivation : We need to find a way to use the full power of a physical computer by distributing its resources to multiple users or environments. Very important for servers, better use of hardware resource.

Virtualization is:

• An illusion
• An abstraction layer for distrubiting computer resources. (CPU, memory, interrupts)

Keywords

• We have virtual machines , abstraction of traditional systems (works like native devices)
• Now we have hypervisor to control virtual machines.
• We have hardware supports like Intel VT-x to support hypervisors.

aarch64

• CPU Virtulization: exception levels
• Memory Virtualization: stage 2 translation
• Virtual interrupts

CPU Virtualization

aarch64 is much more clear architecture in terms of virtualization than Intel. Even it has a dedicated hypervisor level that we can use. In aarch64 we have different exception levels (like rings in x86). From now on I will use ARM for aarch64.

From this perspective we need to put baremetal hypervisor in EL2 and virtual machines into EL1.

Memory Virtualization

We have two stage address translation, since virtual machine's operating system needs to have physical and virtual address spaces. For example in virtual machine there will be a conversion:

• OS controlled address translation ⇒ stage 1
• Hypervisor controlled address translation =⇒ stage 2
• Intermediate Physical Address (IPA)

environment setup

We will use Qemu for ARM emulation (qemu-system-aarch64). Qemu supports HYPERVISOR mode so we can emulate our hypervisor.

sudo apt-get install qemu qemu-system-aarch64

Now we will develop a hypervisor for ARM architecture so we need a proper compiler. In this project, I will use aarch64-linux-gnu .

sudo apt-get install gcc-aarch64-linux-gnu

booting minimal operating system

developing bare-metal hypervisor is similar to developing a operating system. we need to boot our kernel, configure memory region, specific registers etc. to create an hello world application we need to 3 steps:

• entry assembly
• jumping to c code
• linker file

our assembly file is very basic and just jumps to the c program (this file will be changed during hypervisor development).

.global start
start:
    ldr x17, =stack_top                     // setup stack
    mov sp, x17                             // move stack pointer    

    // Enable Interrupts
    // ------------------
    MSR      DAIFClr, #0x3

    bl main                                 // jump to c program

after jumpting the c file we can print some messages over uart.

But to be simple if we can write ( 0x09000000 ) memory mapped register our messages should be printed over the uart.

void print_uart0(const char *s) {
    while(*s != '\0') {
        *UART0DR = (unsigned int)(*s);
        s++;
    }
}
print_uart0("hello hypervisor\n");

A very minimal linker file.

ENTRY(_vm_enter)
SECTIONS
{
. = 0x40000000;
.text : {
    *(.text)
    *(.rodata)
    *(.eh_frame)
}
.data : {
    *(.data)
}
.bss : {
    *(.bss COMMON)
}
. = ALIGN(8);
. = . + 0x1000;
stack_top = .;
}

After the complitaion process we see that:

exception handling

exception handling is very important concept for an operating system, also for an hypervisor. for an formal description:

" Exceptions are conditions or system events that require some action by privileged software (an exception handler) to ensure smooth functioning of the system . (from arm docs)"

processor needs to handle this exceptions which includes:

• IRQ (interrupt request): hardware signal which stops executing program https://en.wikipedia.org/wiki/Interrupt_request_(PC_architecture)
• FIQ (fast interrupt request): higher priority than, used for network etc.

to overcome this exception, we need an exception handler which is stored in exception vector . Exception vector is an table which directs to each exception to specific function. an example:

.global vector
vector:
// ----------------------------------------
sp_el0_sync:        b       start
.align 7 , 0xff // 2^7
sp_el0_irq:         b       interrupt_handler_a

.align 7 , 0xff // 2^7
sp_el0_fiq:         b       interrupt_handler_a

.align 7 , 0xff // 2^7
sp_el0_serror:      b       interrupt_handler_a
// ----------------------------------------
.align 7 , 0xff // 2^7
sp_elx_sync:        b       interrupt_handler_a

.align 7 , 0xff // 2^7
sp_elx_irq:         b       interrupt_handler_a

.align 7 , 0xff // 2^7      // EL2 Timer-IRQ
sp_elx_fiq:         b       interrupt_handler_a

.align 7 , 0xff // 2^7
sp_elx_serror:      b       interrupt_handler_a
// ----------------------------------------

This was the end of the blog post. You can reach me via email umusasadik at gmail com