The project
I've always wondered how programming for an ARM cpu is. So I decided to try to make an OS, written 100% in assembly for an ARM development board. I shouldn't say OS though, every time I write an OS, I really only make: memory management, scheduler, mutex, netcard driver, serial port driver and some small application to run on the "os". It's basically just to learn about the architecture of the device.
The ARMv7 architecture offers a lot of cool features that I am not using. I just want to keep things simple for now. Once I get something working good, I will go deeper in the documentation and try some more advanced stuff.
At first, I wanted to use my beaglebone black to run my OS. But then, I found out that qemu can emulate quite a few boards and it would be easier to do. By using qemu, I get the following advantages over using a real board:
- no need to upload code to the board, I use the image directly
- can reboot the machine easily while working remotely (no need to physically access the board)
- very easy to peek in memory with qemu's monitor command "pmsave"
- can use gdb to debug with qemu
- no need for a separate bootloader. Can boot kernel directly.
I chose to use the "realview-pb-a8" emulated board in qemu. I have never seen that board, I have no idea what it is. It uses a Cortex-A8. So I was able to get a programing guide for that SoC. I started from there.
The fact that I am using qemu makes things easier but removes a lot of fun. For example, qemu boots my kernel directly. On a real board, I would need to write a bootloader (or use u-boot). I would need to initialized SDRAM, initialize clocks and "power domains" and other board initialization. QEMU boots your kernel directly into RAM and you can run from there. So I wouldn't quite call this "bare-metal" programming. I guess I could only call this project "kernel programming for a Cortex-A8".
Getting started
The first step is to create a small test and actually run it. So I created the following program:
Note the qemu command in the Makefile. This allows me to run the test using "make run". It will emulate the ARM board which is the realview-pb-a8
Board specifications
When starting development on a new board, the first thing you need to do is to get a memory map of the device. Because the board will contain sdram, sram, memory mapped peripheral IO etc... From board to board, the physical location of those elements will change. Here is the memory map for the realview-pb-a8
Physical Memory Layout
| Physical address | Description |
|---|---|
| 0x00000000-0x0FFFFFFF | SDRAM mirror |
| 0x10000000-0x1001FFFF | Peripherals |
| 0x10020000-0x1005FFFF | Board specific stuff that I don't need just yet |
| 0x10060000-0x1007FFFF | On board SRAM |
| 0x10080000-0x6FFFFFFF | Board specific stuff that I don't need just yet |
| 0x70000000-0x8FFFFFFF | SDRAM |
| 0x90000000-0xFFFFFFFF | Board specific stuff that I don't need just yet |
A more detailed memory map can be found in the RealView Platform Baseboard for Cortex A8 User Guide.
Booting
Interrupt vector table
This architecture only uses 7 interrupt vectors
| 0x00 | Reset |
| 0x04 | Undefined Instruction |
| 0x08 | Software Interrupt |
| 0x0C | Prefetch Abort |
| 0x10 | Data Abort |
| 0x14 | reserved |
| 0x18 | IRQ |
| 0x1C | FIQ |
The interrupt vector table must be placed at the begining of the memory. Each entry is 32bits wide. It must be an instruction not an address. So you would typically put a branch instruction to jump to the proper handler. Using qemu, my kernel gets loaded at 0x70010000, so putting the IVT at the begining of my kernel would not work. I had to rellocate the IVT to 0x70000000 once the kernel was running. By the way, on that board the SDRAM starts at 0x70000000 but is mirrored to 0x00000000. Still, qemu starts execution at 0x70010000. but if the IVT is at 0x70000000, the CPU will still see it at 0x00000000 because of the mirror.
Setting up the stack
There are 6 CPU modes in this architecture. Each mode will shadow the register r13 (stack pointer). So they each need their own stack. To set those stacks, you must switch mode and set r13 appropriately. I don't set the User mode stack because this will be done on a per-process basis and System mode uses the same registers as user mode.
msr CPSR_c,#0b11010001 // stack for FIQ mode
ldr r13,=STACK_BASE_FIQ
msr CPSR_c,#0b11010010 // stack for IRQ mode
ldr r13,=STACK_BASE_IRQ
msr CPSR_c,#0b11010111 // stack for Abort mode
ldr r13,=STACK_BASE_ABORT
msr CPSR_c,#0b11011011 // stack for Undefined mode
ldr r13,=STACK_BASE_UNDEFINED
msr CPSR_c,#0b11010011 // stack for Supervisor mode. And we will stay in that mode
ldr r13,=STACK_BASE_SUPERVISOR
Memory Management Unit
Creating a paged memory system is not difficult. The MMU offers a 2 level page table system The level1 table has 4096 entries, each mapping 1Mb of virtual addresses. You could create "section" entries to map those 1Mb to physical memory directly. You would then get pages of 1Mb and only 1 table that takes 16k in memory. But if you want 4k pages, then those entries need to be "Coarse table" entries, meaning that each entry will reference a subtable (a level2 table). Each level 2 table contain 256 entries, mapping 4k of memory. So for a 4k paging system you would have 1 Level1 table with 4096 entries (a total of 16k in size) and 4096 level2 tables containing 256 entries each for a total of 4Mb in size.
| Level1, Section | |||||||||||||||||||||||||||||||
|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| 31 | 30 | 29 | 28 | 27 | 26 | 25 | 24 | 23 | 22 | 21 | 20 | 19 | 18 | 17 | 16 | 15 | 14 | 13 | 12 | 11 | 10 | 9 | 8 | 7 | 6 | 5 | 4 | 3 | 2 | 1 | 0 |
| base addr | NS | 0 | nG | S | AP2 | TEX | AP | 0 | domain | XN | C | B | 1 | PXN | |||||||||||||||||
| Level1, Page Table (TODO) | |||||||||||||||||||||||||||||||
|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| 31 | 30 | 29 | 28 | 27 | 26 | 25 | 24 | 23 | 22 | 21 | 20 | 19 | 18 | 17 | 16 | 15 | 14 | 13 | 12 | 11 | 10 | 9 | 8 | 7 | 6 | 5 | 4 | 3 | 2 | 1 | 0 |
| page table addr | 0 | domain | 0 | NS | PXN | 0 | 1 | ||||||||||||||||||||||||
| Level2, Small Page (TODO) | |||||||||||||||||||||||||||||||
|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| 31 | 30 | 29 | 28 | 27 | 26 | 25 | 24 | 23 | 22 | 21 | 20 | 19 | 18 | 17 | 16 | 15 | 14 | 13 | 12 | 11 | 10 | 9 | 8 | 7 | 6 | 5 | 4 | 3 | 2 | 1 | 0 |
| base addr | nG | S | AP[2] | TEX | AP[1:0] | C | B | 1 | XN | ||||||||||||||||||||||
Domains and permissions
The mmu has a concept of domains and access permissions. 16 access domains exist. In a page descriptor, we set the access bits and the domain associated with that page. CP15.register3 contains 2 bits for each domains 0 to 15. These bits determine how page access should be checked. Example: a page is associated to domain 12. CP15.register3 indicates that domain12 is Client. Therefore access permissions in the page will be checked. If domain12 was set to Manager, permissions would have been ignored.
Initializing the MMU
The first thing you need to do is setup the page tables like mentionned above. Obviously, you might want to do an identity mapping for the region of code that is currently running the MMU initialization code so that the mapping does not change after having initialized the MMU.
Level 1 Page Table
70100000 01 40 10 00 01 44 10 00 01 48 10 00 01 4c 10 00
70100010 01 50 10 00 01 54 10 00 01 58 10 00 01 5c 10 00
...
70103fe0 01 20 50 00 01 24 50 00 01 28 50 00 01 2c 50 00
70103ff0 01 30 50 00 01 34 50 00 01 38 50 00 01 3c 50 00
Level 2 Page Tables (all contiguous)
70104000 fe 0f 00 00 fe 1f 00 00 fe 2f 00 00 fe 3f 00 00
70104010 fe 4f 00 00 fe 5f 00 00 fe 6f 00 00 fe 7f 00 00
...
70503fd0 f2 4f ff ff f2 5f ff ff f2 6f ff ff f2 7f ff ff
70503fe0 f2 8f ff ff f2 9f ff ff f2 af ff ff f2 bf ff ff
70503ff0 f2 cf ff ff f2 df ff ff f2 ef ff ff f2 ff ff ff
Then you need to configure the CP15 register.
| Register | Description | Value |
|---|---|---|
| CP15.reg2 | Translation Table Base Register | Load the base address of the Level1 table |
| CP15.reg3 | Domain Access control register | MCR p15, 0, |
| CP15.reg1 | Control | Set bit0 high to enable the MMU. Do this as the last step |
The following registers are also useful but not needed during initialization
| CP15.reg5 | FSR | Read this in you fault hander. It is the fault code |
| CP15.reg6 | FAR | Read this in you fault hander. It is the faulty virtual address |
| CP15.reg8 | Invalidate TLB | Used to invalidate the TLB. invalidate entire TLB: mcr p15, 0, Rd, c8, c7,0 |
| CP15.reg10 | TLB Lockdown | Used mark a TLB entry as persistent so it does not get overwritten by other entries. can increase performance for pages such as those containing interrupt handling code so that the translation is always cached. We could probably use a level1 table entry as a Section of 1Mb for the kernel and lock it down in the TLB. |
Faults
I will not list the different reasons for getting a fault since this is all covered in the reference manual. Basically, if a fault occurs while prefetching an instruction then the Instruction Fetch fault will occur. If the fault occurs while accessing data, a Data fault will occur. The virtual address that caused the fault will be stored in FAR. A more detailed error code will be found in FSR
Processes
To test the multitasking system, I have created some small programs that I build separately and package in the image. The kernel loads the programs in their own process. Ideally, I would have like to create a flash image that contains the kernel at the very begining and then the programs would be appended at the end, kind of like a real hard disk with a bootloader and programs. But I could never get qemu to eumlate a flash file. Even with the "-pflash". The documentation for the board says that when the board is powered on, the flash is mapped at 0x00000000. This will shadow the sdram. To use the sdram,you must remap the flash to some other place. But I could never make that work. I tried creating a flash image and provide it to qemu with the "-pflash" option but that doesn't seem to work. Qemu always wants a kernel file to be provided. I don't know why I can just put my code in a flat binary that would act as flash and get the code running from 0x00000000. The kernel file gets loaded at 0x70010000 which is the sdram. So I am creating a image file containing the kernel and the programs that get loaded in sdram by qemu.
Programs run as domain 1, and in user mode. Their virtual mapping is:
| Level1 table entries (1mb mapping) | Type | Permissions | Description |
|---|---|---|---|
| 0 | Section | PL1 RWX, PL0 - | Kernel code. Identity mapping |
| 1-FF | Section | PL1 -, PL0 - | Unmapped |
| 100 | Page table | PL1 RW, PL0 - | peripherals, and SRAM. Identity mapping |
| 101-1DF | Section | PL1 -, PL0 - | Unmapped |
| 1E0 | Page Table | PL1 RW, PL0 - | Peripherals. Identity mapping |
| 1E1-1FF | Section | PL1 -, PL0 - | Unmapped |
| 200-2FF | Page Table | PL1 RWX, PL0 RWX | Process code |
| 300-6FF | Section | PL1 -, PL0 - | Unmapped |
| 700-8FF | Section | PL1 RWX, PL0 - | Kernel code. Identity mapping |
| 900-EFF | Section | PL1 -, PL0 - | Unmapped |
| F00-FFF | Page tables | PL1 RW, PL0 RW | Process Stack |
The task information page
When creating a process, I add it in a list of process. The list of process is a fixed-size list in kernel memory (accessible by any process in privileged mode) that contains a pointer to the L1 table of the process and several other usefull information for the process. This information is used by the scheduler and is formatted like this:
| Offset | Description |
|---|---|
| 0x0000 | Physical address of the process's L1 page table |
| 0x0004 | saved r13_irq registers |
| 0x0040 | Quantum count |
Using Software Interrupts
When using the SWI instruction, you need to pass it a parameter that would normally be the function number you would want to call. For example: SWI 0x02. Once you are in the SWI handler, you want to get that parameter to know where to dispatch the handler. but the SWI instruction completely ignores the parameter. It is not given to you in any way when you get in your handler. In order to get this, you need to take r14, which contains the return address and substract 4. That would give you the address of the SWI instruction itself. So you can read at that memory area and see what parameter was provided. That is pretty weird in my opinion. I would rather just put the function number in r0 before calling SWI and read r0 once in the handler. That would illimnate an unncessary memory access. Plus, the page at that location will obviously be in the instruction prefetch cache but since we are using "LDM" to load the instruction in a register to read it, it means we will be looking in the data cache. And the page will most probably not be in that cache. So in my project, I will only pass function parameters in a register and ignore the one provided to SWI.
Something that confuses me is that when calling SWI, you enter Supervisor mode. Then you are in privileged mode. That makes sense, But then r13 and r14 gets shadowed. I'm not entirely sure why I would want that. It actually complicates things when multi-tasking. I guess that in some more complex OS design, this is very usefull.
Multi-Tasking
Saving registers of mode X from mode Y
Assume we have a function called schedule(). This function saves the current context, and reloads the context of the next task to run. In my implementation, this function willa lways be called from the IRQ mode. So the schedule function will be called from a non-user mode. The schedule function will need to store the user-mode context (registers r0 to r14). But from the non-user mode of IRQ, registers r13 and r14 are shadowed. r0-r12 will be the same as the user-mode so we need to find a way to save the r13 and r14 of the user mode. For this, the instruction stm/ldm with "^" can be used to store/load the user-mode registers. this will save/load r0-r12 as usual but the r13 and r14 will be the ones of the user mode.
CPSR and SPSR: While in an exception (therefore in a mode different than user or system) the previous cpsr is saved in spsr. Before returning back from the exception, you must reload spsr back into cpsr. This will change the mode automatically, re-enable interrupts etc. To load this and to load r14 in r15 at the same time, look at the notes below about the LDM instruction.
LDM instruction format: Compared to AVR32 and x86, this is pretty complicated in my opinion. The "ldm" instruction has 3 forms. The first form does what it says it does. But the second form which is: ldm Rn,registers_without_r15^ (yes, there is a "^" at the end) loads all user mode registers while you are in a non-user mode. so it is a way to load user registers while they are shadowed. The third form, ldm Rn,registers_with_r15^, will automatically load spsr back into cpsr. You could also use a data instruction with the "S" flag and R15 as a destination. For some reason, it will conveniently reload spsr back into cpsr at the same time... Go figure. For example: movs r15,14; will reload r15 and also reload spsr back into cpsr. I am wondering why they re-purposed a flag like this. That is one small thing that makes me like x86 more than ARM.
Since I re-enable interrupts after entering SWI, the SVC context must be saved also since a context switch could occur while in a service call (that is actually the whole point of re-enabling interrupts in SWI). So my context-switching code also pushes the r13_svc and r14_svc on the the task's IRQ stack.
Context switching
The schedule function needs to do the following:
- save registers r0-r14 user-mode
- save register r14_irq (since this will be done from the IRQ handler)
- save register spsr (which is the usermode cpsr)
- change level1 page table for new process
- flush tlb (unless using ASID)
- restore r0-r14 (for user-mode)
- restore the return address in r14_irq
- restore spsr
- to return, load r14_irq in r15 and spsr into cpsr
Each task have their own stack and they have their own IRQ mode stack. When entering the schedule() function in IRQ mode, the use-mode registers are pushed onto the IRQ stack. The current spsr and r14_irq is also pushed on the IRQ stack. The r13_irq is then saved in a list somewhere. When time comes to restore the task, the page tables are switched back to that task's page tables, and r13_irq is restored from the list. At this point, the task's IRQ stack has been restored. We can then pop everything from it and the context switch is done. Here is a sample of my schedule function
push(r0-r14)^
mrs r0,SPSR
push(r0)
// At this point, whole context is saved on stack
// Determine what is the next task to run
...
// store context: save r13_irq
// r0 points to the entry in the process list (as decribed earlier)
str r13,[r0,#4] // offset 0x04 is r13
// load new page table
ldr r1,[r5] //r5 points to the entry of the next task to run
mcr p15,0,r1,c2,c0,0
// flush TLB (note that there are ways to avoid this
mov r1,#0
mcr p15,0,r1,c8,c7,0
//load r13_irq
ldr r13,[r5,#4]
// now restore context on stack
pop(r0) // this is just SPSR, only reg available to touch is r14
msr SPSR,r0
pop(r0-r14)^
b returnFromInterrupt
This is a sample only. My schedule() function does a bit more than that. But it gives you the general idea.
When r0-r14 will be restored for the user mode, it would restore the task's context as it was before entering the IRQ to schedule(). r13 and r14 of the user mode will be restored and not the banked ones of the currently executing mode.
Note that the TLB must be flushed when reloading the "translation table base register" in CP15 because the cached TLB entries will continue to correspong to the previous mapping. This is a very expensive operation but we can use the concept of ASID by using the CONTEXTIDR register. By setting a unique task ID in CONTEXTIDR, all page translations that gets loaded in the TLB will be tagged with that ID. When doing a lookup, the MMU will ignore entries that do not match the current CONTEXTIDR. So on a context switch, you would change the ID in CONTEXTIDR. This would create duplicate entries in the TLB but with different IDs. So instead of flushing the TLB, entries will be removed only when the TLB is full. See this article for more information about the TLB and ASID.
The schedule() function is called by the timer IRQ handler. But you might want to call it from other places. For example, if a task wants to yield, it should be scheduled out immediately. I could do this in a SWI handler but trying to change context from the SVC mode brings up other challenges. So to keep things simple, I want to do context switches only from the IRQ mode. For this, it is possible to use a "software IRQ". This is well documented in the GIC documentation.