---
title: xvisor
categories: embedded, arm, armv8, arm64, xvisor, hypervisor, virtualization
...
協作者
=====================================
* 2015 年春季
- 沈宗穎, 李育丞, 蘇誌航, 張仁傑, 鄧維岱
Hackpad
=====================================
* `xvisor<https://mycpp.hackpad.com/xvisor-xQUUkRHWwmm>`_
* `design doc整理<https://mycpp.hackpad.com/Xvisor--WzicRuv3qqf>`_
* `報告整理區<https://mycpp.hackpad.com/Xvisor--nSiBMBB9RNN>`_
設計文件整理
======================================
虛擬化技術 (Virtualization)
=====================================
以下簡介虛擬化技術，並以Xvisor on `ARMv8</embedded/ARMv8>`_ 為例探討虛擬化技術的實作
Hypervisor (virtual machine monitor)
--------------------------------------------
- Hypervisor (Virtual Machine Monitor)的功用是去管理Virtual Machine (VM)
- Host Machine: 運行Hypervisor的實體主機，但有時運行在Hypervisor之上
- Guest Machine: 運行在Hypervisor之上的虛擬主機
.. image:: http://electronicdesign.com/site-files/electronicdesign.com/files/archive/electronicdesign.com/files/29/20211/fig_01.jpg
- 使用時機:
- 工作負載整合 (Workload Consolidation)
- 支援舊有軟體 (Legacy Software)
- 啟用多核 (Multicore Enablement)
- 提高可靠性 (Improved Reliability)
- 安全監控 (Secure Monitoring)
Hypervisor的類型
--------------------------------------------
- Type-1: 本地(native)、裸機(bare-metal) hypervisors
- Hypervisor直接運行在host的硬體上，並直接控制硬體及管理Guest作業系統，此時host把Guest作業系統當成一個process
- 如: `XenServer<http://en.wikipedia.org/wiki/Xen>`_ 、 `Hyper-V<http://en.wikipedia.org/wiki/Hyper-V>`_ 、Xvisor
- Type-2: 託管(hosted) hypervisors
- Hypervisor運行在host的作業系統上，再去提供虛擬化服務
- 如: `VMware<http://en.wikipedia.org/wiki/VMware_Workstation>`_、`VirtualBox<http://en.wikipedia.org/wiki/VirtualBox>`_
- 比較:
- Type-1: 有較高的安全性及可靠性，但因直接運行於硬體上，所以時常為single purpose
- Type-2: 支援較多的I/O device及服務，且較易安裝及使用，但效能較Type-1低，常應用於效率較不重要的客戶端
虛擬化指令(ARM)
--------------------------------------------
虛擬化定理 (Virtualization theorems)
................................................................................
- `Popek and Goldberg virtualization requirements<http://en.wikipedia.org/wiki/Popek_and_Goldberg_virtualization_requirements>`_
- 虛擬化需要滿足:
- 等價性 (Equivalence): 在hypervisor下執行的程式行為必須與直接跑在machine上相同。
- 資源控制 (Resource control): hypervisor必須完全控制虛擬化資源。
- 效率 (Efficiency): 統計上經常使用的機器指令hypervisor不應該介入。
- 定理：對於任何傳統的第三代計算機，只要其敏感指令是特權指令的一個子集，就可以為其建立VMM (from Wiki)
- 原本由OS在kernel mode執行的敏感指令，因OS被移到user mode而無法正常執行，所以需要被trap 給 hypervisor來執行。
- 特權指令 (Privileged instructions): 若執行在user mode 會觸發trap
- 控制敏感指令 (Control sensitive instructions): 會改變處理器組態或模式的指令
- 行為敏感指令 (Behavior sensitive instructions): 其行為取決於處理器的狀態
問題指令(Problematic Instructions)
....................................
- Type I: 在user mode執行會產生未定義的指令異常
- MCR、MRC: 需要依賴協處理器(coprocessor)
- Type II: 在user mode執行會沒有作用
- MSR、MRS: 需要操作系統暫存器
- Type III: 在user mode執行會產生不可預測的行為
- MOVS PC, LR: 返回指令，改變PC並跳回user mode，在user mode執行會產生不可預測的結果
- ARM 的敏感指令:
- 存取協處理器: MRC / MCR / CDP / LDC / STC
- 存取SIMD/VFP 系統暫存器: VMRS / VMSR
- 進入TrustZone 安全狀態: SMC
- 存取 Memory-Mapped I/O: Load/Store instructions from/into memory-mapped I/O locations
- 直接存取CPSR: MRS / MSR / CPS / SRS / RFE / LDM (conditional execution) / DPSPC
- 間接存取CPSR: LDRT / STRT – Load/Store Unprivileged (“As User”)
- 存取Banked Register: LDM / STM
Solutions
....................................
- 軟體技術: trap and emulate
- Dynamic Binary Translation
- 把問題指令取代為hypercall以進行trap 及emulate
- Hypercall
- 對type and original instruction bits進行編碼
- trap 到 hypervisor，進行解碼及模擬指令
- 硬體技術:
- 特權指令轉換(待補)
- MMU強制執行trap
- 虛擬化擴充
記憶體虛擬化(without hardware support)
.............................................
- Shadow page tables:
- Map guest virtual address to host physical address
- Guest OS maintain自己的page table到 guest實體記憶體框架
- hypervisor 把所有的guest實體記憶體框架 map 到host實體記憶體框架
.. image:: /embedded/shadow_page_table.png
(from System Virtualization Memory Virtualization - 國立清華大學)
- 為每一個guest page table 建立 Shadow page table
- hypervisor要保護放著guest page table的host frame
.. image:: /embedded/write_protect.png
(from System Virtualization Memory Virtualization - 國立清華大學)
ARM Virtualization Extensions
--------------------------------------------
可參考:`ARMv8#虛擬化<http://wiki.csie.ncku.edu.tw/embedded/ARMv8#%E8%99%9B%E6%93%AC%E5%8C%96-virtualization>`_ (暫存器待補)
CPU virtualization
...................................
- ARM 增加運行在Non-secure privilege level 2 的 Hypervisor mode
- CPU 虛擬化擴充
- Guest OS kernel執行在EL1，userspace執行在EL0
- 使大部分的敏感指令可以本地執行(native-run)在EL1上而不必trap及emulation
- 而仍需要trap的敏感指令會被trap到EL2 (hypervisor mode HYP)
- Guest OS's Load/Store
- 會影響其他Guest OS的指令
- Hypervisor Syndrome Register(HSR) 會保存被trapped的指令的資訊，因此hypervisor就能emulate它
* `Xvisor- cpu_vcpu_helper.c<https://github.com/xvisor/xvisor/blob/master/arch/arm/cpu/arm64/cpu_vcpu_helper.c>`_:
.. code-block:: c
/* Initialize Hypervisor Configuration */
INIT_SPIN_LOCK(&arm_priv(vcpu)->hcr_lock);
arm_priv(vcpu)->hcr = (HCR_TSW_MASK |
HCR_TACR_MASK |
HCR_TIDCP_MASK |
HCR_TSC_MASK |
HCR_TWE_MASK |
HCR_TWI_MASK |
HCR_AMO_MASK |
HCR_IMO_MASK |
HCR_FMO_MASK |
HCR_SWIO_MASK |
HCR_VM_MASK);
將EL1的敏感指令(MCR、MRC、SMC、WFE、WFI)及中斷(irq、fiq)trap到EL2，並啟動stage 2 address translation
Xvisor instruction emulate
............................................................
cpu_entry.S 內初始化Hyp vector base
.. code-block:: c
vectors:
ventry hyp_sync_invalid /* Synchronous EL1t */
ventry hyp_irq_invalid /* IRQ EL1t */
ventry hyp_fiq_invalid /* FIQ EL1t */
ventry hyp_error_invalid /* Error EL1t */
ventry hyp_sync /* Synchronous EL1h */
ventry hyp_irq /* IRQ EL1h */
ventry hyp_fiq_invalid /* FIQ EL1h */
ventry hyp_error_invalid /* Error EL1h */
........
EXCEPTION_HANDLER hyp_sync
PUSH_REGS
mov x1, EXC_HYP_SYNC_SPx
CALL_EXCEPTION_CFUNC do_sync
PULL_REGS
/*
* .macro PUSH_REGS
* sub sp, sp, #0x20
* push x28, x29
* push x26, x27
* push x24, x25
* push x22, x23
* ......
* push x0, x1
* add x21, sp, #0x110
* mrs x22, elr_el2
* mrs x23, spsr_el2
* stp x30, x21, [sp, #0xF0]
* stp x22, x23, [sp, #0x100]
*
*
* .macro CALL_EXCEPTION_CFUNC cfunc
* mov x0, sp x0 放下面 arch_regs_t *regs 的參數
* bl \cfunc
* .endm
*/
- ESR_EL2, Exception Syndrome Register: 保存跳到EL2的exception的syndrome 資訊
cpu_interrupt.c
(請參考手冊p.1905頁對照ESR的編碼)
.. code-block:: c
void do_sync(arch_regs_t *regs, unsigned long mode)
{
.......
esr = mrs(esr_el2);
far = mrs(far_el2);
elr = mrs(elr_el2);
ec = (esr & ESR_EC_MASK) >> ESR_EC_SHIFT;
il = (esr & ESR_IL_MASK) >> ESR_IL_SHIFT;
iss = (esr & ESR_ISS_MASK) >> ESR_ISS_SHIFT;
.......
switch (ec) {
case EC_UNKNOWN:
/* We dont expect to get this trap so error */
rc = VMM_EFAIL;
break;
case EC_TRAP_WFI_WFE:
/* WFI emulation */
rc = cpu_vcpu_emulate_wfi_wfe(vcpu, regs, il, iss);
break;
case EC_TRAP_MCR_MRC_CP15_A32:
/* MCR/MRC CP15 emulation */
rc = cpu_vcpu_emulate_mcr_mrc_cp15(vcpu, regs, il, iss);
break;
.........
break;
case EC_TRAP_HVC_A64:
/* HVC emulation for A64 guest */
rc = cpu_vcpu_emulate_hvc64(vcpu, regs, il, iss);
break;
case EC_TRAP_MSR_MRS_SYSTEM:
/* MSR/MRS/SystemRegs emulation */
rc = cpu_vcpu_emulate_msr_mrs_system(vcpu, regs, il, iss);
break;
case EC_TRAP_LWREL_INST_ABORT:
/* Stage2 instruction abort */
fipa = (mrs(hpfar_el2) & HPFAR_FIPA_MASK) >> HPFAR_FIPA_SHIFT;
fipa = fipa << HPFAR_FIPA_PAGE_SHIFT;
fipa = fipa | (mrs(far_el2) & HPFAR_FIPA_PAGE_MASK);
rc = cpu_vcpu_inst_abort(vcpu, regs, il, iss, fipa);
break;
case EC_TRAP_LWREL_DATA_ABORT:
/* Stage2 data abort */
fipa = (mrs(hpfar_el2) & HPFAR_FIPA_MASK) >> HPFAR_FIPA_SHIFT;
fipa = fipa << HPFAR_FIPA_PAGE_SHIFT;
fipa = fipa | (mrs(far_el2) & HPFAR_FIPA_PAGE_MASK);
rc = cpu_vcpu_data_abort(vcpu, regs, il, iss, fipa);
break;
最後呼叫 emulate_arm.c 或 cpu_vcpu_emulate.c內相對應的函式做指令模擬
Memory virtualization
...................................
請參考: `armv8 virtual-memory-system-architecture <http://wiki.csie.ncku.edu.tw/embedded/ARMv8#%E8%99%9B%E6%93%AC%E8%A8%98%E6%86%B6%E9%AB%94%E7%B3%BB%E7%B5%B1%E6%9E%B6%E6%A7%8B-virtual-memory-system-architecture>`_
- ARM 增加 Intermediate Physical Address，使得Guest OS不能直接存取實體位址(physical address)
- 二階位址轉換 two stage address translation: => 實體位址(physical address)
- 第一階段: 虛擬位址(virtual address) => 中間實體位址(Intermediate physical address)
- 由Guest OS控制，並認為IPA就是PA
- 第二階段: 中間實體位址(Intermediate physical address) => 實體位址(physical address)
- 由hypervisor控制
interrupt virtualization
........................................
GIC(Generic Interrupt Controller)是ARM裡唯一的Interrupt Controller，其中包含一個Interrupt Distributor韌體和一個CPU Interface。
- Interrupt Distributor：
- 可以將不同類型的interrupt安排(route)到該類型的state
- 需在booting time被設定
ARM新增了一個硬體元件Virtual CPU Interface，並透過他提供Virtual Interrupt。如此一來，便不需要去模擬I/O Device，Guest OS也不用trap進Hypervisor便能acknowledge and clear interrupts。不過Hypervisor還是需要模擬Virtual Interrupt Distributor，以提供Guest OS存取trap。
- Virtual Interrupt Distributor：
- 將interrupt分為兩類，分別被安排(route)到Hypervisor或Guest OS的Vector Table
- Virtual CPU interface：
- 協助Hypervisor Designer實作Virtual Interrupt Distributor的硬體
I/O device virtualization
--------------------------------------------
- ARM 增加 Virtual Generic Interrupt Controller 介面去執行interrupt
virtio
.............................
Xvisor
=========================================
booting 流程
------------------------
可參考 `AJ NOTE<https://arm4fun.hackpad.com/ARM64-Booting-in-Xvisor-EdnevstgC0t>`_
- 需要在MMU啟動前在virtual space下啟動Xvisor
- 在assembly time時, 把img加入至.text,使用.incbin
.. image:: /embedded/xvisor_memory.png
* `cpu_entry.S<https://github.com/xvisor/xvisor/blob/master/arch/arm/cpu/arm64/cpu_entry.S>`_
.. code-block:: c
_start_mmu_init:
/* Setup SP as-per load address */
ldr x0, __hvc_stack_end
mov sp, x0
sub sp, sp, x6
add sp, sp, x4
.........
bl _setup_initial_ttbl
* `mmu_lpae.c<https://github.com/xvisor/xvisor/blob/6807a137dbcbef5b182c78b95986079a610af81d/arch/arm/cpu/common/mmu_lpae.c>`_
.. code-block:: c
u8 __attribute__ ((aligned(TTBL_TABLE_SIZE))) def_ttbl[TTBL_INITIAL_TABLE_SIZE] = { 0 };
初始化translation table
* `mmu_lpae_entry_ttbl.c<https://github.com/xvisor/xvisor/blob/master/arch/arm/cpu/common/mmu_lpae_entry_ttbl.c>`_
.. code-block:: c
void __attribute__ ((section(".entry")))
_setup_initial_ttbl(virtual_addr_t load_start, virtual_addr_t load_end,
virtual_addr_t exec_start, virtual_addr_t exec_end)
{
..........
lpae_entry.ttbl_base = to_load_pa((virtual_addr_t)&def_ttbl); /* def_ttbl之後要放到 ttbr0_el2(Translation Base Register) 裡*/
lpae_entry.next_ttbl = (u64 *)lpae_entry.ttbl_base;
..........
/* Map physical = logical
* Note: This mapping is using at boot time only
*/
__setup_initial_ttbl(&lpae_entry, load_start, load_end, load_start,
AINDEX_NORMAL_WB, TRUE);
/* Map to logical addresses which are
* covered by read-only linker sections
* Note: This mapping is used at runtime
*/
SETUP_RO_SECTION(lpae_entry, text);
SETUP_RO_SECTION(lpae_entry, init);
SETUP_RO_SECTION(lpae_entry, cpuinit);
SETUP_RO_SECTION(lpae_entry, spinlock);
SETUP_RO_SECTION(lpae_entry, rodata);
/* Map rest of logical addresses which are
* not covered by read-only linker sections
* Note: This mapping is used at runtime
*/
__setup_initial_ttbl(&lpae_entry, exec_start, exec_end, load_start,
AINDEX_NORMAL_WB, TRUE);
}
void __attribute__ ((section(".entry")))
__setup_initial_ttbl(struct mmu_lpae_entry_ctrl *lpae_entry,
virtual_addr_t map_start, virtual_addr_t map_end,
virtual_addr_t pa_start, u32 aindex, bool writeable)
{
........
u64 *ttbl;
/* align start addresses */
map_start &= TTBL_L3_MAP_MASK; /* 0xFFFFFFFFFFFFF000ULL 後面12的bit 直接map 到 output*/
pa_start &= TTBL_L3_MAP_MASK;
page_addr = map_start;
while (page_addr < map_end) {
/* Setup level1 table */
ttbl = (u64 *) lpae_entry->ttbl_base;
index = (page_addr & TTBL_L1_INDEX_MASK) >> TTBL_L1_INDEX_SHIFT;
if (ttbl[index] & TTBL_VALID_MASK) {
/* Find level2 table */
ttbl =
(u64 *) (unsigned long)(ttbl[index] &
TTBL_OUTADDR_MASK);
} else {
/* Allocate new level2 table */
if (lpae_entry->ttbl_count == TTBL_INITIAL_TABLE_COUNT) {
while (1) ; /* No initial table available */
}
for (i = 0; i < TTBL_TABLE_ENTCNT; i++) {
lpae_entry->next_ttbl[i] = 0x0ULL;
}
lpae_entry->ttbl_tree[lpae_entry->ttbl_count] =
((virtual_addr_t) ttbl -
lpae_entry->ttbl_base) >> TTBL_TABLE_SIZE_SHIFT;
lpae_entry->ttbl_count++;
ttbl[index] |=
(((virtual_addr_t) lpae_entry->next_ttbl) &
TTBL_OUTADDR_MASK);
ttbl[index] |= (TTBL_TABLE_MASK | TTBL_VALID_MASK);
ttbl = lpae_entry->next_ttbl;
lpae_entry->next_ttbl += TTBL_TABLE_ENTCNT;
}
/* Setup level2 table */
index = (page_addr & TTBL_L2_INDEX_MASK) >> TTBL_L2_INDEX_SHIFT;
if (ttbl[index] & TTBL_VALID_MASK) {
/* Find level3 table */
ttbl =
(u64 *) (unsigned long)(ttbl[index] &
TTBL_OUTADDR_MASK);
} else {
/* Allocate new level3 table */
......
}
/* Setup level3 table */
index = (page_addr & TTBL_L3_INDEX_MASK) >> TTBL_L3_INDEX_SHIFT;
if (!(ttbl[index] & TTBL_VALID_MASK)) {
/* Update level3 table */
.......
}
/* Point to next page */
page_addr += TTBL_L3_BLOCK_SIZE;
}
}
TCR_EL2, Translation Control Register (EL2)
.. image:: /embedded/vtcr.png
.. code-block:: c
/* Setup Hypervisor Translation Control Register */
ldr x0, __htcr_set
msr tcr_el2, x0
- __htcr_set: (TCR_T0SZ_VAL(39) | TCR_PS_40BITS | (0x3 << TCR_SH0_SHIFT) | (0x1 << TCR_ORGN0_SHIFT) | (0x1 << TCR_IRGN0_SHIFT))
- TCR_T0SZ_VAL(39) = (64-39) & 0x3f: The size offset of the memory region addressed by TTBR0_EL2. The region size is 2^64-T0SZ bytes. 將 region size 設為 2^39 bytes
- Initial lookup level為level 1
- TCR_PS_40BITS = 2 << 16: Physical Address Size 設為40 bits, 1 TB。
- TCR_SH0_SHIFT = 12; 0x3: Shareability attribute for memory associated: Inner Shareable
- TCR_ORGN0_SHIFT = 10; 0x1: Outer cacheability attribute for memory associated: Write-Back Write-Allocate Cacheable
- TCR_IRGN0_SHIFT = 8; 0x1: Inner cacheability attribute for memory associated: Write-Back Write-Allocate Cacheable
.. code-block:: c
/* Setup Hypervisor Translation Base Register */
ldr x0, __httbr_set /* def_ttbl 的位置 */
msr ttbr0_el2, x0
GIC 配置
-------------------------------------------------------------
先看foundation model的手冊
- DGIC_DIST_BASE=0x2c001000
- DGIC_CPU_BASE=0x2c002000
.. image:: /embedded/gic_dist_map.png
(from GICv2 Architecture Specification P.74)
.. code-block:: c
/* GIC Distributor Interface Init */
mrs x4, mpidr_el1
ldr x5, __mpidr_mask
and x4, x4, x5 /* CPU affinity */
__gic_dist_init:
ldr x0, __gic_dist_base /* Dist GIC base */
mov x1, #0 /* non-0 cpus should at least */
cmp x4, xzr /* program IGROUP0 */
bne 1f
mov x1, #3 /* Enable group0 & group1 */
str w1, [x0, #0x00] /* Ctrl Register */
.. image:: /embedded/gic1.png
(from GICv2 Architecture Specification)
.. code-block:: c
ldr w1, [x0, #0x04] /* Type Register */
1: and x1, x1, #0x1f /* No. of IGROUPn registers */
add x2, x0, #0x080 /* IGROUP0 Register */
movn x3, #0 /* All interrupts to group-1 */
2: str w3, [x2], #4
subs x1, x1, #1
bge 2b
.. image:: /embedded/gic_cpu_interface.png
(from GICv2 Architecture Specification P.76)
- GICC_CTLR
.. image:: /embedded/gic_cpu_reg.png
- GICC_PMR
.. image:: /embedded/gicc_pmr.png
.. code-block:: c
__gic_cpu_init:
/* GIC Secured CPU Interface Init */
ldr x0, __gic_cpu_base /* GIC CPU base */
mov x1, #0x80
str w1, [x0, #0x4] /* GIC CPU Priority Mask */
mov x1, #0x3 /* Enable group0 & group1 */
str w1, [x0] /* GIC CPU Control */
在不同Arm架構下的虛擬化機制
=====================================
without virtualization extension
-------------------------------------------------------------
arm32 指令虛擬化
....................................
- `elf2cpatch.py<https://github.com/xvisor/xvisor/blob/6807a137dbcbef5b182c78b95986079a610af81d/arch/arm/cpu/arm32/elf2cpatch.py>`_
- 將敏感的非特權指令編碼成SVC 指令，並給每一個被編碼的指令一個獨特的inst_id
- 以mrs為例
.. code-block:: python
# MSR (immediate)
# Syntax:
# msr<c> <spec_reg>, #<const>
# Fields:
# cond = bits[31:28]
# R = bits[22:22]
# mask = bits[19:16]
# imm12 = bits[11:0]
# Hypercall Fields:
# inst_cond[31:28] = cond
# inst_op[27:24] = 0xf
# inst_id[23:20] = 0
# inst_subid[19:17] = 2
# inst_fields[16:13] = mask
# inst_fields[12:1] = imm12
# inst_fields[0:0] = R
def convert_msr_i_inst(hxstr):
hx = int(hxstr, 16)
inst_id = 0
inst_subid = 2
cond = (hx >> 28) & 0xF
R = (hx >> 22) & 0x1
mask = (hx >> 16) & 0xF
imm12 = (hx >> 0) & 0xFFF
rethx = 0x0F000000
rethx = rethx | (cond << 28)
rethx = rethx | (inst_id << 20)
rethx = rethx | (inst_subid << 17)
rethx = rethx | (mask << 13)
rethx = rethx | (imm12 << 1)
rethx = rethx | (R << 0)
return rethx
- instruction emulation
`cpu_entry.S<https://github.com/xvisor/xvisor/blob/6807a137dbcbef5b182c78b95986079a610af81d/arch/arm/cpu/arm32/cpu_entry.S>`_中註冊Exception vector
.. code-block:: c
_start_vect:
ldr pc, __reset
ldr pc, __undefined_instruction
ldr pc, __software_interrupt /* __software_interrupt: .word _soft_irq */
......
`cpu_interrupts.c<https://github.com/xvisor/xvisor/blob/6807a137dbcbef5b182c78b95986079a610af81d/arch/arm/cpu/arm32/cpu_interrupts.c>`_
.. code-block:: c
void do_soft_irq(arch_regs_t *regs)
{
........
/* If vcpu priviledge is user then generate exception
* and return without emulating instruction
*/
if ((arm_priv(vcpu)->cpsr & CPSR_MODE_MASK) == CPSR_MODE_USER) {
vmm_vcpu_irq_assert(vcpu, CPU_SOFT_IRQ, 0x0);
} else {
if (regs->cpsr & CPSR_THUMB_ENABLED) {
rc = cpu_vcpu_hypercall_thumb(vcpu, regs,
*((u32 *)regs->pc));
} else {
rc = cpu_vcpu_hypercall_arm(vcpu, regs,
*((u32 *)regs->pc));
}
}
........
}
以arm emulation為例， in `cpu_vcpu_hypercall_arm.c<https://github.com/xvisor/xvisor/blob/6807a137dbcbef5b182c78b95986079a610af81d/arch/arm/cpu/arm32/cpu_vcpu_hypercall_arm.c>`_:
.. code-block:: c
int cpu_vcpu_hypercall_arm(struct vmm_vcpu *vcpu,
arch_regs_t *regs, u32 inst)
{
u32 id = ARM_INST_DECODE(inst,
ARM_INST_HYPERCALL_ID_MASK,
ARM_INST_HYPERCALL_ID_SHIFT);
return hcall_funcs[id](id, inst, regs, vcpu);
}
將之前編碼後的指令解碼
.. code-block:: c
static int (* const hcall_funcs[])(u32 id, u32 inst,
arch_regs_t *regs, struct vmm_vcpu *vcpu) = {
arm_hypercall_cps_and_co, /* ARM_HYPERCALL_CPS_ID */
arm_hypercall_ldm_ue, /* ARM_HYPERCALL_LDM_UE_ID0 */
arm_hypercall_ldm_ue, /* ARM_HYPERCALL_LDM_UE_ID1 */
arm_hypercall_ldm_ue, /* ARM_HYPERCALL_LDM_UE_ID2 */
.......
};
以id = 0的為例(CPS, MRS, MSR, RFE, SRS, WFI, WFE, SEV, SMC等) (Q: 如何分類的？)
.. code-block:: c
static int arm_hypercall_cps_and_co(u32 id, u32 inst,
arch_regs_t *regs, struct vmm_vcpu *vcpu)
{
u32 subid = ARM_INST_DECODE(inst,
ARM_INST_HYPERCALL_SUBID_MASK,
ARM_INST_HYPERCALL_SUBID_SHIFT);
return cps_and_co_funcs[subid](inst, regs, vcpu);
}
再做一次decode(即可知其指令為何)，呼叫模擬該指令的function
.. code-block:: c
static int (* const cps_and_co_funcs[])(u32 inst,
arch_regs_t *regs, struct vmm_vcpu *vcpu) = {
arm_hypercall_cps, /* ARM_HYPERCALL_CPS_SUBID */
arm_hypercall_mrs, /* ARM_HYPERCALL_MRS_SUBID */
arm_hypercall_msr_i, /* ARM_HYPERCALL_MSR_I_SUBID */
arm_hypercall_msr_r, /* ARM_HYPERCALL_MSR_R_SUBID */
arm_hypercall_rfe, /* ARM_HYPERCALL_RFE_SUBID */
arm_hypercall_srs, /* ARM_HYPERCALL_SRS_SUBID */
arm_hypercall_wfx, /* ARM_HYPERCALL_WFI_SUBID */
arm_hypercall_smc /* ARM_HYPERCALL_SMC_SUBID */
};
以mrs為例：
.. code-block:: c
/* Emulate 'mrs' hypercall */
static int arm_hypercall_mrs(u32 inst,
arch_regs_t *regs, struct vmm_vcpu *vcpu)
{
register u32 Rd;
Rd = ARM_INST_BITS(inst,
ARM_HYPERCALL_MRS_RD_END,
ARM_HYPERCALL_MRS_RD_START);
if (Rd == 15) {
arm_unpredictable(regs, vcpu, inst, __func__);
return VMM_EFAIL;
}
if (ARM_INST_BIT(inst, ARM_HYPERCALL_MRS_R_START)) {
cpu_vcpu_reg_write(vcpu, regs, Rd,
cpu_vcpu_spsr_retrieve(vcpu));
} else {
cpu_vcpu_reg_write(vcpu, regs, Rd,
cpu_vcpu_cpsr_retrieve(vcpu, regs));
}
regs->pc += 4;
return VMM_OK;
}
armv5
....................
- ARM9: With MMU
-
armv7
.........................
- 有安全性擴展，但沒有虛擬化擴展
with virtualization extension
-------------------------------------------------------------
armv7ve
........................
- 有安全性擴展、LPAE及虛擬化擴展
- Cortex-A15 / Cortex-A7 (with big-LITTLE support)
armv8
.................
設計文件整理
=====================================
原文： https://github.com/xvisor/xvisor/blob/master/docs/DesignDoc
Modeling Virtual Machine
--------------------------------------------
* 何謂VM(virtual machine),通常分為兩種
- system virtual machine: support the execution of a complete OS
- process virtual machine: support a single process
* Xvisor為硬體系統的虛擬化軟體，可直接運行於主機機器，為Native/Type-1的Hypervisor/VMM system virtual machine通常稱為guest,而guest裏的CPU稱為VCPU(Virtual CPU),VCPU又可分為兩種
- 屬於Guest的稱為 Normal VCPU
- 不屬於任何Guest的稱為 Orphan VCPU (Orphan VCPU是為了不同的背景程序及運行中的管理daemon而建立的)
* 當今CPU至少有兩種privilege mode:
- User mode為最低特權，運行Normal VCPUs
- Supervisor mode為最高特權，運行Orphan VCPUs
* Xvisor當執行various background process和執行management daemons使用Orphan VCPUs
下圖為Xvisor的System Virtual Machine Model
.. image:: /embedded/xvisor_model.png
Xvisor 測試
=====================================
1. 註冊ARM帳號
2. 下載ARM `Foundation v8 model<http://www.arm.com/zh/products/tools/models/fast-models/foundation-model.php>`_
3. 解壓縮到workspace
4. 安裝toolchain
.. code-block:: prettyprint
sudo apt-get install gcc-aarch64-linux-gnu
sudo apt-get install genext2fs
4. 下載及編譯Xvisor
.. code-block:: prettyprint
cd workspace
git clone https://github.com/avpatel/xvisor-next.git && cd xvisor-next
export CROSS_COMPILE=aarch64-linux-gnu-
make ARCH=arm generic-v8-defconfig
make
make dtbs
5. 準備guest OS
.. code-block:: prettyprint
make -C tests/arm64/virt-v8/basic
mkdir -p ./build/disk/images/arm64/virt-v8
./build/tools/dtc/dtc -I dts -O dtb -o ./build/disk/images/arm64/virt-v8x2.dtb ./tests/arm64/virt-v8/virt-v8x2.dts
cp -f ./build/tests/arm64/virt-v8/basic/firmware.bin ./build/disk/images/arm64/virt-v8/firmware.bin
cp -f ./tests/arm64/virt-v8/basic/nor_flash.list ./build/disk/images/arm64/virt-v8/nor_flash.list
genext2fs -B 1024 -b 16384 -d ./build/disk ./build/disk.img
./tools/scripts/memimg.py -a 0x80010000 -o ./build/foundation_v8_boot.img ./build/vmm.bin@0x80010000 ./build disk.img@0x81000000
aarch64-linux-gnu-gcc -nostdlib -nostdinc -e _start -Wl,-Ttext=0x80000000 -DGENTIMER_FREQ=100000000 -DUART_BASE=0x1c090000 -DGIC_DIST_BASE=0x2c001000 -DGIC_CPU_BASE=0x2c002000 -DSPIN_LOOP_ADDR=0x84000000 -DIMAGE=./build/foundation_v8_boot.img -DDTB=./build/arch/arm/board/generic/dts/foundation-v8/one_guest_virt-v8.dtb ./docs/arm/foundation_v8_boot.S -o ./build/foundation_v8_boot.axf
6. 測試
.. code-block:: prettyprint
cd tests/arm64/virt-v8/basic
tclsh armv8_test.tcl ~/workspace/Foundation_Platformpkg/models/Linux64_GCC-4.1/Foundation_Platform
(需先安裝tclsh，後面路徑假設Foundation v8 model解壓縮在workspace)
reference
=====================================
* http://www.slideshare.net/jserv/xvisor
* https://github.com/xvisor/xvisor/blob/master/docs/DesignDoc
* https://samlin.hackpad.com/Xvisor--chtGSqPYWG8
* http://www.slideshare.net/badaindonesia/linux-on-arm-64bit-architecture?related=1
* `ARMv8-A_Architecture_Reference_Manual_(Issue_A.a) (需登入)<http://infocenter.arm.com/help/index.jsp?topic=/com.arm.doc.ddi0487a.e/index.html>`_
* `A Virtualization Infrastructure that Supports Pervasive Computing.<http://165.193.233.120/files/pdf/partners/academic/vmware-pervasive-computing_LRudolph-en.pdf>`_
* `A Choices Hypervisor on the ARM architecture<http://www.russellgreenspan.com/software/A%20Choices%20Hypervisor%20on%20the%20ARM%20Architecture.pdf>`_
* `Extensions to the ARMv7-A architecture<http://www.hotchips.org/wp-content/uploads/hc_archives/hc22/HC22.23.220-1-Brash-ARMv7A.pdf>`_
* `前瞻 資訊科技 - 虛擬化 (1) - Virtualization( V12N ). 薛智文 教授<http://www.slideserve.com/quynn-dickson/cwhsueh-csie-ntu-tw-csie-ntu-tw-cwhsueh>`_
* `前瞻 資訊科技 - 虛擬化 (2) - Virtualization( V12N ). 薛智文 教授<http://www.slideserve.com/candid/cwhsueh-csie-ntu-edu-tw-http-www-csie-ntu-edu-tw-cwhsueh-100-fall-nov-4-fri-678-dth-104>`_
* ` An Overview of Microkernel, Hypervisor and Microvisor Virtualization Approaches for Embedded Systems, Asif Iqbal, Nayeema Sadeque and Rafika Ida Mutia, Lund University, Sweden<http://www.eit.lth.se/fileadmin/eit/project/142/virtApproaches.pdf>`_
* `Hardware accelerated Virtualization in the ARM Cortex™ Processors<http://www-archive.xenproject.org/files/xensummit_seoul11/nov2/2_XSAsia11_JGoodacre_HW_accelerated_virtualization_in_the_ARM_Cortex_processors.pdf>`_
* `Popek and Goldberg virtualization requirements<http://en.wikipedia.org/wiki/Popek_and_Goldberg_virtualization_requirements>`_
* `Syllabus for CS5410 Virtualization Techniques (國立清華大學)<http://www.cs.nthu.edu.tw/~ychung/syllabus/Virtualization.htm>`_ (內有課程簡報)
* `GICv2 Architecture Specification<http://www.cl.cam.ac.uk/research/srg/han/ACS-P35/zynq/arm_gic_architecture_specification.pdf>`_