diff --git a/.claude/skills/arch-platform-porting/SKILL.md b/.claude/skills/arch-platform-porting/SKILL.md index 33021e02b5..78594a6e78 100644 --- a/.claude/skills/arch-platform-porting/SKILL.md +++ b/.claude/skills/arch-platform-porting/SKILL.md @@ -33,6 +33,7 @@ Current Axvisor LoongArch QEMU bring-up uses the dynamic UEFI platform path. The - **Runtime CPU limits**: treat generated `CPU_CAPACITY`/`SMP` as a build-time capacity for const generics, per-CPU arrays, and linker/percpu layout only. Actual online/usable CPU count must flow through `ax_hal::cpu_num()`, which caps the platform-discovered count by capacity. - **IRQ namespace rules**: keep CPU trap vectors, platform `IrqId { domain, hwirq }`, firmware sources (`IrqSource::AcpiGsi`, `IrqSource::AcpiGsiRoute`, explicit `IrqSource::ControllerLine`, and driver binding metadata such as `BindingIrqSource::FdtInterrupt`), controller-local hardware lines (`HwIrq`), and guest GSI/vector values in separate namespaces. New runtime IRQ registrations must use `IrqId`, not `usize`; legacy `IrqNumber(raw)` is only for static or still-unmigrated platform boundaries and must live in OS/HAL-facing layers such as `ax-plat`, `ax-hal`, or `axklib`, not `irq-framework` or `somehal`. `irq-framework` owns generic registry, affinity, execution, and boxed callback dispatch semantics; platform rebase work must preserve `BoxedIrqHandler`, `IrqExecution`, and `IrqRequest::new_boxed` while adapting the surrounding platform code to `IrqId`. `LEGACY_IRQ_DOMAIN` and `CPU_LOCAL_IRQ_DOMAIN` remain fixed compatibility domains, while dynamic `somehal` external controller domains such as GIC, PLIC, IOAPIC, EIOINTC, and PCH-PIC are allocated at controller probe time and must be reached through `alloc_irq_domain`, `domain_by_kind`, `domain_by_owner`, or `domain_is_kind`, not by constructing fixed numeric controller domains in dynamic-platform code. Do not derive a host IRQ with arithmetic such as `0x20 + gsi`, `PCI_INTX_VECTOR_BASE + gsi`, or by subtracting a trap-vector base in Axvisor/device code. Resolve firmware/device descriptions with `ax_hal::irq::resolve_irq_source(...)` / platform resolver and register the returned `IrqId`. When ACPI supplies trigger/polarity/controller metadata, carry it as `IrqSource::AcpiGsiRoute` instead of flattening it to a bare `AcpiGsi`, because PCI INTx routes may use a low GSI with non-ISA level/low semantics. Likewise, FDT device bindings should carry the raw interrupt specifier plus its controller owner in `BindingIrqSource::FdtInterrupt` until the OS/platform layer can resolve that owner to a controller domain and configure it; do not expose parentless FDT cells from `irq-framework` or configure a controller in generic driver probe merely to obtain a legacy number. `rdif_intc` controllers must expose fallible `translate_fdt` / `translate_acpi` methods that return controller-local hardware line and trigger metadata; the registering platform allocates or looks up a domain owner entry for the concrete `rdrive::DeviceId`, passes that domain to `rdif_intc::Intc::new(domain, driver)`, and the wrapper combines that domain with the local `HwIrq` before `configure` / `configure_acpi` programs trigger, polarity, vector, or mask state. Platform `irq_set_enable` and `irq_set_affinity` paths must route by the incoming domain's registered owner/kind and return an error on missing controllers, lock failures, unsupported affinity, or backend/type mismatches instead of silently no-oping. Empty, malformed, out-of-range, or unsupported firmware specifiers must return `IrqError` instead of IRQ 0, a base vector, or a guessed legacy number. If an FDT PCI host bridge preconfigures a controller-level legacy INTx route, store that route as a native `BindingIrq` source (plus any temporary raw compatibility value) and let child endpoints reuse it before falling back to PCI `interrupt-map` parsing. - **Domain expectations**: x86 LAPIC timer and IOAPIC are distinct domains, so trap vector `0x20` is not `AcpiGsi(0)`. On aarch64, GIC INTID is the `HwIrq` within the GIC domain. On riscv64, PLIC source is the `HwIrq` within the PLIC domain. On loongarch64, EIOINTC and PCH-PIC must remain separate domains. A platform that cannot resolve an `IrqSource` must return `IrqError::Unsupported` instead of guessing a numeric IRQ. +- **x86 QEMU IRQ contract**: the dynamic x86 path targets modern QEMU `q35` with ACPI/MADT, Local APIC or x2APIC, IOAPIC, and PCI INTx routing. Do not add 8259/PIC fallback, i440fx-specific IRQ assumptions, non-ACPI IRQ probing, raw GSI enable bypasses, or vector arithmetic outside the IOAPIC controller. LAPIC/x2APIC owns timer, IPI, EOI, and spurious handling; `X86IoApicIntc` owns external GSI route state, vector conflict checks, trigger/polarity, mask, and affinity updates through `rdif_intc::Intc`. x2APIC paths must preserve full `u32` APIC IDs for CPU-local operations, while xAPIC and IOAPIC physical destinations must reject APIC IDs that cannot be encoded without truncation. - **Runtime console selection**: Dynamic platforms expose the firmware-selected hardware console through `somehal::console_device_id()` and `ax_hal::console::device_id()`. The value is `Result` derived from bootargs `console=`, ACPI SPCR, or FDT `stdout-path`; static platforms return `Err(NotSpecified)`. OS code such as Starry should match `Ok(id)` against probed serial devices, use `ttyS0` as the Linux-style hardware-console fallback only for `Err(NotSpecified)`, and leave `/dev/console` unbound (`ENODEV`) for non-hardware console selections, unmatched selected hardware devices, or when no serial console TTY exists. Do not reparse FDT or bootargs in the tty layer. - **Runtime console ownership**: once Starry or another OS runtime binds the firmware-selected UART to an interrupt-driven tty/serial driver, call `ax_hal::console::claim_runtime_output()` and stop the low-level boot/platform console path from writing the same UART registers directly. The hardware console must have one runtime register owner; otherwise kernel log output and tty output can interleave at the UART register level and corrupt test markers or user input/output. - **Dynamic firmware devices**: for `rdrive` ACPI probes, real non-empty ACPI ID lists enumerate namespace `Device` nodes and expose `_CRS` memory, I/O port, and IRQ resources through `AcpiInfo`; empty ID lists or synthetic root IDs are reserved for root-table style callbacks. @@ -52,6 +53,7 @@ Current Axvisor LoongArch QEMU bring-up uses the dynamic UEFI platform path. The - Allocate and align boot stack, per-CPU areas, secondary stacks, boot arguments, and page tables before enabling SMP. - Install trap vectors before enabling interrupts, timer interrupts, MMU faults, or secondary CPU execution. - On x86 QEMU, do not trust CPUID timing leaves unless the reported TSC frequency is plausible; some virtual CPU combinations expose invalid zero or tiny values. Prefer a trusted hypervisor timing leaf, then CPUID timing data, then PIT-based TSC calibration before falling back to processor base frequency. +- On x86 QEMU, initialize LAPIC/x2APIC once and keep APIC IDs as firmware IDs, not logical CPU indices. Use x2APIC MSRs when x2APIC is enabled, bound IPI delivery waits, reject xAPIC AP startup/IPI destinations above 255, and keep external IOAPIC INTx programming in the runtime `X86IoApicIntc` path instead of someboot or HAL bypass helpers. - On AArch64, keep the someboot `hv` feature scoped to the EL2 kernel path. For non-`hv` EL1 boot, choose the EL1 arch timer at runtime from the boot EL: use CNTP when EL2 is available and CNTV when EL2 is unavailable, and keep the FDT timer interrupt index consistent with the selected mode. - Build page tables for identity/firmware access, direct map, kernel high map, MMIO, and per-CPU data as the arch requires. - Flush TLB/cache and use architecture barriers around page table writes, boot argument writes, and secondary CPU release. diff --git a/components/irq-framework/src/registry.rs b/components/irq-framework/src/registry.rs index fc0f68c6d1..83c119bfbc 100644 --- a/components/irq-framework/src/registry.rs +++ b/components/irq-framework/src/registry.rs @@ -574,6 +574,9 @@ impl Registry { self.ops.set_enabled(irq, Some(cpu), enabled) } Some(cpu) => { + if self.ops.in_irq_context() { + return Err(IrqError::InIrqContext); + } let mut request = RemoteEnable { registry: self as *const Self as *mut (), irq, diff --git a/components/irq-framework/src/types.rs b/components/irq-framework/src/types.rs index 65146c7dba..1561e0333c 100644 --- a/components/irq-framework/src/types.rs +++ b/components/irq-framework/src/types.rs @@ -273,6 +273,8 @@ pub enum IrqError { InvalidCpu, /// The target CPU is offline. CpuOffline, + /// A synchronous IRQ operation timed out. + Timeout, /// IRQ line/action sharing rules reject the operation. Busy, /// Allocation failed. diff --git a/components/irq-framework/tests/std_sim.rs b/components/irq-framework/tests/std_sim.rs index ea5debf8bb..9203034554 100644 --- a/components/irq-framework/tests/std_sim.rs +++ b/components/irq-framework/tests/std_sim.rs @@ -1119,6 +1119,40 @@ fn remote_per_cpu_enable_uses_run_on_cpu_sync() { })); } +#[test] +fn remote_per_cpu_enable_from_irq_context_is_rejected_without_ipi() { + let ops = MockOps::with_cpus(4); + ops.set_current_cpu(0); + ops.set_line_enabled(13, Some(2), false); + let registry = Registry::new(ops.clone()); + let counter = AtomicUsize::new(0); + let data = NonNull::from(&counter).cast(); + + let handle = registry + .request( + irq(13), + IrqRequest::new(count_handler, data) + .scope(IrqScope::PerCpu { + cpus: CpuMask::from_cpu(CpuId(2)), + }) + .auto_enable(AutoEnable::No), + ) + .unwrap(); + ops.inner.remote_calls.store(0, Ordering::SeqCst); + ops.clear_calls(); + + ops.set_in_irq(true); + assert_eq!(registry.enable(handle), Err(IrqError::InIrqContext)); + ops.set_in_irq(false); + + assert_eq!(ops.inner.remote_calls.load(Ordering::SeqCst), 0); + assert!(!ops.calls().contains(&OpCall::SetEnabled { + irq: 13, + cpu: Some(2), + enabled: true, + })); +} + #[test] fn failed_per_cpu_enable_rolls_back_action_state() { let ops = MockOps::with_cpus(4); diff --git a/components/someboot/src/arch/x86_64/power.rs b/components/someboot/src/arch/x86_64/power.rs index f3911da7c0..e787353516 100644 --- a/components/someboot/src/arch/x86_64/power.rs +++ b/components/someboot/src/arch/x86_64/power.rs @@ -4,7 +4,9 @@ use core::{ sync::atomic::{AtomicBool, AtomicUsize, Ordering}, }; -use x86::msr::{IA32_APIC_BASE, rdmsr}; +use x86::msr::{ + IA32_APIC_BASE, IA32_X2APIC_APICID, IA32_X2APIC_ESR, IA32_X2APIC_ICR, rdmsr, wrmsr, +}; use crate::{mem::phys_to_virt, power::CpuOnError, smp::PerCpuMeta}; @@ -16,6 +18,9 @@ const AP_START_TIMEOUT_US: u64 = 500_000; const LAPIC_REG_ESR: u32 = 0x280; const LAPIC_REG_ICR_LOW: u32 = 0x300; const LAPIC_REG_ICR_HIGH: u32 = 0x310; +const ICR_DELIVERY_PENDING: u32 = 1 << 12; +const IPI_DELIVERY_WAIT_SPINS: usize = 1_000_000; +const IA32_APIC_BASE_X2APIC_ENABLE: u64 = 1 << 10; // INIT IPI (level-triggered): assert then deassert. const ICR_INIT_ASSERT: u32 = 0x0000_c500; @@ -26,6 +31,12 @@ const ICR_STARTUP_BASE: u32 = 0x0000_4600; static START_LOCK: AtomicBool = AtomicBool::new(false); static AP_BOOTED_ID: AtomicUsize = AtomicUsize::new(usize::MAX); +#[derive(Clone, Copy, Debug, Eq, PartialEq)] +enum ApicMode { + XApic, + X2Apic, +} + global_asm!( r#" .section .text.ap_trampoline, "ax" @@ -136,17 +147,20 @@ pub(crate) fn notify_ap_started(apic_id: usize) { } fn current_apic_id() -> usize { - x86::cpuid::CpuId::new() - .get_feature_info() - .map(|info| info.initial_local_apic_id() as usize) - .unwrap_or(0) + match current_apic_mode() { + ApicMode::X2Apic => unsafe { rdmsr(IA32_X2APIC_APICID) as usize }, + ApicMode::XApic => x86::cpuid::CpuId::new() + .get_feature_info() + .map(|info| info.initial_local_apic_id() as usize) + .unwrap_or(0), + } } pub(crate) fn cpu_on(apic_id: usize, entry: usize, arg: usize) -> Result<(), CpuOnError> { if apic_id == current_apic_id() { return Err(CpuOnError::AlreadyOn); } - let apic_id = u8::try_from(apic_id).map_err(|_| CpuOnError::InvalidParameters)?; + let apic_id = u32::try_from(apic_id).map_err(|_| CpuOnError::InvalidParameters)?; let meta = unsafe { &*(phys_to_virt(arg) as *const PerCpuMeta) }; if meta.boot_table_paddr > u32::MAX as usize { return Err(CpuOnError::Other(anyhow::anyhow!( @@ -165,21 +179,13 @@ pub(crate) fn cpu_on(apic_id: usize, entry: usize, arg: usize) -> Result<(), Cpu entry_virt as u64, ); - unsafe { - send_ipi(apic_id, ICR_INIT_ASSERT); - } + send_ipi(apic_id, ICR_INIT_ASSERT)?; delay_us(10_000); - unsafe { - send_ipi(apic_id, ICR_INIT_DEASSERT); - } + send_ipi(apic_id, ICR_INIT_DEASSERT)?; delay_us(200); - unsafe { - send_ipi(apic_id, ICR_STARTUP_BASE | AP_TRAMPOLINE_VECTOR as u32); - } + send_ipi(apic_id, ICR_STARTUP_BASE | AP_TRAMPOLINE_VECTOR as u32)?; delay_us(200); - unsafe { - send_ipi(apic_id, ICR_STARTUP_BASE | AP_TRAMPOLINE_VECTOR as u32); - } + send_ipi(apic_id, ICR_STARTUP_BASE | AP_TRAMPOLINE_VECTOR as u32)?; let start = super::trap::ticks_now(); let timeout_ticks = us_to_tsc_ticks(AP_START_TIMEOUT_US); @@ -294,16 +300,72 @@ unsafe fn lapic_read(offset: u32) -> u32 { unsafe { ptr.read_volatile() } } -unsafe fn send_ipi(apic_id: u8, icr_low: u32) { +fn current_apic_mode() -> ApicMode { + let base = unsafe { rdmsr(IA32_APIC_BASE) }; + if base & IA32_APIC_BASE_X2APIC_ENABLE != 0 { + ApicMode::X2Apic + } else { + ApicMode::XApic + } +} + +fn xapic_destination(apic_id: u32) -> Result { + let dest = u8::try_from(apic_id).map_err(|_| CpuOnError::InvalidParameters)?; + Ok(u32::from(dest) << 24) +} + +fn x2apic_icr(apic_id: u32, icr_low: u32) -> u64 { + (u64::from(apic_id) << 32) | u64::from(icr_low) +} + +fn send_ipi(apic_id: u32, icr_low: u32) -> Result<(), CpuOnError> { + match current_apic_mode() { + ApicMode::X2Apic => send_x2apic_ipi(x2apic_icr(apic_id, icr_low)), + ApicMode::XApic => send_xapic_ipi(xapic_destination(apic_id)?, icr_low), + } +} + +fn send_xapic_ipi(destination: u32, icr_low: u32) -> Result<(), CpuOnError> { unsafe { lapic_write(LAPIC_REG_ESR, 0); lapic_write(LAPIC_REG_ESR, 0); - lapic_write(LAPIC_REG_ICR_HIGH, (apic_id as u32) << 24); + lapic_write(LAPIC_REG_ICR_HIGH, destination); lapic_write(LAPIC_REG_ICR_LOW, icr_low); - while lapic_read(LAPIC_REG_ICR_LOW) & (1 << 12) != 0 { - spin_loop(); + } + wait_xapic_delivery() +} + +fn send_x2apic_ipi(icr: u64) -> Result<(), CpuOnError> { + unsafe { + wrmsr(IA32_X2APIC_ESR, 0); + wrmsr(IA32_X2APIC_ESR, 0); + wrmsr(IA32_X2APIC_ICR, icr); + } + wait_x2apic_delivery() +} + +fn wait_xapic_delivery() -> Result<(), CpuOnError> { + for _ in 0..IPI_DELIVERY_WAIT_SPINS { + if unsafe { lapic_read(LAPIC_REG_ICR_LOW) } & ICR_DELIVERY_PENDING == 0 { + return Ok(()); + } + spin_loop(); + } + Err(CpuOnError::Other(anyhow::anyhow!( + "timeout waiting xAPIC IPI delivery" + ))) +} + +fn wait_x2apic_delivery() -> Result<(), CpuOnError> { + for _ in 0..IPI_DELIVERY_WAIT_SPINS { + if unsafe { rdmsr(IA32_X2APIC_ICR) } & u64::from(ICR_DELIVERY_PENDING) == 0 { + return Ok(()); } + spin_loop(); } + Err(CpuOnError::Other(anyhow::anyhow!( + "timeout waiting x2APIC IPI delivery" + ))) } struct StartupGuard; @@ -325,3 +387,25 @@ impl Drop for StartupGuard { START_LOCK.store(false, Ordering::Release); } } + +#[cfg(test)] +mod tests { + use super::*; + + #[test] + fn xapic_destination_rejects_high_apic_ids_without_truncation() { + assert!(matches!(xapic_destination(0xff), Ok(0xff00_0000))); + assert!(matches!( + xapic_destination(0x100), + Err(CpuOnError::InvalidParameters) + )); + } + + #[test] + fn x2apic_icr_encodes_full_destination_id() { + let icr = x2apic_icr(0x1234_5678, ICR_STARTUP_BASE | AP_TRAMPOLINE_VECTOR as u32); + + assert_eq!(icr >> 32, 0x1234_5678); + assert_eq!(icr as u32, ICR_STARTUP_BASE | AP_TRAMPOLINE_VECTOR as u32); + } +} diff --git a/components/someboot/src/arch/x86_64/trap.rs b/components/someboot/src/arch/x86_64/trap.rs index f52c6d28ff..9aa559a58e 100644 --- a/components/someboot/src/arch/x86_64/trap.rs +++ b/components/someboot/src/arch/x86_64/trap.rs @@ -31,7 +31,14 @@ const LAPIC_LVT_TIMER_TSC_DEADLINE: u32 = 1 << 18; const LAPIC_SVR_ENABLE: u32 = 1 << 8; const LAPIC_BASE_MASK: u64 = 0xffff_f000; const IA32_APIC_BASE_ENABLE: u64 = 1 << 11; +const IA32_APIC_BASE_X2APIC_ENABLE: u64 = 1 << 10; const LAPIC_TIMER_DIVIDE_BY_16: u32 = 0b0011; +const IA32_X2APIC_EOI: u32 = 0x80b; +const IA32_X2APIC_SIVR: u32 = 0x80f; +const IA32_X2APIC_LVT_TIMER: u32 = 0x832; +const IA32_X2APIC_INIT_COUNT: u32 = 0x838; +const IA32_X2APIC_CUR_COUNT: u32 = 0x839; +const IA32_X2APIC_DIV_CONF: u32 = 0x83e; const PIT_CHANNEL2_PORT: u16 = 0x42; const PIT_COMMAND_PORT: u16 = 0x43; const PIT_CONTROL_PORT: u16 = 0x61; @@ -52,6 +59,12 @@ static LAPIC_READY: AtomicBool = AtomicBool::new(false); static TSC_INFO_STATE: AtomicU8 = AtomicU8::new(0); static IDT_STATE: AtomicU8 = AtomicU8::new(0); +#[derive(Clone, Copy, Debug, Eq, PartialEq)] +enum ApicMode { + XApic, + X2Apic, +} + #[derive(Clone, Copy, Debug)] #[repr(C)] pub struct InterruptStackFrame { @@ -338,6 +351,9 @@ fn load_idt() { fn init_lapic() { let mut base = unsafe { rdmsr(msr::IA32_APIC_BASE) }; base |= IA32_APIC_BASE_ENABLE; + if cpu_has_x2apic() { + base |= IA32_APIC_BASE_X2APIC_ENABLE; + } unsafe { wrmsr(msr::IA32_APIC_BASE, base); } @@ -423,14 +439,53 @@ fn ticks_to_apic_counts(ticks: u64) -> u32 { } fn read_lapic_reg(offset: u32) -> u32 { - let ptr = lapic_ptr(offset); - unsafe { ptr.read_volatile() } + match current_apic_mode() { + ApicMode::X2Apic => unsafe { rdmsr(x2apic_msr(offset)) as u32 }, + ApicMode::XApic => { + let ptr = lapic_ptr(offset); + unsafe { ptr.read_volatile() } + } + } } fn write_lapic_reg(offset: u32, value: u32) { - let ptr = lapic_ptr(offset); - unsafe { - ptr.write_volatile(value); + match current_apic_mode() { + ApicMode::X2Apic => unsafe { + wrmsr(x2apic_msr(offset), u64::from(value)); + }, + ApicMode::XApic => { + let ptr = lapic_ptr(offset); + unsafe { + ptr.write_volatile(value); + } + } + } +} + +fn cpu_has_x2apic() -> bool { + CpuId::new() + .get_feature_info() + .is_some_and(|info| info.has_x2apic()) +} + +fn current_apic_mode() -> ApicMode { + let base = unsafe { rdmsr(msr::IA32_APIC_BASE) }; + if base & IA32_APIC_BASE_X2APIC_ENABLE != 0 { + ApicMode::X2Apic + } else { + ApicMode::XApic + } +} + +fn x2apic_msr(offset: u32) -> u32 { + match offset { + LAPIC_REG_EOI => IA32_X2APIC_EOI, + LAPIC_REG_SVR => IA32_X2APIC_SIVR, + LAPIC_REG_LVT_TIMER => IA32_X2APIC_LVT_TIMER, + LAPIC_REG_TIMER_INIT_COUNT => IA32_X2APIC_INIT_COUNT, + LAPIC_REG_TIMER_CUR_COUNT => IA32_X2APIC_CUR_COUNT, + LAPIC_REG_TIMER_DIV => IA32_X2APIC_DIV_CONF, + _ => panic!("unsupported x2APIC register offset {offset:#x}"), } } diff --git a/os/arceos/modules/axhal/src/irq.rs b/os/arceos/modules/axhal/src/irq.rs index 5a305373ba..4107594eb2 100644 --- a/os/arceos/modules/axhal/src/irq.rs +++ b/os/arceos/modules/axhal/src/irq.rs @@ -35,11 +35,6 @@ pub fn handle_irq(vector: usize) -> bool { handled } -#[cfg(all(plat_dyn, target_arch = "x86_64"))] -pub fn set_ioapic_gsi_enabled_from_irq(gsi: u32, enabled: bool) -> Result<(), IrqError> { - axplat_dyn::set_ioapic_gsi_enabled_from_irq(gsi, enabled) -} - /// Installs the default ArceOS IRQ dispatcher into `ax-cpu`'s runtime hook. /// /// This is intended for runtimes that dispatch traps through diff --git a/os/arceos/modules/axipi/src/lib.rs b/os/arceos/modules/axipi/src/lib.rs index cabc271023..7324a723a7 100644 --- a/os/arceos/modules/axipi/src/lib.rs +++ b/os/arceos/modules/axipi/src/lib.rs @@ -6,6 +6,7 @@ extern crate log; extern crate alloc; +use alloc::sync::Arc; use core::sync::atomic::{AtomicBool, AtomicU8, Ordering}; use ax_hal::{irq::IpiTarget, percpu::this_cpu_id}; @@ -33,6 +34,7 @@ static IPI_CPU_STATE: [AtomicU8; build_info::CPU_CAPACITY] = [const { AtomicU8::new(IPI_CPU_NOT_READY) }; build_info::CPU_CAPACITY]; static IPI_READY_CPUS: core::sync::atomic::AtomicUsize = core::sync::atomic::AtomicUsize::new(0); +const SYNC_IPI_SPIN_LIMIT: usize = 10_000_000; /// Initialize the per-CPU IPI event queue. pub fn init() { @@ -129,24 +131,30 @@ pub unsafe fn run_on_cpu_sync_raw( struct SyncCall { done: AtomicBool, f: unsafe fn(*mut ()), - arg: *mut (), + arg: usize, } - let call = SyncCall { + let call = Arc::new(SyncCall { done: AtomicBool::new(false), f, - arg, - }; - let call_ptr = &call as *const SyncCall as usize; + arg: arg as usize, + }); + let remote_call = Arc::clone(&call); run_on_cpu(dest_cpu, move || { - let call = unsafe { &*(call_ptr as *const SyncCall) }; - unsafe { (call.f)(call.arg) }; - call.done.store(true, Ordering::Release); + unsafe { (remote_call.f)(remote_call.arg as *mut ()) }; + remote_call.done.store(true, Ordering::Release); }); - while !call.done.load(Ordering::Acquire) { + wait_for_sync_call(&call.done) +} + +fn wait_for_sync_call(done: &AtomicBool) -> Result<(), ax_hal::irq::IrqError> { + for _ in 0..SYNC_IPI_SPIN_LIMIT { + if done.load(Ordering::Acquire) { + return Ok(()); + } core::hint::spin_loop(); } - Ok(()) + Err(ax_hal::irq::IrqError::Timeout) } /// Executes a callback on all other CPUs via IPI. @@ -186,3 +194,25 @@ pub fn ipi_handler() { callback.call(); } } + +#[cfg(test)] +mod tests { + use super::*; + + #[test] + fn sync_call_wait_returns_timeout_when_remote_cpu_does_not_complete() { + let done = AtomicBool::new(false); + + assert_eq!( + wait_for_sync_call(&done), + Err(ax_hal::irq::IrqError::Timeout) + ); + } + + #[test] + fn sync_call_wait_returns_ok_after_completion() { + let done = AtomicBool::new(true); + + assert_eq!(wait_for_sync_call(&done), Ok(())); + } +} diff --git a/os/arceos/modules/axruntime/src/fs/block.rs b/os/arceos/modules/axruntime/src/fs/block.rs index 9dddd31ff3..082c6ce7f5 100644 --- a/os/arceos/modules/axruntime/src/fs/block.rs +++ b/os/arceos/modules/axruntime/src/fs/block.rs @@ -129,6 +129,7 @@ fn map_block_irq_error(err: ax_hal::irq::IrqError) -> ax_errno::AxError { ax_hal::irq::IrqError::Busy | ax_hal::irq::IrqError::InIrqContext => { ax_errno::AxError::ResourceBusy } + ax_hal::irq::IrqError::Timeout => ax_errno::AxError::TimedOut, ax_hal::irq::IrqError::NoMemory => ax_errno::AxError::NoMemory, ax_hal::irq::IrqError::NotFound => ax_errno::AxError::NotFound, ax_hal::irq::IrqError::Controller => ax_errno::AxError::Io, diff --git a/os/arceos/modules/axruntime/src/irq.rs b/os/arceos/modules/axruntime/src/irq.rs index e9b51b89bc..48755a05ad 100644 --- a/os/arceos/modules/axruntime/src/irq.rs +++ b/os/arceos/modules/axruntime/src/irq.rs @@ -232,7 +232,7 @@ fn map_net_irq_error(err: IrqError) -> ax_net::EthernetIrqRegistrationError { IrqError::Unsupported | IrqError::CpuOffline => { ax_net::EthernetIrqRegistrationError::Unsupported } - IrqError::NoMemory | IrqError::NotFound | IrqError::Controller => { + IrqError::NoMemory | IrqError::NotFound | IrqError::Timeout | IrqError::Controller => { ax_net::EthernetIrqRegistrationError::Other } } diff --git a/os/arceos/modules/axruntime/src/klib.rs b/os/arceos/modules/axruntime/src/klib.rs index a3383c80e0..eaaf9b68f8 100644 --- a/os/arceos/modules/axruntime/src/klib.rs +++ b/os/arceos/modules/axruntime/src/klib.rs @@ -37,6 +37,7 @@ fn map_irq_error(err: IrqError) -> AxError { match err { IrqError::InvalidIrq | IrqError::InvalidCpu => AxError::InvalidInput, IrqError::CpuOffline | IrqError::Unsupported => AxError::Unsupported, + IrqError::Timeout => AxError::TimedOut, IrqError::Busy | IrqError::InIrqContext => AxError::ResourceBusy, IrqError::NoMemory => AxError::NoMemory, IrqError::NotFound => AxError::NotFound, diff --git a/platforms/axplat-dyn/src/lib.rs b/platforms/axplat-dyn/src/lib.rs index 74dde791bc..98268fc875 100644 --- a/platforms/axplat-dyn/src/lib.rs +++ b/platforms/axplat-dyn/src/lib.rs @@ -34,11 +34,3 @@ pub fn ipi_irq() -> ax_plat::irq::IrqId { } #[cfg(all(feature = "irq", target_arch = "riscv64", feature = "hv"))] pub use irq::register_virtual_irq_injector; - -#[cfg(all(feature = "irq", target_arch = "x86_64"))] -pub fn set_ioapic_gsi_enabled_from_irq( - gsi: u32, - enabled: bool, -) -> Result<(), ax_plat::irq::IrqError> { - somehal::arch::set_ioapic_gsi_enabled_from_irq(gsi, enabled) -} diff --git a/platforms/somehal/src/arch/x86_64/lapic.rs b/platforms/somehal/src/arch/x86_64/lapic.rs new file mode 100644 index 0000000000..e0cefbd3eb --- /dev/null +++ b/platforms/somehal/src/arch/x86_64/lapic.rs @@ -0,0 +1,125 @@ +use super::vector::{APIC_IPI_VECTOR, lapic_ipi_irq_id}; +use crate::irq::{IrqError, IrqId}; + +const LAPIC_REG_EOI: u32 = 0x0b0; +const LAPIC_REG_ICR_LOW: u32 = 0x300; +const LAPIC_REG_ICR_HIGH: u32 = 0x310; +const ICR_DELIVERY_PENDING: u32 = 1 << 12; +pub(super) const ICR_FIXED_BASE: u32 = 0x0000_4000; +pub(super) const ICR_DEST_SELF: u32 = 0x0004_0000; +pub(super) const ICR_DEST_ALL_EXCLUDING_SELF: u32 = 0x000c_0000; +const IPI_DELIVERY_WAIT_SPINS: usize = 1_000_000; + +const IA32_APIC_BASE_MSR: u32 = 0x1b; +const IA32_APIC_BASE_X2APIC_ENABLE: u64 = 1 << 10; +const IA32_X2APIC_EOI: u32 = 0x80b; +const IA32_X2APIC_ICR: u32 = 0x830; + +#[derive(Clone, Copy, Debug, Eq, PartialEq)] +enum ApicMode { + XApic, + X2Apic, +} + +pub(super) fn eoi() { + unsafe { + match current_apic_mode() { + ApicMode::X2Apic => x86::msr::wrmsr(IA32_X2APIC_EOI, 0), + ApicMode::XApic => lapic_write(LAPIC_REG_EOI, 0), + } + } +} + +pub(super) fn ipi_vector(irq: IrqId) -> Result { + if irq == lapic_ipi_irq_id() { + Ok(APIC_IPI_VECTOR as u8) + } else { + Err(IrqError::InvalidIrq) + } +} + +fn current_apic_mode() -> ApicMode { + let base = unsafe { x86::msr::rdmsr(IA32_APIC_BASE_MSR) }; + if base & IA32_APIC_BASE_X2APIC_ENABLE != 0 { + ApicMode::X2Apic + } else { + ApicMode::XApic + } +} + +pub(super) fn xapic_destination(apic_id: u32) -> Result { + let dest = u8::try_from(apic_id).map_err(|_| IrqError::InvalidCpu)?; + Ok(u32::from(dest) << 24) +} + +pub(super) fn x2apic_icr(apic_id: u32, icr_low: u32) -> u64 { + (u64::from(apic_id) << 32) | u64::from(icr_low) +} + +pub(super) fn send_ipi_to_apic_id(apic_id: u32, icr_low: u32) -> Result<(), IrqError> { + match current_apic_mode() { + ApicMode::X2Apic => send_x2apic_ipi(x2apic_icr(apic_id, icr_low)), + ApicMode::XApic => send_xapic_ipi(xapic_destination(apic_id)?, icr_low), + } +} + +pub(super) fn send_ipi(destination: u32, icr_low: u32) -> Result<(), IrqError> { + match current_apic_mode() { + ApicMode::X2Apic => send_x2apic_ipi(u64::from(icr_low)), + ApicMode::XApic => send_xapic_ipi(destination, icr_low), + } +} + +fn send_xapic_ipi(destination: u32, icr_low: u32) -> Result<(), IrqError> { + unsafe { + lapic_write(LAPIC_REG_ICR_HIGH, destination); + lapic_write(LAPIC_REG_ICR_LOW, icr_low); + } + wait_xapic_delivery() +} + +fn send_x2apic_ipi(icr: u64) -> Result<(), IrqError> { + unsafe { + x86::msr::wrmsr(IA32_X2APIC_ICR, icr); + } + wait_x2apic_delivery() +} + +fn wait_xapic_delivery() -> Result<(), IrqError> { + for _ in 0..IPI_DELIVERY_WAIT_SPINS { + if unsafe { lapic_read(LAPIC_REG_ICR_LOW) } & ICR_DELIVERY_PENDING == 0 { + return Ok(()); + } + core::hint::spin_loop(); + } + Err(IrqError::Timeout) +} + +fn wait_x2apic_delivery() -> Result<(), IrqError> { + for _ in 0..IPI_DELIVERY_WAIT_SPINS { + if unsafe { x86::msr::rdmsr(IA32_X2APIC_ICR) } & u64::from(ICR_DELIVERY_PENDING) == 0 { + return Ok(()); + } + core::hint::spin_loop(); + } + Err(IrqError::Timeout) +} + +unsafe fn lapic_read(offset: u32) -> u32 { + let ptr = lapic_ptr(offset) as *const u32; + unsafe { ptr.read_volatile() } +} + +unsafe fn lapic_write(offset: u32, value: u32) { + let ptr = lapic_ptr(offset); + unsafe { + ptr.write_volatile(value); + } +} + +fn lapic_ptr(offset: u32) -> *mut u32 { + const IA32_APIC_BASE: u32 = 0x1b; + const LAPIC_BASE_MASK: u64 = 0xffff_f000; + let base = unsafe { x86::msr::rdmsr(IA32_APIC_BASE) & LAPIC_BASE_MASK } as usize; + unsafe { someboot::mem::phys_to_virt(base).add(offset as usize) }.cast() +} diff --git a/platforms/somehal/src/arch/x86_64/mod.rs b/platforms/somehal/src/arch/x86_64/mod.rs index e76d598928..3dbe1ead16 100644 --- a/platforms/somehal/src/arch/x86_64/mod.rs +++ b/platforms/somehal/src/arch/x86_64/mod.rs @@ -1,5 +1,4 @@ use alloc::vec::Vec; -use core::sync::atomic::{AtomicU64, Ordering}; use rdif_intc::{AcpiGsiRoute, AcpiIrqPolarity, AcpiIrqTrigger}; use rdrive::{ @@ -16,101 +15,19 @@ use crate::{ irq::{CPU_LOCAL_IRQ_DOMAIN, HwIrq, IrqError, IrqId, IrqSource, X86_LAPIC_DOMAIN}, }; -pub struct Plat; - -const APIC_TIMER_VECTOR: usize = 0x20; -const APIC_IPI_VECTOR: usize = 0xf3; -const LAPIC_REG_EOI: u32 = 0x0b0; -const LAPIC_REG_ICR_LOW: u32 = 0x300; -const LAPIC_REG_ICR_HIGH: u32 = 0x310; -const ICR_DELIVERY_PENDING: u32 = 1 << 12; -const ICR_FIXED_BASE: u32 = 0x0000_4000; -const ICR_DEST_SELF: u32 = 0x0004_0000; -const ICR_DEST_ALL_EXCLUDING_SELF: u32 = 0x000c_0000; - -const UNMAPPED_VECTOR_ROUTE: u64 = u64::MAX; -const UNMAPPED_IOAPIC_GSI_ROUTE: u64 = 0; -static VECTOR_ROUTES: [AtomicU64; 256] = [const { AtomicU64::new(UNMAPPED_VECTOR_ROUTE) }; 256]; -static IOAPIC_GSI_ROUTES: [AtomicU64; 256] = - [const { AtomicU64::new(UNMAPPED_IOAPIC_GSI_ROUTE) }; 256]; - -fn pack_irq(irq: IrqId) -> u64 { - (u64::from(irq.domain.0) << 32) | u64::from(irq.hwirq.0) -} - -fn unpack_irq(raw: u64) -> IrqId { - IrqId::new( - crate::irq::IrqDomainId((raw >> 32) as u16), - HwIrq(raw as u32), - ) -} - -fn pack_ioapic_gsi_route(address: u64, input: u8) -> Result { - if address & 0xfff != 0 { - return Err(IrqError::InvalidIrq); - } - let page = address >> 12; - if page == 0 || page >= (1u64 << 56) { - return Err(IrqError::InvalidIrq); - } - Ok((page << 8) | u64::from(input)) -} - -fn unpack_ioapic_gsi_route(route: u64) -> Option<(u64, u8)> { - (route != UNMAPPED_IOAPIC_GSI_ROUTE).then_some(((route >> 8) << 12, route as u8)) -} - -fn lapic_timer_irq_id() -> IrqId { - IrqId::new(X86_LAPIC_DOMAIN, HwIrq(0)) -} - -fn lapic_ipi_irq_id() -> IrqId { - IrqId::new(CPU_LOCAL_IRQ_DOMAIN, HwIrq(APIC_IPI_VECTOR as u32)) -} - -fn cpu_local_vector_irq_id(raw: usize) -> IrqId { - IrqId::new(CPU_LOCAL_IRQ_DOMAIN, HwIrq(raw as u32)) -} +mod lapic; +mod vector; #[cfg(test)] -fn ioapic_gsi_irq_id(gsi: u32) -> IrqId { - IrqId::new(crate::irq::IrqDomainId(7), HwIrq(gsi)) -} - -fn trap_vector_irq_id(raw: usize) -> Option { - if raw == APIC_TIMER_VECTOR { - return Some(lapic_timer_irq_id()); - } - - if raw == APIC_IPI_VECTOR { - return Some(lapic_ipi_irq_id()); - } +use vector::{APIC_IPI_VECTOR, APIC_TIMER_VECTOR, ioapic_gsi_irq_id}; +use vector::{ + SPURIOUS_VECTOR, lapic_ipi_irq_id, lapic_timer_irq_id, local_vector_irq_id, + validate_external_vector, +}; - let mapped = VECTOR_ROUTES - .get(raw) - .map(|route| route.load(Ordering::Acquire)) - .filter(|raw| *raw != UNMAPPED_VECTOR_ROUTE) - .map(unpack_irq); - Some(mapped.unwrap_or_else(|| cpu_local_vector_irq_id(raw))) -} +const MASKED_IOAPIC_PLACEHOLDER_VECTOR: u8 = 0x21; -pub fn set_ioapic_gsi_enabled_from_irq(gsi: u32, enabled: bool) -> Result<(), IrqError> { - let route = IOAPIC_GSI_ROUTES - .get(gsi as usize) - .and_then(|slot| unpack_ioapic_gsi_route(slot.load(Ordering::Acquire))) - .ok_or(IrqError::NotFound)?; - - let (address, input) = route; - let mut ioapic = unsafe { IoApic::new(someboot::mem::phys_to_virt(address as usize) as u64) }; - unsafe { - if enabled { - ioapic.enable_irq(input); - } else { - ioapic.disable_irq(input); - } - } - Ok(()) -} +pub struct Plat; module_driver!( name: "ACPI IOAPIC", @@ -128,6 +45,7 @@ module_driver!( struct X86IoApicIntc { ioapics: Vec, routes: Vec, + vector_routes: Vec<(usize, IrqId)>, destinations: Vec<(usize, u8)>, } @@ -136,10 +54,34 @@ impl X86IoApicIntc { Self { ioapics: ioapics.iter().copied().map(X86IoApic::new).collect(), routes: Vec::new(), + vector_routes: Vec::new(), destinations: Vec::new(), } } + fn remember_vector_route(&mut self, vector: usize, irq: IrqId) -> Result { + let vector_u8 = validate_external_vector(vector)?; + if let Some((_, existing)) = self + .vector_routes + .iter_mut() + .find(|(known_vector, _)| *known_vector == vector) + { + if *existing == irq { + return Ok(vector_u8); + } + return Err(IrqError::Busy); + } + + self.vector_routes.push((vector, irq)); + Ok(vector_u8) + } + + fn irq_for_vector(&self, vector: usize) -> Option { + self.vector_routes + .iter() + .find_map(|(known_vector, irq)| (*known_vector == vector).then_some(*irq)) + } + fn remember_route(&mut self, route: AcpiGsiRoute) { if let Some(existing) = self.routes.iter_mut().find(|r| { r.controller_id == route.controller_id @@ -249,7 +191,7 @@ impl X86IoApic { let redirection_entries = max_entry.saturating_add(1); unsafe { - ioapic.init(irq_vector_base(info.gsi_base) as u8); + ioapic.init(MASKED_IOAPIC_PLACEHOLDER_VECTOR); for input in 0..=max_entry { let mut entry = ioapic.table_entry(input); entry.set_flags(entry.flags() | IrqFlags::MASKED); @@ -357,10 +299,9 @@ impl rdif_intc::Interface for X86IoApicIntc { if translation.id.hwirq != HwIrq(route.gsi) { return Err(IrqError::InvalidIrq); } - install_vector_route(route.vector, translation.id)?; + self.remember_vector_route(route.vector, translation.id)?; self.remember_route(*route); if self.set_route_enable(route, false) { - publish_ioapic_gsi_route(route)?; Ok(()) } else { Err(IrqError::Unsupported) @@ -409,7 +350,9 @@ impl PlatOp for Plat { } if crate::irq::domain_is_kind(irq.domain, crate::irq::IrqDomainKind::X86IoApic) { - return set_ioapic_gsi_enabled_from_irq(irq.hwirq.0, enable); + let intc = crate::irq::intc_by_domain(irq.domain)?; + let mut intc = intc.try_lock().map_err(|_| IrqError::Busy)?; + return intc.set_enabled(irq.hwirq, enable); } Err(IrqError::InvalidIrq) @@ -429,10 +372,10 @@ impl PlatOp for Plat { let Some(apic_id) = someboot::smp::cpu_idx_to_id(cpu_id) else { return Err(IrqError::InvalidCpu); }; - apic_id as u8 + u8::try_from(apic_id).map_err(|_| IrqError::InvalidCpu)? } }; - if set_ioapic_gsi_destination(irq.hwirq.0, dest) { + if set_ioapic_gsi_destination(irq.domain, irq.hwirq.0, dest)? { Ok(()) } else { Err(IrqError::NotFound) @@ -440,27 +383,34 @@ impl PlatOp for Plat { } fn send_ipi(irq: IrqId, target: crate::irq::IpiTarget) { - let vector = irq.hwirq.0 as u8; + let Ok(vector) = lapic::ipi_vector(irq) else { + warn!("refuse to send non-runtime IPI IRQ {irq:?}"); + return; + }; - unsafe { - match target { - crate::irq::IpiTarget::Current { .. } => { - send_lapic_ipi(0, ICR_FIXED_BASE | ICR_DEST_SELF | u32::from(vector)) - } - crate::irq::IpiTarget::Other { cpu_id } => { - let Some(apic_id) = someboot::smp::cpu_idx_to_id(cpu_id) else { - warn!("failed to resolve CPU index {cpu_id} to APIC ID"); - return; - }; - send_lapic_ipi(raw_apic_id(apic_id), ICR_FIXED_BASE | u32::from(vector)); - } - crate::irq::IpiTarget::AllExceptCurrent { .. } => { - send_lapic_ipi( - 0, - ICR_FIXED_BASE | ICR_DEST_ALL_EXCLUDING_SELF | u32::from(vector), - ); - } + let result = match target { + crate::irq::IpiTarget::Current { .. } => lapic::send_ipi( + 0, + lapic::ICR_FIXED_BASE | lapic::ICR_DEST_SELF | u32::from(vector), + ), + crate::irq::IpiTarget::Other { cpu_id } => { + let Some(apic_id) = someboot::smp::cpu_idx_to_id(cpu_id) else { + warn!("failed to resolve CPU index {cpu_id} to APIC ID"); + return; + }; + lapic::send_ipi_to_apic_id( + apic_id as u32, + lapic::ICR_FIXED_BASE | u32::from(vector), + ) } + crate::irq::IpiTarget::AllExceptCurrent { .. } => lapic::send_ipi( + 0, + lapic::ICR_FIXED_BASE | lapic::ICR_DEST_ALL_EXCLUDING_SELF | u32::from(vector), + ), + }; + + if let Err(err) = result { + warn!("failed to send runtime IPI vector {vector:#x}: {err:?}"); } } @@ -469,7 +419,27 @@ impl PlatOp for Plat { } fn begin_irq(raw: usize) -> Option { - trap_vector_irq_id(raw).map(ActiveIrq::new) + if raw == SPURIOUS_VECTOR { + return None; + } + + if let Some(irq) = local_vector_irq_id(raw) { + return Some(ActiveIrq::new(irq)); + } + + match ioapic_irq_for_vector(raw) { + Ok(Some(irq)) => Some(ActiveIrq::new(irq)), + Ok(None) => { + warn!("unrouted x86 interrupt vector {raw:#x}"); + lapic::eoi(); + None + } + Err(err) => { + warn!("failed to resolve x86 interrupt vector {raw:#x}: {err:?}"); + lapic::eoi(); + None + } + } } fn active_irq_id(active: &Self::ActiveIrq) -> IrqId { @@ -523,7 +493,7 @@ impl ActiveIrq { impl Drop for ActiveIrq { fn drop(&mut self) { - lapic_eoi(); + lapic::eoi(); } } @@ -551,15 +521,6 @@ fn resolve_acpi_route(route: irq_framework::AcpiGsiRoute) -> Result Result<(), IrqError> { - let slot = IOAPIC_GSI_ROUTES - .get(route.gsi as usize) - .ok_or(IrqError::InvalidIrq)?; - let packed = pack_ioapic_gsi_route(route.controller_address, route.controller_input)?; - slot.store(packed, Ordering::Release); - Ok(()) -} - fn route_to_irq_framework(route: AcpiGsiRoute) -> irq_framework::AcpiGsiRoute { irq_framework::AcpiGsiRoute { gsi: route.gsi, @@ -604,40 +565,29 @@ fn route_to_rdif(route: irq_framework::AcpiGsiRoute) -> AcpiGsiRoute { } } -fn install_vector_route(vector: usize, irq: IrqId) -> Result { - let vector_u8 = u8::try_from(vector).map_err(|_| IrqError::InvalidIrq)?; - if matches!(vector, APIC_TIMER_VECTOR | APIC_IPI_VECTOR) { - return Err(IrqError::Busy); - } - - let Some(slot) = VECTOR_ROUTES.get(vector) else { - return Err(IrqError::InvalidIrq); - }; - let raw = pack_irq(irq); - match slot.compare_exchange( - UNMAPPED_VECTOR_ROUTE, - raw, - Ordering::AcqRel, - Ordering::Acquire, - ) { - Ok(_) => {} - Err(old) if old == raw => {} - Err(_) => return Err(IrqError::Busy), - } - Ok(vector_u8) +fn set_ioapic_gsi_destination( + domain: crate::irq::IrqDomainId, + gsi: u32, + dest: u8, +) -> Result { + let intc = crate::irq::intc_by_domain(domain)?; + let mut intc = intc.try_lock().map_err(|_| IrqError::Busy)?; + let ioapic = intc + .typed_mut::() + .ok_or(IrqError::Unsupported)?; + Ok(ioapic.set_gsi_destination(gsi, dest)) } -fn set_ioapic_gsi_destination(gsi: u32, dest: u8) -> bool { - for intc in rdrive::get_list::() { - if intc.descriptor().name.starts_with("ACPI IOAPIC") - && let Ok(ioapic) = intc.downcast::() - && let Ok(mut ioapic) = ioapic.try_lock() - && ioapic.set_gsi_destination(gsi, dest) - { - return true; - } - } - false +fn ioapic_irq_for_vector(vector: usize) -> Result, IrqError> { + let Some(domain) = crate::irq::domain_by_kind_fast(crate::irq::IrqDomainKind::X86IoApic) else { + return Ok(None); + }; + let intc = crate::irq::intc_by_domain(domain)?; + let mut intc = intc.try_lock().map_err(|_| IrqError::Busy)?; + let ioapic = intc + .typed_mut::() + .ok_or(IrqError::Unsupported)?; + Ok(ioapic.irq_for_vector(vector)) } fn intx_flags(trigger: AcpiIrqTrigger, polarity: AcpiIrqPolarity) -> IrqFlags { @@ -651,55 +601,17 @@ fn intx_flags(trigger: AcpiIrqTrigger, polarity: AcpiIrqPolarity) -> IrqFlags { flags } -fn irq_vector_base(gsi_base: u32) -> usize { - rdrive::probe::acpi::PCI_INTX_VECTOR_BASE + gsi_base as usize -} - -fn lapic_eoi() { - unsafe { - lapic_write(LAPIC_REG_EOI, 0); - } -} - -fn raw_apic_id(id: usize) -> u32 { - (id as u32) << 24 -} - -unsafe fn send_lapic_ipi(destination: u32, icr_low: u32) { - unsafe { - lapic_write(LAPIC_REG_ICR_HIGH, destination); - lapic_write(LAPIC_REG_ICR_LOW, icr_low); - while lapic_read(LAPIC_REG_ICR_LOW) & ICR_DELIVERY_PENDING != 0 { - core::hint::spin_loop(); - } - } -} - -unsafe fn lapic_read(offset: u32) -> u32 { - let ptr = lapic_ptr(offset) as *const u32; - unsafe { ptr.read_volatile() } -} - -unsafe fn lapic_write(offset: u32, value: u32) { - let ptr = lapic_ptr(offset); - unsafe { - ptr.write_volatile(value); - } -} - -fn lapic_ptr(offset: u32) -> *mut u32 { - const IA32_APIC_BASE: u32 = 0x1b; - const LAPIC_BASE_MASK: u64 = 0xffff_f000; - let base = unsafe { x86::msr::rdmsr(IA32_APIC_BASE) & LAPIC_BASE_MASK } as usize; - unsafe { someboot::mem::phys_to_virt(base).add(offset as usize) }.cast() -} - #[cfg(all(test, any(unix, windows)))] mod tests { use super::*; - fn clear_vector_route(vector: usize) { - VECTOR_ROUTES[vector].store(UNMAPPED_VECTOR_ROUTE, Ordering::Release); + fn empty_ioapic_intc() -> X86IoApicIntc { + X86IoApicIntc { + ioapics: Vec::new(), + routes: Vec::new(), + vector_routes: Vec::new(), + destinations: Vec::new(), + } } #[test] @@ -712,7 +624,7 @@ mod tests { fn lapic_ipi_vector_is_cpu_local_not_ioapic_gsi() { let irq = lapic_ipi_irq_id(); assert_eq!(irq.domain, CPU_LOCAL_IRQ_DOMAIN); - assert_eq!(trap_vector_irq_id(APIC_IPI_VECTOR), Some(irq)); + assert_eq!(local_vector_irq_id(APIC_IPI_VECTOR), Some(irq)); assert_ne!( irq, ioapic_gsi_irq_id((APIC_IPI_VECTOR - rdrive::probe::acpi::PCI_INTX_VECTOR_BASE) as u32) @@ -729,45 +641,87 @@ mod tests { fn ioapic_vector_reverse_route_does_not_assume_base_plus_gsi() { let vector = rdrive::probe::acpi::PCI_INTX_VECTOR_BASE + 3; let irq = ioapic_gsi_irq_id(18); - install_vector_route(vector, irq).unwrap(); + let mut intc = empty_ioapic_intc(); + intc.remember_vector_route(vector, irq).unwrap(); - assert_eq!(trap_vector_irq_id(vector), Some(irq)); - assert_ne!(trap_vector_irq_id(vector), Some(ioapic_gsi_irq_id(3))); - - clear_vector_route(vector); + assert_eq!(intc.irq_for_vector(vector), Some(irq)); + assert_ne!(intc.irq_for_vector(vector), Some(ioapic_gsi_irq_id(3))); } #[test] - fn unknown_vector_is_cpu_local_so_it_can_still_eoi() { + fn unknown_vector_is_not_dispatched_as_cpu_local_irq() { let vector = 0x71; - clear_vector_route(vector); - let irq = trap_vector_irq_id(vector).expect("unknown vectors still need ActiveIrq"); - assert_eq!(irq.domain, CPU_LOCAL_IRQ_DOMAIN); - assert_eq!(irq.hwirq, HwIrq(vector as u32)); - assert_ne!(irq, ioapic_gsi_irq_id(vector as u32)); + assert_eq!(local_vector_irq_id(vector), None); + } + + #[test] + fn spurious_vector_is_not_dispatched() { + assert_eq!(local_vector_irq_id(SPURIOUS_VECTOR), None); } #[test] fn vector_route_rejects_reserved_out_of_range_and_collision() { + let mut intc = empty_ioapic_intc(); + assert_eq!( + intc.remember_vector_route(APIC_TIMER_VECTOR, ioapic_gsi_irq_id(1)), + Err(IrqError::Busy) + ); assert_eq!( - install_vector_route(APIC_TIMER_VECTOR, ioapic_gsi_irq_id(1)), + intc.remember_vector_route(APIC_IPI_VECTOR, ioapic_gsi_irq_id(1)), Err(IrqError::Busy) ); assert_eq!( - install_vector_route(usize::from(u8::MAX) + 1, ioapic_gsi_irq_id(1)), + intc.remember_vector_route(SPURIOUS_VECTOR, ioapic_gsi_irq_id(1)), + Err(IrqError::Busy) + ); + assert_eq!( + intc.remember_vector_route(0x1f, ioapic_gsi_irq_id(1)), + Err(IrqError::Busy) + ); + assert_eq!( + intc.remember_vector_route(usize::from(u8::MAX) + 1, ioapic_gsi_irq_id(1)), Err(IrqError::InvalidIrq) ); let vector = 0x72; let irq = ioapic_gsi_irq_id(7); - install_vector_route(vector, irq).unwrap(); + intc.remember_vector_route(vector, irq).unwrap(); assert_eq!( - install_vector_route(vector, ioapic_gsi_irq_id(8)), + intc.remember_vector_route(vector, ioapic_gsi_irq_id(8)), Err(IrqError::Busy) ); - assert_eq!(install_vector_route(vector, irq), Ok(vector as u8)); - clear_vector_route(vector); + assert_eq!(intc.remember_vector_route(vector, irq), Ok(vector as u8)); + } + + #[test] + fn ipi_vector_requires_runtime_ipi_irq_identity() { + assert_eq!( + lapic::ipi_vector(lapic_ipi_irq_id()), + Ok(APIC_IPI_VECTOR as u8) + ); + assert_eq!( + lapic::ipi_vector(lapic_timer_irq_id()), + Err(IrqError::InvalidIrq) + ); + assert_eq!( + lapic::ipi_vector(IrqId::new(CPU_LOCAL_IRQ_DOMAIN, HwIrq(0x41))), + Err(IrqError::InvalidIrq) + ); + } + + #[test] + fn xapic_destination_rejects_high_apic_ids_without_truncation() { + assert_eq!(lapic::xapic_destination(0xfe), Ok(0xfe00_0000)); + assert_eq!(lapic::xapic_destination(0x100), Err(IrqError::InvalidCpu)); + } + + #[test] + fn x2apic_icr_encodes_full_destination_id() { + let icr = lapic::x2apic_icr(0x1234_5678, lapic::ICR_FIXED_BASE | APIC_IPI_VECTOR as u32); + + assert_eq!(icr >> 32, 0x1234_5678); + assert_eq!(icr as u32, lapic::ICR_FIXED_BASE | APIC_IPI_VECTOR as u32); } #[test] diff --git a/platforms/somehal/src/arch/x86_64/vector.rs b/platforms/somehal/src/arch/x86_64/vector.rs new file mode 100644 index 0000000000..a25b558812 --- /dev/null +++ b/platforms/somehal/src/arch/x86_64/vector.rs @@ -0,0 +1,43 @@ +use crate::irq::{CPU_LOCAL_IRQ_DOMAIN, HwIrq, IrqError, IrqId, X86_LAPIC_DOMAIN}; + +pub(super) const APIC_TIMER_VECTOR: usize = 0x20; +pub(super) const APIC_IPI_VECTOR: usize = 0xf3; +pub(super) const SPURIOUS_VECTOR: usize = 0xff; + +pub(super) fn lapic_timer_irq_id() -> IrqId { + IrqId::new(X86_LAPIC_DOMAIN, HwIrq(0)) +} + +pub(super) fn lapic_ipi_irq_id() -> IrqId { + IrqId::new(CPU_LOCAL_IRQ_DOMAIN, HwIrq(APIC_IPI_VECTOR as u32)) +} + +#[cfg(test)] +pub(super) fn ioapic_gsi_irq_id(gsi: u32) -> IrqId { + IrqId::new(crate::irq::IrqDomainId(7), HwIrq(gsi)) +} + +pub(super) fn local_vector_irq_id(raw: usize) -> Option { + if raw == APIC_TIMER_VECTOR { + return Some(lapic_timer_irq_id()); + } + + if raw == APIC_IPI_VECTOR { + return Some(lapic_ipi_irq_id()); + } + + None +} + +pub(super) fn validate_external_vector(vector: usize) -> Result { + let vector_u8 = u8::try_from(vector).map_err(|_| IrqError::InvalidIrq)?; + if vector < 0x20 + || matches!( + vector, + APIC_TIMER_VECTOR | APIC_IPI_VECTOR | SPURIOUS_VECTOR + ) + { + return Err(IrqError::Busy); + } + Ok(vector_u8) +} diff --git a/platforms/somehal/src/irq.rs b/platforms/somehal/src/irq.rs index ee192bbfc0..3862faac25 100644 --- a/platforms/somehal/src/irq.rs +++ b/platforms/somehal/src/irq.rs @@ -161,11 +161,6 @@ pub fn intc_by_domain(domain: IrqDomainId) -> Result, IrqError> { } pub fn set_controller_irq_enabled(irq: IrqId, enabled: bool) -> Result<(), IrqError> { - #[cfg(target_arch = "x86_64")] - if domain_is_kind(irq.domain, IrqDomainKind::X86IoApic) { - return crate::arch::set_ioapic_gsi_enabled_from_irq(irq.hwirq.0, enabled); - } - let intc = intc_by_domain(irq.domain)?; let mut intc = intc.try_lock().map_err(|_| IrqError::Busy)?; intc.set_enabled(irq.hwirq, enabled) diff --git a/virtualization/axvm/src/host/arceos.rs b/virtualization/axvm/src/host/arceos.rs index c67c730ad6..6c4df20c69 100644 --- a/virtualization/axvm/src/host/arceos.rs +++ b/virtualization/axvm/src/host/arceos.rs @@ -252,19 +252,9 @@ pub(crate) fn make_irq_id(domain: u16, hwirq: u32) -> ArceOsIrqId { ) } -#[cfg(all(target_arch = "x86_64", not(test), not(feature = "plat-dyn")))] -pub(crate) fn set_irq_enable(irq: ArceOsIrqId, enabled: bool) { - if let Err(err) = modules::ax_hal::irq::set_enable(irq, enabled) { - warn!("failed to set forwarded host IRQ {irq:?} enabled={enabled}: {err:?}"); - } -} - -#[cfg(all(feature = "plat-dyn", target_arch = "x86_64", not(test)))] -pub(crate) fn set_ioapic_gsi_enabled_from_irq( - gsi: u32, - enabled: bool, -) -> Result<(), ArceOsIrqError> { - modules::ax_hal::irq::set_ioapic_gsi_enabled_from_irq(gsi, enabled) +#[cfg(all(target_arch = "x86_64", not(test)))] +pub(crate) fn set_irq_enable(irq: ArceOsIrqId, enabled: bool) -> Result<(), ArceOsIrqError> { + modules::ax_hal::irq::set_enable(irq, enabled) } #[cfg(target_arch = "x86_64")] diff --git a/virtualization/axvm/src/host/irq.rs b/virtualization/axvm/src/host/irq.rs index 63b5a244fe..76bb6c767b 100644 --- a/virtualization/axvm/src/host/irq.rs +++ b/virtualization/axvm/src/host/irq.rs @@ -27,63 +27,23 @@ pub(crate) fn request_shared_irq( #[cfg(test)] static TEST_ENABLED_IRQ_RAW: AtomicUsize = AtomicUsize::new(usize::MAX); -#[cfg(any(test, not(feature = "plat-dyn")))] -pub(crate) fn set_irq_enable(irq: IrqId, enabled: bool) { - set_irq_enable_impl(irq, enabled); +pub(crate) fn set_host_irq_enable(irq: IrqId, enabled: bool) -> Result<(), IrqError> { + set_host_irq_enable_impl(irq, enabled) } #[cfg(test)] -fn set_irq_enable_impl(irq: IrqId, enabled: bool) { +fn set_host_irq_enable_impl(irq: IrqId, enabled: bool) -> Result<(), IrqError> { if enabled { TEST_ENABLED_IRQ_RAW.store(irq_to_test_raw(irq), Ordering::Release); } else if TEST_ENABLED_IRQ_RAW.load(Ordering::Acquire) == irq_to_test_raw(irq) { TEST_ENABLED_IRQ_RAW.store(usize::MAX, Ordering::Release); } -} - -#[cfg(all(not(test), not(feature = "plat-dyn")))] -fn set_irq_enable_impl(irq: IrqId, enabled: bool) { - arceos::set_irq_enable(irq, enabled); -} - -pub(crate) fn set_ioapic_gsi_enabled_from_irq( - gsi: u32, - irq: IrqId, - enabled: bool, -) -> Result<(), IrqError> { - set_ioapic_gsi_enabled_from_irq_impl(gsi, irq, enabled) -} - -#[cfg(test)] -fn set_ioapic_gsi_enabled_from_irq_impl( - gsi: u32, - irq: IrqId, - enabled: bool, -) -> Result<(), IrqError> { - let _ = gsi; - set_irq_enable(irq, enabled); Ok(()) } -#[cfg(all(feature = "plat-dyn", not(test)))] -fn set_ioapic_gsi_enabled_from_irq_impl( - gsi: u32, - irq: IrqId, - enabled: bool, -) -> Result<(), IrqError> { - let _ = gsi; - arceos::set_ioapic_gsi_enabled_from_irq(irq.hwirq.0, enabled) -} - -#[cfg(all(not(feature = "plat-dyn"), not(test)))] -fn set_ioapic_gsi_enabled_from_irq_impl( - gsi: u32, - irq: IrqId, - enabled: bool, -) -> Result<(), IrqError> { - let _ = gsi; - set_irq_enable(irq, enabled); - Ok(()) +#[cfg(not(test))] +fn set_host_irq_enable_impl(irq: IrqId, enabled: bool) -> Result<(), IrqError> { + arceos::set_irq_enable(irq, enabled) } pub(crate) fn resolve_irq_source(source: IrqSource) -> Result { diff --git a/virtualization/axvm/src/runtime/x86_irq.rs b/virtualization/axvm/src/runtime/x86_irq.rs index 62b86f87cb..9eb1b11e36 100644 --- a/virtualization/axvm/src/runtime/x86_irq.rs +++ b/virtualization/axvm/src/runtime/x86_irq.rs @@ -485,7 +485,7 @@ fn set_forwarded_host_gsi_enabled(gsi: usize, enabled: bool) { return; } let irq = raw_to_host_irq(raw); - if let Err(err) = irq::set_ioapic_gsi_enabled_from_irq(gsi as u32, irq, enabled) { + if let Err(err) = irq::set_host_irq_enable(irq, enabled) { warn!( "failed to set forwarded IOAPIC GSI {gsi} host IRQ {irq:?} enabled={enabled}: {err:?}" ); @@ -508,7 +508,7 @@ fn mask_forwarded_host_gsi(gsi: usize) -> bool { } let irq = raw_to_host_irq(raw); - if let Err(err) = irq::set_ioapic_gsi_enabled_from_irq(gsi as u32, irq, false) { + if let Err(err) = irq::set_host_irq_enable(irq, false) { IOAPIC_IRQ_MASKED.fetch_and(!bit, Ordering::AcqRel); warn!("failed to mask forwarded IOAPIC GSI {gsi} host IRQ {irq:?}: {err:?}"); return false; @@ -534,7 +534,7 @@ fn unmask_forwarded_host_gsi(gsi: usize) { } let irq = raw_to_host_irq(raw); - if let Err(err) = irq::set_ioapic_gsi_enabled_from_irq(gsi as u32, irq, true) { + if let Err(err) = irq::set_host_irq_enable(irq, true) { warn!("failed to unmask forwarded IOAPIC GSI {gsi} host IRQ {irq:?}: {err:?}"); return; }