api: CPU placement policy, topology reporting and structured error codes - #155
Draft
rucoder wants to merge 4 commits into
Draft
api: CPU placement policy, topology reporting and structured error codes#155rucoder wants to merge 4 commits into
rucoder wants to merge 4 commits into
Conversation
…r codes One-shot additive API for topology-aware CPU placement (SMT/NUMA-aware pinning and isolation), sized for all delivery phases so later phases only populate and honor a growing subset: - info/hardware.proto: extend the per-logical-CPU CPU message with topology coordinates (socket_id, core_id, numa_node, l2_id, l3_id), core class (P/E/LP) and base/max frequency; add CacheDomain (cache -> cores linkage) and CPUCapabilities (CPU-silicon RDT facts) to CPUInfo; add granular kernel/boot-level isolation facts (NodeCapabilities) to HardwareInventory — consumers derive higher-level notions such as "isolation tier achievable for this workload type" from these ingredients instead of the node precomputing them; add NUMA/CPU affinity to PCIDevice for future device-local placement and RDT-for-I/O. NodeCapabilities is deliberately scoped to what the hardware and the RUNNING KERNEL provide; what the EVE software is able to do with those facts is reported separately (see below), so a hardware fact never changes meaning when EVE gains or loses a feature. - info/info.proto: report EVE-software ability separately from hardware facts. Add OptionalCapabilities.managed_cpu_isolation (EVE can derive a CPU-isolation kernel command line from its own placement plan and apply it, reboot-gated) — an EVE-software ability, not a hardware or kernel fact. Add API_CAPABILITY_CPU_PLACEMENT_POLICY so a controller can tell whether an EVE parses and honors the new VmConfig fields at all, following the existing api_capability convention. A consumer deciding whether a feature is offerable must consult both channels: hardware/ kernel capable AND EVE software capable — they fail differently, one is fixable by updating EVE, the other is not fixable on that node. - config/vm.proto: per-workload CPU placement intent on VmConfig (cpu_policy, full_pcpus_only, threads_per_core, numa_policy, io_placement, isolation_tier, disruption_policy), Kubernetes-aligned vocabulary. Legacy pin_cpu=true maps to CPU_POLICY_DEDICATED with default allocation. Document the two-channel capability gating a controller must apply before offering or sending these fields, mirroring the API_CAPABILITY_ENFORCED_NET_INTERFACE_ORDER precedent. - info/common.proto: machine-parseable, namespaced error_code on ErrorInfo (e.g. cpu.placement.insufficient) alongside the free-text description. Signed-off-by: Mikhail Malyshev <mike.malyshev@gmail.com>
…d structured error codes - add generated files. Signed-off-by: Mikhail Malyshev <mike.malyshev@gmail.com>
The design requires the node to report, per CPU pool (housekeeping / dedicated / isolated), its CPU set and how much of it is still free, so a controller can answer "will this workload fit?" before a deploy and explain a cpu.placement.insufficient failure after one. Both the sets and the summary counts are reported. Counts answer "how much is left" directly; the sets let a consumer group CPUs into physical cores via the (socket_id, core_id) coordinates already in CPUInfo, which is what distinguishes free threads sitting on partly-allocated cores from genuinely free whole cores. A single free count would answer the question wrongly for one of the two request shapes. This is dynamic state, so it rides ZInfoDevice (change-driven and periodic) rather than the cached hardware inventory. Signed-off-by: Mikhail Malyshev <mike.malyshev@gmail.com>
The design requires the node to report, per CPU pool (housekeeping / dedicated / isolated), its CPU set and how much of it is still free, so a controller can answer "will this workload fit?" before a deploy and explain a cpu.placement.insufficient failure after one. Both the sets and the summary counts are reported. Counts answer "how much is left" directly; the sets let a consumer group CPUs into physical cores via the (socket_id, core_id) coordinates already in CPUInfo, which is what distinguishes free threads sitting on partly-allocated cores from genuinely free whole cores. A single free count would answer the question wrongly for one of the two request shapes. This is dynamic state, so it rides ZInfoDevice (change-driven and periodic) rather than the cached hardware inventory. - add generated files. Signed-off-by: Mikhail Malyshev <mike.malyshev@gmail.com>
rucoder
added a commit
to rucoder/eve
that referenced
this pull request
Aug 17, 2026
The device-side code in this branch needs API that has not landed upstream yet: the VmConfig CPU placement fields, the CPU topology and capability reporting on device info, ZInfoDevice.cpu_pools, API_CAPABILITY_CPU_PLACEMENT_POLICY and ErrorInfo.error_code. Without them nothing here compiles, so this commit points pillar and evetest at the fork carrying the proposed API (lf-edge/eve-api#155) through a replace directive, and vendors it. This commit exists only so the branch can be built, run and reviewed while the API is under discussion. It must be dropped and replaced by an ordinary `make bump-eve-api` once the API lands in lf-edge/eve-api: a replace directive pointing at a personal fork breaks dependency tracking and SBOM/licensing, and is never acceptable on master. Signed-off-by: Mikhail Malyshev <mike.malyshev@gmail.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Adds the API surface for SMT/NUMA-aware CPU placement of edge applications: the
controller states what a workload needs, and the device reports what it can do
and what it did.
Draft for API review. The device-side implementation lives in
lf-edge/eve#6335 and is currently built against this branch through a
replacedirective. Nothing here is final; field numbers in particular arewhatever came out of writing the messages, and are free to move.
Config: what a workload asks for (
config/vm.proto)Seven new
VmConfigfields, named after the Kubernetes CPUManager and TopologyManager concepts they mirror, so a controller that already speaks Kubernetes has
no new vocabulary to learn:
cpu_policy—sharedordedicated. Dedicated is self-sufficient: itimplies the legacy
pin_cpuflag, which stays honoured on its own for oldercontrollers.
full_pcpus_only— allocate in whole physical cores, so no other workloadruns on a sibling thread of a core this one owns.
threads_per_core— how many SMT siblings of each owned core become vCPUs.2 exposes both; 1 parks the sibling, which the workload still consumes: a
neighbour placed there would evict its cache lines and contend for its
execution units, which is the whole point of asking for a whole core.
numa_policy—none,best_effort,restricted,single_numa_node.io_placement— whether the emulator/IO threads stay on the workload's owncores or are kept off them.
isolation_tier— the strength of isolation the workload needs, as intentrather than as a set of mechanisms.
disruption_policy— whether a node-level action may take the workload down.The zero value of every field means "no preference", so a controller that sets
none of them gets exactly today's behaviour.
Info: what the node can do, and what it did
info/hardware.proto— CPU entries carry their real topology coordinates(socket, physical core, NUMA node, L3 domain) and frequencies; cache domains
report which CPUs share them; PCI devices report their NUMA node and local
CPUs.
NodeCapabilitiesstates the node's granular abilities rather than aprecomputed tier, so tiers are derived from capabilities instead of being
hardcoded on the device.
info/info.proto—ZInfoDevice.cpu_poolsreports per-pool CPU utilization(housekeeping, dedicated, isolated) with both CPU sets and whole-core counts,
which is what lets a controller answer "will this fit?" before a deploy and
explain a shortage after one.
API_CAPABILITY_CPU_PLACEMENT_POLICYis thegate a controller uses to know the device honours the config fields at all.
info/common.proto—ErrorInfo.error_codecarries a stable, machine-readablecode alongside the existing free text, so a controller can distinguish a
shortage a repack would fix from one nothing would fix, and from a request that
can never be satisfied, without pattern-matching prose.
Open questions for the controller team
restrictedandsingle_numa_nodeare currently the same thing on thedevice. They differ in Kubernetes only by how many other aligned resources
are weighed, and CPU placement weighs none yet; they are kept distinct in the
API so the distinction can become real once NUMA-aware placement of PCI
devices lands.
isolation_tier: hardanddisruption_policy: protectare refused by thedevice today rather than accepted and quietly not enforced. A controller
should treat them as not-yet-available.
safe because the controller is expected to gate on
api_capability— if youwould rather have the device reject unknown values, say so.