Skip to content

api: CPU placement policy, topology reporting and structured error codes - #155

Draft
rucoder wants to merge 4 commits into
lf-edge:mainfrom
rucoder:rucoder/cpu-placement-api
Draft

api: CPU placement policy, topology reporting and structured error codes#155
rucoder wants to merge 4 commits into
lf-edge:mainfrom
rucoder:rucoder/cpu-placement-api

Conversation

@rucoder

@rucoder rucoder commented Aug 17, 2026

Copy link
Copy Markdown

Adds the API surface for SMT/NUMA-aware CPU placement of edge applications: the
controller states what a workload needs, and the device reports what it can do
and what it did.

Draft for API review. The device-side implementation lives in
lf-edge/eve#6335 and is currently built against this branch through a
replace directive. Nothing here is final; field numbers in particular are
whatever came out of writing the messages, and are free to move.

Config: what a workload asks for (config/vm.proto)

Seven new VmConfig fields, named after the Kubernetes CPUManager and Topology
Manager concepts they mirror, so a controller that already speaks Kubernetes has
no new vocabulary to learn:

  • cpu_policyshared or dedicated. Dedicated is self-sufficient: it
    implies the legacy pin_cpu flag, which stays honoured on its own for older
    controllers.
  • full_pcpus_only — allocate in whole physical cores, so no other workload
    runs on a sibling thread of a core this one owns.
  • threads_per_core — how many SMT siblings of each owned core become vCPUs.
    2 exposes both; 1 parks the sibling, which the workload still consumes: a
    neighbour placed there would evict its cache lines and contend for its
    execution units, which is the whole point of asking for a whole core.
  • numa_policynone, best_effort, restricted, single_numa_node.
  • io_placement — whether the emulator/IO threads stay on the workload's own
    cores or are kept off them.
  • isolation_tier — the strength of isolation the workload needs, as intent
    rather than as a set of mechanisms.
  • disruption_policy — whether a node-level action may take the workload down.

The zero value of every field means "no preference", so a controller that sets
none of them gets exactly today's behaviour.

Info: what the node can do, and what it did

  • info/hardware.proto — CPU entries carry their real topology coordinates
    (socket, physical core, NUMA node, L3 domain) and frequencies; cache domains
    report which CPUs share them; PCI devices report their NUMA node and local
    CPUs. NodeCapabilities states the node's granular abilities rather than a
    precomputed tier, so tiers are derived from capabilities instead of being
    hardcoded on the device.
  • info/info.protoZInfoDevice.cpu_pools reports per-pool CPU utilization
    (housekeeping, dedicated, isolated) with both CPU sets and whole-core counts,
    which is what lets a controller answer "will this fit?" before a deploy and
    explain a shortage after one. API_CAPABILITY_CPU_PLACEMENT_POLICY is the
    gate a controller uses to know the device honours the config fields at all.
  • info/common.protoErrorInfo.error_code carries a stable, machine-readable
    code alongside the existing free text, so a controller can distinguish a
    shortage a repack would fix from one nothing would fix, and from a request that
    can never be satisfied, without pattern-matching prose.

Open questions for the controller team

  1. restricted and single_numa_node are currently the same thing on the
    device. They differ in Kubernetes only by how many other aligned resources
    are weighed, and CPU placement weighs none yet; they are kept distinct in the
    API so the distinction can become real once NUMA-aware placement of PCI
    devices lands.
  2. isolation_tier: hard and disruption_policy: protect are refused by the
    device today rather than accepted and quietly not enforced. A controller
    should treat them as not-yet-available.
  3. Unknown enum values are treated by the device as "no preference". That is only
    safe because the controller is expected to gate on api_capability — if you
    would rather have the device reject unknown values, say so.

…r codes

One-shot additive API for topology-aware CPU placement (SMT/NUMA-aware
pinning and isolation), sized for all delivery phases so later phases
only populate and honor a growing subset:

- info/hardware.proto: extend the per-logical-CPU CPU message with
  topology coordinates (socket_id, core_id, numa_node, l2_id, l3_id),
  core class (P/E/LP) and base/max frequency; add CacheDomain (cache ->
  cores linkage) and CPUCapabilities (CPU-silicon RDT facts) to CPUInfo;
  add granular kernel/boot-level isolation facts (NodeCapabilities) to
  HardwareInventory — consumers derive higher-level notions such as
  "isolation tier achievable for this workload type" from these
  ingredients instead of the node precomputing them; add NUMA/CPU
  affinity to PCIDevice for future device-local placement and
  RDT-for-I/O. NodeCapabilities is deliberately scoped to what the
  hardware and the RUNNING KERNEL provide; what the EVE software is able
  to do with those facts is reported separately (see below), so a
  hardware fact never changes meaning when EVE gains or loses a feature.

- info/info.proto: report EVE-software ability separately from hardware
  facts. Add OptionalCapabilities.managed_cpu_isolation (EVE can derive a
  CPU-isolation kernel command line from its own placement plan and apply
  it, reboot-gated) — an EVE-software ability, not a hardware or kernel
  fact. Add API_CAPABILITY_CPU_PLACEMENT_POLICY so a controller can tell
  whether an EVE parses and honors the new VmConfig fields at all,
  following the existing api_capability convention. A consumer deciding
  whether a feature is offerable must consult both channels: hardware/
  kernel capable AND EVE software capable — they fail differently, one is
  fixable by updating EVE, the other is not fixable on that node.

- config/vm.proto: per-workload CPU placement intent on VmConfig
  (cpu_policy, full_pcpus_only, threads_per_core, numa_policy,
  io_placement, isolation_tier, disruption_policy), Kubernetes-aligned
  vocabulary. Legacy pin_cpu=true maps to CPU_POLICY_DEDICATED with
  default allocation. Document the two-channel capability gating a
  controller must apply before offering or sending these fields,
  mirroring the API_CAPABILITY_ENFORCED_NET_INTERFACE_ORDER precedent.

- info/common.proto: machine-parseable, namespaced error_code on
  ErrorInfo (e.g. cpu.placement.insufficient) alongside the free-text
  description.

Signed-off-by: Mikhail Malyshev <mike.malyshev@gmail.com>
…d structured error codes

- add generated files.

Signed-off-by: Mikhail Malyshev <mike.malyshev@gmail.com>
The design requires the node to report, per CPU pool (housekeeping /
dedicated / isolated), its CPU set and how much of it is still free, so
a controller can answer "will this workload fit?" before a deploy and
explain a cpu.placement.insufficient failure after one.

Both the sets and the summary counts are reported. Counts answer "how
much is left" directly; the sets let a consumer group CPUs into physical
cores via the (socket_id, core_id) coordinates already in CPUInfo, which
is what distinguishes free threads sitting on partly-allocated cores
from genuinely free whole cores. A single free count would answer the
question wrongly for one of the two request shapes.

This is dynamic state, so it rides ZInfoDevice (change-driven and
periodic) rather than the cached hardware inventory.

Signed-off-by: Mikhail Malyshev <mike.malyshev@gmail.com>
The design requires the node to report, per CPU pool (housekeeping /
dedicated / isolated), its CPU set and how much of it is still free, so
a controller can answer "will this workload fit?" before a deploy and
explain a cpu.placement.insufficient failure after one.

Both the sets and the summary counts are reported. Counts answer "how
much is left" directly; the sets let a consumer group CPUs into physical
cores via the (socket_id, core_id) coordinates already in CPUInfo, which
is what distinguishes free threads sitting on partly-allocated cores
from genuinely free whole cores. A single free count would answer the
question wrongly for one of the two request shapes.

This is dynamic state, so it rides ZInfoDevice (change-driven and
periodic) rather than the cached hardware inventory.

- add generated files.

Signed-off-by: Mikhail Malyshev <mike.malyshev@gmail.com>
rucoder added a commit to rucoder/eve that referenced this pull request Aug 17, 2026
The device-side code in this branch needs API that has not landed upstream yet:
the VmConfig CPU placement fields, the CPU topology and capability reporting on
device info, ZInfoDevice.cpu_pools, API_CAPABILITY_CPU_PLACEMENT_POLICY and
ErrorInfo.error_code. Without them nothing here compiles, so this commit points
pillar and evetest at the fork carrying the proposed API (lf-edge/eve-api#155)
through a replace directive, and vendors it.

This commit exists only so the branch can be built, run and reviewed while the
API is under discussion. It must be dropped and replaced by an ordinary
`make bump-eve-api` once the API lands in lf-edge/eve-api: a replace directive
pointing at a personal fork breaks dependency tracking and SBOM/licensing, and
is never acceptable on master.

Signed-off-by: Mikhail Malyshev <mike.malyshev@gmail.com>
@rucoder
rucoder requested a review from gkodali-zededa August 17, 2026 20:12
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant