diff --git a/CAPABILITIES.md b/CAPABILITIES.md new file mode 100644 index 00000000..6678d312 --- /dev/null +++ b/CAPABILITIES.md @@ -0,0 +1,113 @@ +# Capabilities + +A Device reports what it supports so a Controller can avoid sending +configuration the Device would ignore, and avoid waiting for messages the Device +will never send. There are **three independent mechanisms**, all in `ZInfoDevice` +(see [info.proto](./proto/info/info.proto)), with **different semantics**. +Conflating them is a correctness bug. + +| mechanism | field | kind | +| --- | --- | --- | +| `APICapability` | `api_capability` | monotonic level | +| `OptionalCapabilities` | `optional_capabilities` | independent booleans | +| `Capabilities` | `capabilities` | independent booleans (hardware) | + +## APICapability — a level, not a set + +`APICapability` covers two kinds of support: + +1. **`EdgeDevConfig` fields the Device parses.** Set one on a Device below the + level and it is silently ignored. +2. **Messages or fields the Device sends.** A Controller expecting one from a + Device below the level waits for it indefinitely. + +**A larger value implies every smaller one.** A Controller MUST test with `>=`: + +```text +if device.api_capability >= API_CAPABILITY_MTU { + // safe to set NetworkConfig.mtu and NetworkInstanceConfig.mtu +} +``` + +Equality and set-membership tests are wrong. A Device reporting +`API_CAPABILITY_SMART_REPORT` (20) also supports 1 through 19, and EVE-OS +reports a single top value rather than a set. + +`API_CAPABILITY_UNSPECIFIED` (0) also covers Devices predating the field, so it +means "none of the below", not "unknown, try anyway". + +### What each value covers + +`cfg` = an `EdgeDevConfig` field the Device parses. `rpt` = something the Device +sends. Rows marked **(?)** are inferred from the enum comment and the commit that +introduced the value, not stated anywhere authoritative — corrections welcome. + +| value | | covers | +| --- | --- | --- | +| `RETRY_UPDATE` = 1 | cfg | `BaseOS.retry_update` | +| `SHUTDOWN` = 2 | cfg | `EdgeDevConfig.shutdown` | +| `START_DELAY_IN_SECONDS` = 3 | cfg | `AppInstanceConfig.start_delay_in_seconds` | +| `EDGEVIEW` = 4 | cfg | `EdgeDevConfig.edgeview`, `EdgeViewConfig.token` | +| `VOLUME_SNAPSHOTS` = 5 | cfg | `AppInstanceConfig.snapshot`, `SnapshotConfig` **(?)** | +| `NETWORK_INSTANCE_ROUTING` = 6 | cfg | `NetworkInstanceConfig.static_routes`, `.propagate_connected_routes`, and `IPRoute` | +| `BOOT_MODE` = 7 | cfg | `VmConfig.boot_mode` | +| `MTU` = 8 | cfg | `NetworkConfig.mtu`, `NetworkInstanceConfig.mtu` | +| `ADAPTER_USER_LABELS` = 9 | cfg | `SystemAdapter.shared_labels` | +| `ENFORCED_NET_INTERFACE_ORDER` = 10 | cfg | `VmConfig.enforce_network_interface_order`, and hence `NetworkAdapter.interface_order` and `Adapter.interface_order` | +| `NTPS_FQDN` = 11 | cfg | NTP servers as FQDN, and more than one — `ipspec.ntp`, `ipspec.more_ntp` **(?)** | +| `WIN_LIC_PASSTHROUGH` = 12 | cfg | `VmConfig.enable_oem_win_license_key` | +| `VOLUME_SNAPSHOTS_IMMEDIATE` = 13 | cfg | `SnapshotType.SNAPSHOT_TYPE_IMMEDIATE` **(?)** | +| `ENCRYPTED_PATCH_ENVELOPE` = 14 | cfg | `EveBinaryArtifact.encrypted_inline`, `.encrypted_volumeref`, `.metadata_cipher_data` | +| `SINGLE_STACK_IP_NETWORK` = 15 | cfg | `NetworkType.V4Only`, `NetworkType.V6Only` | +| `CELLULAR_ATTACH_CONFIG` = 16 | cfg | `CellularAccessPoint.attach_apn`, `.attach_ip_type`, `.attach_auth_protocol` | +| `EDGEVIEW_AUTHENTICATION` = 17 | cfg | EdgeView command authentication; no single field — see [EDGEVIEW-CONTAINER-API.md](https://github.com/lf-edge/eve/blob/master/docs/EDGEVIEW-CONTAINER-API.md) **(?)** | +| `DISABLE_VTPM` = 18 | cfg | `VmConfig.disable_vtpm` | +| `LOC_REBOOT_COLLECT_INFO` = 19 | cfg | LOC-initiated reboot and collect-info; `LOCConfig.datastore_collect_info_id` **(?)** | +| `SMART_REPORT` = 20 | rpt | S.M.A.R.T. data in `ZHardwareHealth.disks`, superseding the deprecated `ZInfoHardware.disks` | +| `REPORT_TPM_EVENTLOG` = 21 | rpt | `ZAttestQuote.tpm_binary_event_log`, superseding the deprecated `ZAttestQuote.event_log` | + +### Adding a value + +Appending value *N* asserts that a Device reporting it also supports everything +below. So: + +1. Append at the end. Never insert, never renumber. +2. Name the field(s) or message(s) it covers, in the enum comment and in the + table above. +3. If the capability depends on build flavor or hardware rather than on version, + it belongs in `OptionalCapabilities` instead. + +## OptionalCapabilities — independent booleans + +Not monotonic; test each separately. These describe properties that vary by +EVE-OS build flavor rather than by version. + +| field | meaning | matters because | +| --- | --- | --- | +| `hv_type_kubevirt` | Device runs the Kubevirt hypervisor | required before sending `EdgeNodeCluster`; the KVM flavor cannot join a cluster | +| `hw_inventory_support` | Device can produce `HardwareInventory` | distinguishes "found no hardware" from "cannot report" when `ZInfoHardware.inventory` is empty | +| `etcd_snapshot` | Device supports etcd snapshots | EVE-k cluster operations | + +An absent boolean means "not supported or not reported". Do not infer support +from a Device predating the field. + +## Capabilities — hardware + +Independent booleans describing the platform, not the software: + +| field | meaning | +| --- | --- | +| `HWAssistedVirtualization` | VMX/SVM on amd64, virtualization extensions on arm64 | +| `IOVirtualization` | IOMMU / I/O virtualization support | + +These bound what a Device can ever run: device passthrough +(`AppInstanceConfig.adapters`, SR-IOV VFs) will not work without +`IOVirtualization`, whatever `APICapability` says. + +## Checklist for Controller implementers + +1. Test `APICapability` with `>=`; treat `0` as "none". +2. Test each `OptionalCapabilities` and `Capabilities` boolean individually. +3. Check the gate before sending any field in the table above. +4. A Device does not report being sent a gated field it does not understand — it + just ignores it. That silence is why these gates exist. diff --git a/README.md b/README.md index 51e99d58..36947c51 100644 --- a/README.md +++ b/README.md @@ -17,6 +17,7 @@ Documentation is in markdown files in this directory, specifically: * The protocol in [APIv2.md](./APIv2.md) * Local profile overrides in [PROFILE.md](./PROFILE.md) * Object signing in [OBJECT-SIGNING.md](./OBJECT-SIGNING.md) +* Device capability reporting in [CAPABILITIES.md](./CAPABILITIES.md) ### Message definitions diff --git a/go/info/info.pb.go b/go/info/info.pb.go index e212bc7a..f0aa73eb 100644 --- a/go/info/info.pb.go +++ b/go/info/info.pb.go @@ -870,11 +870,15 @@ func (StorageTypeInfo) EnumDescriptor() ([]byte, []int) { return file_info_info_proto_rawDescGZIP(), []int{12} } -// Capabilities indicates features in the EdgeDevConfig where there is -// no easy way to otherwise determine whether or not they are parsed and -// supported by EVE-OS -// A larger number indicates all lower numbers are also supported thus -// this works similar to a version field for the EdgeDevConfig support. +// APICapability indicates API support a controller cannot otherwise detect: +// both EdgeDevConfig fields EVE-OS parses, and messages EVE-OS sends - without +// which a controller would wait indefinitely for e.g. the S.M.A.R.T. +// information in ZHardwareHealth. +// A larger number implies all lower numbers, so this works like a version +// field and a controller must compare with >= rather than test for equality. +// API_CAPABILITY_UNSPECIFIED also covers EVE-OS versions predating the field. +// Contrast OptionalCapabilities: independent booleans, varying by build flavor. +// CAPABILITIES.md lists what each value below covers. type APICapability int32 const ( @@ -891,12 +895,12 @@ const ( APICapability_API_CAPABILITY_ENFORCED_NET_INTERFACE_ORDER APICapability = 10 // EVE is able to enforce the user-defined order of application network interfaces APICapability_API_CAPABILITY_NTPS_FQDN APICapability = 11 // Allow to set NTP server via FQDN instead of only IP and allow setting several NTP servers APICapability_API_CAPABILITY_WIN_LIC_PASSTHROUGH APICapability = 12 // Support for passing through OEM Windows license from host's the ACPI tables to the VM - APICapability_API_CAPABILITY_VOLUME_SNAPSHOTS_IMMEDIATE APICapability = 13 // Volume snapshots supported + APICapability_API_CAPABILITY_VOLUME_SNAPSHOTS_IMMEDIATE APICapability = 13 // SNAPSHOT_TYPE_IMMEDIATE supported APICapability_API_CAPABILITY_ENCRYPTED_PATCH_ENVELOPE APICapability = 14 // Support for Patch Envelope Encryption APICapability_API_CAPABILITY_SINGLE_STACK_IP_NETWORK APICapability = 15 // Support for V4Only and V6Only NetworkType APICapability_API_CAPABILITY_CELLULAR_ATTACH_CONFIG APICapability = 16 // Support cellular attach configuration APICapability_API_CAPABILITY_EDGEVIEW_AUTHENTICATION APICapability = 17 // EdgeView authentication, see https://github.com/lf-edge/eve/blob/master/docs/EDGEVIEW-CONTAINER-API.md - APICapability_API_CAPABILITY_DISABLE_VTPM APICapability = 18 // Support for disabling + APICapability_API_CAPABILITY_DISABLE_VTPM APICapability = 18 // VmConfig.disable_vtpm supported APICapability_API_CAPABILITY_LOC_REBOOT_COLLECT_INFO APICapability = 19 // Support for rebooting and collect-info from LOC APICapability_API_CAPABILITY_SMART_REPORT APICapability = 20 // Support for S.M.A.R.T. info on physical storage devices APICapability_API_CAPABILITY_REPORT_TPM_EVENTLOG APICapability = 21 // Support for reporting TPM Event Log "as is" without parsing and selectively reporting events @@ -4178,7 +4182,7 @@ type ZInfoDevice struct { ShutdownConfigCounter uint32 `protobuf:"varint,49,opt,name=shutdown_config_counter,json=shutdownConfigCounter,proto3" json:"shutdown_config_counter,omitempty"` // state of attestation process of eve AttestationInfo *AttestationInfo `protobuf:"bytes,50,opt,name=attestation_info,json=attestationInfo,proto3" json:"attestation_info,omitempty"` - // Capability indicating which new EdgeDevConfig fields which are supported + // Level of API support; see APICapability ApiCapability APICapability `protobuf:"varint,51,opt,name=api_capability,json=apiCapability,proto3,enum=org.lfedge.eve.info.APICapability" json:"api_capability,omitempty"` // Reports the remote access status RemoteAccessDisabled bool `protobuf:"varint,52,opt,name=remote_access_disabled,json=remoteAccessDisabled,proto3" json:"remote_access_disabled,omitempty"` @@ -4610,14 +4614,17 @@ func (x *ZInfoDevice) GetEnrolledCerts() []*CertInfo { // OptionalCapabilities indicates any additional capabilities device wants // to publish to controller. For example Kubevirt hypervisor is not supported by -// all eve flavors. +// all eve flavors. Unlike APICapability these are independent booleans, each +// tested separately; they vary by build flavor rather than by version. type OptionalCapabilities struct { state protoimpl.MessageState sizeCache protoimpl.SizeCache unknownFields protoimpl.UnknownFields // Virtualization type Kubevirt - HvTypeKubevirt bool `protobuf:"varint,1,opt,name=hv_type_kubevirt,json=hvTypeKubevirt,proto3" json:"hv_type_kubevirt,omitempty"` + HvTypeKubevirt bool `protobuf:"varint,1,opt,name=hv_type_kubevirt,json=hvTypeKubevirt,proto3" json:"hv_type_kubevirt,omitempty"` + // Device can report HardwareInventory in ZInfoHardware. When false, an empty + // inventory means the device cannot produce one, not that it found no hardware. HwInventorySupport bool `protobuf:"varint,2,opt,name=hw_inventory_support,json=hwInventorySupport,proto3" json:"hw_inventory_support,omitempty"` // Device supports etcd snapshots (e.g. eve-k flavor) EtcdSnapshot bool `protobuf:"varint,3,opt,name=etcd_snapshot,json=etcdSnapshot,proto3" json:"etcd_snapshot,omitempty"` @@ -7681,7 +7688,8 @@ func (*ZInfoMsg_ClusterInfo) isZInfoMsg_InfoContent() {} func (*ZInfoMsg_ClusterUpdateInfo) isZInfoMsg_InfoContent() {} -// Information about hardware capabilities of node +// Information about hardware capabilities of node. Distinct from APICapability +// (API support level) and OptionalCapabilities (per-flavor software support). type Capabilities struct { state protoimpl.MessageState sizeCache protoimpl.SizeCache diff --git a/proto/info/info.proto b/proto/info/info.proto index 4192b4d1..934819e3 100644 --- a/proto/info/info.proto +++ b/proto/info/info.proto @@ -620,7 +620,7 @@ message ZInfoDevice { // state of attestation process of eve AttestationInfo attestation_info = 50; - // Capability indicating which new EdgeDevConfig fields which are supported + // Level of API support; see APICapability APICapability api_capability = 51; // Reports the remote access status @@ -644,20 +644,27 @@ message ZInfoDevice { // OptionalCapabilities indicates any additional capabilities device wants // to publish to controller. For example Kubevirt hypervisor is not supported by -// all eve flavors. +// all eve flavors. Unlike APICapability these are independent booleans, each +// tested separately; they vary by build flavor rather than by version. message OptionalCapabilities { // Virtualization type Kubevirt bool hv_type_kubevirt = 1; + // Device can report HardwareInventory in ZInfoHardware. When false, an empty + // inventory means the device cannot produce one, not that it found no hardware. bool hw_inventory_support = 2; // Device supports etcd snapshots (e.g. eve-k flavor) bool etcd_snapshot = 3; } -// Capabilities indicates features in the EdgeDevConfig where there is -// no easy way to otherwise determine whether or not they are parsed and -// supported by EVE-OS -// A larger number indicates all lower numbers are also supported thus -// this works similar to a version field for the EdgeDevConfig support. +// APICapability indicates API support a controller cannot otherwise detect: +// both EdgeDevConfig fields EVE-OS parses, and messages EVE-OS sends - without +// which a controller would wait indefinitely for e.g. the S.M.A.R.T. +// information in ZHardwareHealth. +// A larger number implies all lower numbers, so this works like a version +// field and a controller must compare with >= rather than test for equality. +// API_CAPABILITY_UNSPECIFIED also covers EVE-OS versions predating the field. +// Contrast OptionalCapabilities: independent booleans, varying by build flavor. +// CAPABILITIES.md lists what each value below covers. enum APICapability { API_CAPABILITY_UNSPECIFIED = 0; API_CAPABILITY_RETRY_UPDATE = 1; // BaseOs.retry_update counter supported @@ -672,16 +679,17 @@ enum APICapability { API_CAPABILITY_ENFORCED_NET_INTERFACE_ORDER = 10; // EVE is able to enforce the user-defined order of application network interfaces API_CAPABILITY_NTPS_FQDN = 11; // Allow to set NTP server via FQDN instead of only IP and allow setting several NTP servers API_CAPABILITY_WIN_LIC_PASSTHROUGH = 12; // Support for passing through OEM Windows license from host's the ACPI tables to the VM - API_CAPABILITY_VOLUME_SNAPSHOTS_IMMEDIATE = 13; // Volume snapshots supported + API_CAPABILITY_VOLUME_SNAPSHOTS_IMMEDIATE = 13; // SNAPSHOT_TYPE_IMMEDIATE supported API_CAPABILITY_ENCRYPTED_PATCH_ENVELOPE = 14; // Support for Patch Envelope Encryption API_CAPABILITY_SINGLE_STACK_IP_NETWORK = 15; // Support for V4Only and V6Only NetworkType API_CAPABILITY_CELLULAR_ATTACH_CONFIG = 16; // Support cellular attach configuration API_CAPABILITY_EDGEVIEW_AUTHENTICATION = 17; // EdgeView authentication, see https://github.com/lf-edge/eve/blob/master/docs/EDGEVIEW-CONTAINER-API.md - API_CAPABILITY_DISABLE_VTPM = 18; // Support for disabling + API_CAPABILITY_DISABLE_VTPM = 18; // VmConfig.disable_vtpm supported API_CAPABILITY_LOC_REBOOT_COLLECT_INFO = 19; // Support for rebooting and collect-info from LOC API_CAPABILITY_SMART_REPORT = 20; // Support for S.M.A.R.T. info on physical storage devices API_CAPABILITY_REPORT_TPM_EVENTLOG = 21; // Support for reporting TPM Event Log "as is" without parsing and selectively reporting events - // Add new values as new EdgeDevConfig API features are implemented + // Append new values as new API features are implemented; never insert or + // renumber, since a value asserts that all lower values are also supported. } // Different reasons for a boot/reboot @@ -1285,7 +1293,8 @@ message ZInfoMsg { google.protobuf.Timestamp atTimeStamp = 6; } -// Information about hardware capabilities of node +// Information about hardware capabilities of node. Distinct from APICapability +// (API support level) and OptionalCapabilities (per-flavor software support). message Capabilities { // VMX/SVM for amd64 or Arm virtualization extensions for arm64 bool HWAssistedVirtualization = 2;