confd: add a per-port egress rate limit

Nothing in the model could hold a port below its link speed, which a
virtual port has none of, so there was no way to put the traffic
classes under load there and see the scheduler work.  The design has
the node for it, one token bucket on the whole port.

Add /interfaces/interface/qos/egress/rate-limit with a rate in bits per
second and an optional burst, rendered as a 'tc tbf' root with the ets
scheduler beneath it:

    tc qdisc add dev e1 root handle 1: tbf rate 10000000bit burst 12500 latency 100ms
    tc qdisc add dev e1 parent 1:1 handle 2: ets bands 8 strict 8 priomap ...

An unset burst covers ten milliseconds at rate, never less than one
frame.  mqprio can only be the root, so a rate limited port schedules
with ets.  The operational offload list gains 'rate-limit', reported
when the driver marks the tbf offloaded.  NET_SCH_TBF is enabled on all
boards.

Signed-off-by: Joachim Wiberg <troglobit@gmail.com>

Author: troglobit

board/aarch64/linux_defconfig

@@ -246,6 +246,7 @@ CONFIG_VLAN_8021Q_MVRP=y
 CONFIG_NET_SCHED=y
 CONFIG_NET_SCH_MQPRIO=y
 CONFIG_NET_SCH_ETS=y
+CONFIG_NET_SCH_TBF=y
 CONFIG_NET_SCH_INGRESS=y
 CONFIG_NET_CLS_BASIC=y
 CONFIG_NET_CLS_BPF=y

board/arm/linux_defconfig

@@ -230,6 +230,7 @@ CONFIG_VLAN_8021Q_MVRP=y
 CONFIG_NET_SCHED=y
 CONFIG_NET_SCH_MQPRIO=y
 CONFIG_NET_SCH_ETS=y
+CONFIG_NET_SCH_TBF=y
 CONFIG_NET_SCH_INGRESS=y
 CONFIG_NET_CLS_BASIC=y
 CONFIG_NET_CLS_BPF=y

board/riscv64/linux_defconfig

@@ -194,6 +194,7 @@ CONFIG_VLAN_8021Q_MVRP=y
 CONFIG_NET_SCHED=y
 CONFIG_NET_SCH_MQPRIO=y
 CONFIG_NET_SCH_ETS=y
+CONFIG_NET_SCH_TBF=y
 CONFIG_NET_SCH_INGRESS=y
 CONFIG_NET_CLS_BASIC=y
 CONFIG_NET_CLS_BPF=y

board/x86_64/linux_defconfig

@@ -221,6 +221,7 @@ CONFIG_VLAN_8021Q_MVRP=y
 CONFIG_NET_SCHED=y
 CONFIG_NET_SCH_MQPRIO=y
 CONFIG_NET_SCH_ETS=y
+CONFIG_NET_SCH_TBF=y
 CONFIG_NET_SCH_INGRESS=y
 CONFIG_NET_CLS_BASIC=y
 CONFIG_NET_CLS_BPF=y

doc/ChangeLog.md

@@ -24,6 +24,9 @@ All notable changes to the project are documented in this file.
   of the box on every port: trust PCP, one traffic class per
   transmit queue mapped per IEEE 802.1Q-2022 Table 8-5, replacing the
   fixed boot-time queue setup, see [QoS](qos.md)
+- Add a per-port egress rate limit under
+  `/interfaces/interface/qos/egress/rate-limit`, one token bucket above
+  the traffic classes, see [QoS](qos.md)
 - Transmission selection is rendered with the `ets` scheduler on every
   port, so drivers that offload it take the class algorithms and weights
   into the switch fabric, not only the priority to class map

doc/qos.md

@@ -246,6 +246,30 @@ admin@example:/config/interface/e1/qos/egress/> <b>leave</b>
 </code></pre>
 
 
+### Rate Limit
+
+A port can be held below its link speed with one token bucket on the
+whole port, configured under `qos egress rate-limit`.  `rate` is the
+sustained rate in bits per second and `burst` how many bytes may leave
+at once from a full bucket; left unset, the burst covers ten
+milliseconds at the configured rate, never less than one maximum-size
+frame.  The bucket sits above the traffic classes: it bounds what the
+port sends in total, and transmission selection decides which class
+gets the bounded rate.
+
+<pre class="cli"><code>admin@example:/config/> <b>edit interface e1 qos egress</b>
+admin@example:/config/interface/e1/qos/egress/> <b>set rate-limit rate 10000000</b>
+admin@example:/config/interface/e1/qos/egress/> <b>leave</b>
+</code></pre>
+
+The limit is rendered as a `tc tbf` root qdisc with the scheduler below
+it, so on a port without offload it applies to the frames the CPU
+sends, and on a port whose driver takes the bucket to frames the fabric
+forwards as well; `offload` under `capabilities` then lists
+`rate-limit`.  A rate limited port always schedules with `tc ets`,
+since `tc mqprio` can only be the root qdisc.
+
+
 ### Egress Remarking
 
 Configured under `qos egress remark`, both leaves default to `none`:
@@ -352,8 +376,8 @@ and each port reports it under `qos capabilities`:
 
 `supported-trust-order` is absent when the driver has no ingress
 classification support.  `offload` lists `classification`, `remarking`
-and `transmission-selection` as the driver takes them; a stage not
-listed runs in the kernel.
+`transmission-selection` and `rate-limit` as the driver takes them; a
+stage not listed runs in the kernel.
 
 Each feature maps to one Linux mechanism, and whether it reaches the
 hardware depends on the driver implementing the matching hook:
@@ -364,6 +388,7 @@ hardware depends on the driver implementing the matching hook:
 | Trust order            | `dcb apptrust`  | `dcbnl_setapptrust` | Rule order, software |
 | Egress remarking       | `dcb rewr`      | `dcbnl_setrewr`     | DSCP only, software  |
 | Transmission selection | `tc ets`        | `TC_SETUP_QDISC_ETS`| software             |
+| Port rate limit        | `tc tbf`        | `TC_SETUP_QDISC_TBF`| software             |
 /// table-caption
 QoS features and their Linux backends.
 ///

src/confd/src/if-qos.c

@@ -1,5 +1,6 @@
 /* SPDX-License-Identifier: BSD-3-Clause */
 
+#include <inttypes.h>
 #include <stdbool.h>
 #include <dirent.h>
 
@@ -64,6 +65,8 @@ struct qos_egress {
 	uint8_t  map[NUM_PRIO];
 	enum tsa algo[MAX_TC];
 	uint32_t weight[MAX_TC];
+	uint64_t rate;		/* bits/s, 0 = no rate limit */
+	uint32_t burst;		/* bytes */
 };
 
 /*
@@ -328,6 +331,18 @@ static int qos_parse_egress(struct lyd_node *egress, const char *ifname, struct
 			eg->weight[id] = strtoul(val, NULL, 10);
 	}
 
+	table = lydx_get_child(egress, "rate-limit");
+	if (table) {
+		val = lydx_get_cattr(table, "rate");
+		eg->rate = val ? strtoull(val, NULL, 10) : 0;
+
+		/* Ten milliseconds at rate, never below one frame */
+		val = lydx_get_cattr(table, "burst");
+		eg->burst = val ? strtoul(val, NULL, 10) : eg->rate / 8 / 100;
+		if (eg->burst < 1518)
+			eg->burst = 1518;
+	}
+
 	return 0;
 }
 
@@ -691,13 +706,26 @@ static void gen_dcb_log(FILE *fp, const char *ifname, struct lyd_node *ingress,
  */
 static void gen_egress(FILE *fp, const char *ifname, struct qos_egress *eg)
 {
+	const char *attach = "root";
 	int i, nstrict = 0;
 
 	fprintf(fp, "tc qdisc del dev %s root 2>/dev/null\n", ifname);
+
+	/*
+	 * The rate limit is one bucket on the whole port, so it takes
+	 * the root and the scheduler hangs below it.  mqprio can only
+	 * be the root, so a rate limited port always schedules with ets.
+	 */
+	if (eg->rate) {
+		fprintf(fp, "tc qdisc add dev %s root handle 1: tbf rate %" PRIu64 "bit burst %u"
+			" latency 100ms\n", ifname, eg->rate, eg->burst);
+		attach = "parent 1:1 handle 2:";
+	}
+
 	if (eg->num_tc < 2)
 		return;
 
-	if (qos_mqprio(ifname) && !iface_has_quirk(ifname, "broken-mqprio")) {
+	if (!eg->rate && qos_mqprio(ifname) && !iface_has_quirk(ifname, "broken-mqprio")) {
 		fprintf(fp, "tc qdisc add dev %s root mqprio num_tc %d map", ifname, eg->num_tc);
 		for (i = 0; i < NUM_PRIO; i++)
 			fprintf(fp, " %d", eg->map[i]);
@@ -710,7 +738,7 @@ static void gen_egress(FILE *fp, const char *ifname, struct qos_egress *eg)
 	for (i = eg->num_tc - 1; i >= 0 && eg->algo[i] == TSA_STRICT; i--)
 		nstrict++;
 
-	fprintf(fp, "tc qdisc add dev %s root ets bands %d strict %d", ifname, eg->num_tc, nstrict);
+	fprintf(fp, "tc qdisc add dev %s %s ets bands %d strict %d", ifname, attach, eg->num_tc, nstrict);
 	if (nstrict < eg->num_tc) {
 		fputs(" quanta", fp);
 		for (i = eg->num_tc - 1 - nstrict; i >= 0; i--)
@@ -831,7 +859,8 @@ int netdag_gen_qos(sr_session_ctx_t *session, struct dagger *net, struct lyd_nod
 	 */
 	if (lydx_get_op(dif) == LYDX_OP_CREATE ||
 	    qos_has_change(lydx_get_descendant(lyd_child(dqos), "egress", "traffic-class-table", NULL)) ||
-	    qos_has_change(lydx_get_descendant(lyd_child(dqos), "egress", "traffic-class", NULL)))
+	    qos_has_change(lydx_get_descendant(lyd_child(dqos), "egress", "traffic-class", NULL)) ||
+	    qos_has_change(lydx_get_descendant(lyd_child(dqos), "egress", "rate-limit", NULL)))
 		gen_egress(fp, ifname, &eg);
 	fclose(fp);
 

src/confd/yang/confd/infix-if-qos.yang

@@ -179,6 +179,9 @@ submodule infix-if-qos {
               description "Traffic classes and transmission selection
                  run in the switch fabric.";
             }
+            enum rate-limit {
+              description "The port rate limit runs in the switch fabric.";
+            }
           }
           description "Pipeline stages the driver has taken into
              hardware.  A stage not listed runs in the kernel, which on
@@ -479,6 +482,34 @@ submodule infix-if-qos {
             description "DSCP rewrite on transmit.";
           }
         }
+
+        /* Rendered as a tc tbf root above the scheduler. */
+        container rate-limit {
+          presence "Bounds the transmit rate of the port.";
+          description "One token bucket on the whole port, above the
+             traffic classes.  It bounds what the port sends in total
+             and knows nothing about classes: transmission selection
+             decides which class gets the bounded rate.";
+
+          leaf rate {
+            type uint64 {
+              range "8..max";
+            }
+            units "bits/second";
+            mandatory true;
+            description "Sustained transmit rate.";
+          }
+
+          leaf burst {
+            type uint32 {
+              range "1518..max";
+            }
+            units "bytes";
+            description "Bytes that may leave at once when the bucket is
+               full.  Unset, the daemon sizes it for ten milliseconds
+               at rate, never below one maximum-size frame.";
+          }
+        }
       }
     }
   }

src/statd/python/yanger/ietf_interfaces/link.py

@@ -170,8 +170,8 @@ def qos_capabilities(iplink, qdiscs):
         result["supported-trust-order"] = trust
 
     # Stages the driver runs.  DCB tables exist only on drivers with the
-    # operations, so only those ports are asked.  ets reports offloaded
-    # when the driver took it.  mqprio never sets that flag,
+    # operations, so only those ports are asked.  ets and tbf report
+    # offloaded when the driver took them.  mqprio never sets that flag,
     # but it is only ever installed with hw 1, which the kernel refuses
     # without driver support, so its presence means the same.
     offload = []
@@ -187,6 +187,8 @@ def qos_capabilities(iplink, qdiscs):
         kind, offloaded = qdisc.get("kind"), qdisc.get("offloaded", False)
         if kind == "mqprio" and qdisc.get("root") or kind == "ets" and offloaded:
             offload.append("transmission-selection")
+        if kind == "tbf" and qdisc.get("root") and offloaded:
+            offload.append("rate-limit")
 
     if offload:
         result["offload"] = offload

test/case/interfaces/qos_traffic_class/test.adoc

@@ -13,9 +13,11 @@ strict bands first, the weights as quanta, and the priomap inverted as
 
 The port's class count comes from its transmit queues, or eight for a
 single-queue port.  Switching the table to the ieee-sr preset must put
-the SR classes, priorities 2 and 3, on the two highest classes.  Removing
-the qos container must restore the default table, IEEE 802.1Q-2022
-Table 8-5, and the operational datastore must report whether transmission
+the SR classes, priorities 2 and 3, on the two highest classes.  A rate
+limit must put a tbf root qdisc above the scheduler, with the same ets
+table beneath it, and go away with the setting.  Removing the qos
+container must restore the default table, IEEE 802.1Q-2022 Table 8-5,
+and the operational datastore must report whether transmission
 selection is offloaded throughout.
 
 ==== Topology
@@ -29,6 +31,8 @@ image::topology.svg[QoS Traffic Classes and Transmission Selection topology, ali
 . Verify the root qdisc carries the custom map and weights
 . Verify the offload capability matches the qdisc
 . Switch to the ieee-sr preset and verify SR classes on top
+. Add a 10 Mbit/s rate limit and verify the tbf root above the ets table
+. Remove the rate limit and verify the scheduler is the root again
 . Remove qos configuration and verify the default table is back