Repository navigation
Commit 96dbbd5
Implements the
- Declares `kKvSaveBlocksTotal` (`MetricType::kCounter`) along with `outcome` label values `kOutcomeRejected` (`"rejected"`) and `kOutcomeFailed` (`"failed"`) in `metrics_backend.h`.
- Instruments `KVCacheStore::Insert` (`tier="local_dram"`, `outcome="rejected"` for the whole batch when `InsertOption::kInsertAndPin` fails, either for lack of unpinned space or because an existing hash could not be pinned), `KVCacheStore::SaveLocal` (`tier="local_dram"`; host block allocation failure counts `outcome="rejected"` on `RESOURCE_EXHAUSTED` and `"failed"` on any other error; storage write-through counts `"ok"` / `"failed"` when the offload settles), `HostOffloadBackend::Save` (`tier="local_dram"`, `outcome="ok"` / `"failed"`), and `HostOffloadBackend::BeginWriteRemote` (`tier="remote_dram"`, `outcome="ok"`, `"rejected"`, and `"failed"` across peer RPC and transfer settle paths).
- Adds unit tests verifying `kv_save_blocks_total` across `local_dram` (including storage write-through) and `remote_dram` tiers.
PiperOrigin-RevId: 999559866kv_save_blocks_total counter metric (tpu_raiden_kv_save_blocks_total{dp_rank, tier, outcome}) to track block save outcomes across KV cache offload tiers:1 parent b001246 commit 96dbbd5
8 files changed
Lines changed: 407 additions & 15 deletions
File tree
- tpu_sync
- kv_cache
- telemetry
- python
| Original file line number | Diff line number | Diff line change | |
|---|---|---|---|
| |||
998 | 998 | | |
999 | 999 | | |
1000 | 1000 | | |
| 1001 | + | |
1001 | 1002 | | |
1002 | 1003 | | |
1003 | 1004 | | |
| |||
| Original file line number | Diff line number | Diff line change | |
|---|---|---|---|
| |||
112 | 112 | | |
113 | 113 | | |
114 | 114 | | |
| 115 | + | |
| 116 | + | |
| 117 | + | |
| 118 | + | |
| 119 | + | |
| 120 | + | |
| 121 | + | |
| 122 | + | |
| 123 | + | |
| 124 | + | |
| 125 | + | |
| 126 | + | |
| 127 | + | |
| 128 | + | |
| 129 | + | |
| 130 | + | |
| 131 | + | |
| 132 | + | |
| 133 | + | |
115 | 134 | | |
116 | 135 | | |
117 | 136 | | |
| |||
793 | 812 | | |
794 | 813 | | |
795 | 814 | | |
| 815 | + | |
| 816 | + | |
| 817 | + | |
| 818 | + | |
| 819 | + | |
| 820 | + | |
| 821 | + | |
| 822 | + | |
| 823 | + | |
| 824 | + | |
| 825 | + | |
796 | 826 | | |
797 | 827 | | |
798 | 828 | | |
799 | | - | |
800 | | - | |
| 829 | + | |
| 830 | + | |
| 831 | + | |
| 832 | + | |
| 833 | + | |
801 | 834 | | |
802 | 835 | | |
803 | 836 | | |
| |||
806 | 839 | | |
807 | 840 | | |
808 | 841 | | |
809 | | - | |
| 842 | + | |
810 | 843 | | |
811 | 844 | | |
812 | 845 | | |
| |||
820 | 853 | | |
821 | 854 | | |
822 | 855 | | |
823 | | - | |
| 856 | + | |
| 857 | + | |
| 858 | + | |
| 859 | + | |
824 | 860 | | |
825 | 861 | | |
826 | 862 | | |
| |||
837 | 873 | | |
838 | 874 | | |
839 | 875 | | |
840 | | - | |
| 876 | + | |
| 877 | + | |
841 | 878 | | |
842 | 879 | | |
843 | 880 | | |
844 | 881 | | |
845 | 882 | | |
846 | 883 | | |
847 | 884 | | |
| 885 | + | |
| 886 | + | |
| 887 | + | |
848 | 888 | | |
849 | 889 | | |
850 | 890 | | |
| |||
853 | 893 | | |
854 | 894 | | |
855 | 895 | | |
856 | | - | |
857 | | - | |
| 896 | + | |
| 897 | + | |
| 898 | + | |
| 899 | + | |
858 | 900 | | |
859 | 901 | | |
860 | 902 | | |
| |||
887 | 929 | | |
888 | 930 | | |
889 | 931 | | |
| 932 | + | |
| 933 | + | |
| 934 | + | |
| 935 | + | |
| 936 | + | |
890 | 937 | | |
891 | 938 | | |
892 | 939 | | |
| |||
915 | 962 | | |
916 | 963 | | |
917 | 964 | | |
918 | | - | |
| 965 | + | |
| 966 | + | |
919 | 967 | | |
920 | 968 | | |
921 | 969 | | |
| |||
938 | 986 | | |
939 | 987 | | |
940 | 988 | | |
| 989 | + | |
| 990 | + | |
| 991 | + | |
| 992 | + | |
| 993 | + | |
941 | 994 | | |
942 | 995 | | |
943 | 996 | | |
| |||
1403 | 1456 | | |
1404 | 1457 | | |
1405 | 1458 | | |
1406 | | - | |
| 1459 | + | |
| 1460 | + | |
1407 | 1461 | | |
1408 | 1462 | | |
1409 | 1463 | | |
1410 | 1464 | | |
1411 | 1465 | | |
1412 | | - | |
| 1466 | + | |
| 1467 | + | |
1413 | 1468 | | |
1414 | 1469 | | |
1415 | 1470 | | |
| |||
| Original file line number | Diff line number | Diff line change | |
|---|---|---|---|
| |||
51 | 51 | | |
52 | 52 | | |
53 | 53 | | |
| 54 | + | |
54 | 55 | | |
55 | 56 | | |
56 | 57 | | |
| |||
213 | 214 | | |
214 | 215 | | |
215 | 216 | | |
| 217 | + | |
| 218 | + | |
| 219 | + | |
| 220 | + | |
| 221 | + | |
| 222 | + | |
| 223 | + | |
| 224 | + | |
| 225 | + | |
| 226 | + | |
| 227 | + | |
| 228 | + | |
| 229 | + | |
| 230 | + | |
| 231 | + | |
| 232 | + | |
| 233 | + | |
| 234 | + | |
| 235 | + | |
216 | 236 | | |
217 | 237 | | |
218 | 238 | | |
| |||
1068 | 1088 | | |
1069 | 1089 | | |
1070 | 1090 | | |
| 1091 | + | |
| 1092 | + | |
| 1093 | + | |
1071 | 1094 | | |
1072 | 1095 | | |
1073 | 1096 | | |
| |||
1151 | 1174 | | |
1152 | 1175 | | |
1153 | 1176 | | |
1154 | | - | |
| 1177 | + | |
| 1178 | + | |
1155 | 1179 | | |
| 1180 | + | |
| 1181 | + | |
| 1182 | + | |
| 1183 | + | |
| 1184 | + | |
| 1185 | + | |
1156 | 1186 | | |
1157 | 1187 | | |
1158 | 1188 | | |
| |||
1193 | 1223 | | |
1194 | 1224 | | |
1195 | 1225 | | |
1196 | | - | |
| 1226 | + | |
| 1227 | + | |
| 1228 | + | |
1197 | 1229 | | |
1198 | 1230 | | |
1199 | 1231 | | |
| |||
1216 | 1248 | | |
1217 | 1249 | | |
1218 | 1250 | | |
1219 | | - | |
| 1251 | + | |
| 1252 | + | |
1220 | 1253 | | |
1221 | 1254 | | |
1222 | 1255 | | |
| |||
0 commit comments