Skip to content

shreds: flag lagging publishers on the shred publishers page #666

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Open
SinaVafadar wants to merge 4 commits into
base: main
Choose a base branch
from
Open

shreds: flag lagging publishers on the shred publishers page #666

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
4 commits
Select commit Hold shift + click to select a range
1f7877e
shreds: flag lagging publishers on the shred publishers page
SinaVafadar Jun 20, 2026
8b1c7e9
Merge branch 'main' into sv/publisher-lagging-status
SinaVafadar Jun 20, 2026
5e75951
web: rename lagging column to fast shreds and reorder
SinaVafadar Jun 23, 2026
592ee77
remotetables: update setup for remote tables
SinaVafadar Jun 23, 2026
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
4 changes: 4 additions & 0 deletions admin/remotetables/setup.go
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -36,6 +36,10 @@ var externalRemoteTables = []struct {
{"dzdp", "offsets"},
{"dzdp", "location_decisions"},
{"dzdp", "location_state"},
// lagged_validators is filled by an external process and only exists in prod.
// Proxy just this table so non-prod envs can read it; validator_rewards in the
// same database is ingested natively by the indexer and is left untouched.
{"dzf_data", "lagged_validators"},
}

// externalRemoteDatabases lists remote databases to mirror in full, discovering
Expand Down
19 changes: 18 additions & 1 deletion api/config/config.go
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -35,6 +35,9 @@ var publisherDB = "shredder"
// dzdpDB is the ClickHouse database name for DZDP tables e.g. location_state (default: "dzdp").
var dzdpDB = "dzdp"

// dzfDataDB is the ClickHouse database name for dzf_data tables (default: "dzf_data").
var dzfDataDB = "dzf_data"

// EnvDBs maps environment names to their ClickHouse connection pools.
// The mainnet-beta entry always points to DB.
var EnvDBs map[string]driver.Conn
Expand Down Expand Up @@ -83,6 +86,16 @@ func SetDZDPDB(db string) {
dzdpDB = db
}

// GetDZFDataDB returns the dzf_data database name.
func GetDZFDataDB() string {
return dzfDataDB
}

// SetDZFDataDB sets the dzf_data database name.
func SetDZFDataDB(db string) {

@ben-dz ben-dz Jul 23, 2026

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

SetDZFDataDB has no callers — the env-var override in Load() is the only write path. Drop the setter.

dzfDataDB = db
}

// Database returns the configured database name.
func Database() string {
return cfg.Database
Expand Down Expand Up @@ -154,6 +167,10 @@ func Load() error {
dzdpDB = db
}

if db := os.Getenv("CLICKHOUSE_DZF_DATA_DB"); db != "" {
dzfDataDB = db
}

// Build env -> database mapping.
// Devnet and testnet databases default to lake_devnet / lake_testnet
// unless overridden by env vars or disabled with CLICKHOUSE_NO_DEVNET / CLICKHOUSE_NO_TESTNET.
Expand All @@ -177,7 +194,7 @@ func Load() error {

secure := os.Getenv("CLICKHOUSE_SECURE") == "true"

slog.Info("connecting to ClickHouse", "addr", cfg.Addr, "database", cfg.Database, "username", cfg.Username, "secure", secure, "shredder_db", shredderDB, "publisher_db", publisherDB, "dzdp_db", dzdpDB)
slog.Info("connecting to ClickHouse", "addr", cfg.Addr, "database", cfg.Database, "username", cfg.Username, "secure", secure, "shredder_db", shredderDB, "publisher_db", publisherDB, "dzdp_db", dzdpDB, "dzf_data_db", dzfDataDB)

// Create connection pool
opts := &clickhouse.Options{
Expand Down
1 change: 1 addition & 0 deletions api/handlers/api.go
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -43,6 +43,7 @@ type API struct {
ShredderDB string
PublisherDB string
DZDPDB string
DZFDataDB string

// PostgreSQL
PgPool *pgxpool.Pool
Expand Down
47 changes: 47 additions & 0 deletions api/handlers/publisher_check.go
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -67,6 +67,8 @@ type PublisherCheckItem struct {
ValidatorName string `json:"validator_name"`
ValidatorVersionOk bool `json:"validator_version_ok"`
IsBackup bool `json:"is_backup"`
Lagging bool `json:"lagging"`
LaggingStatus string `json:"lagging_status"`
}

// PublisherCheckResponse is the response for the publisher check endpoint.
Expand Down Expand Up @@ -325,6 +327,20 @@ func (a *API) FetchPublisherCheckData(ctx context.Context, q string, epochsParam
publishers = []PublisherCheckItem{}
}

// Mark publishers whose validator is reported as lagging (status != Healthy)
// in dzf_data.lagged_validators. Non-fatal: if the lookup fails we still
// return publisher data without lagging annotations.
if lagging, err := a.fetchLaggedValidators(ctx); err != nil {
slog.Warn("publisher check: lagged validators query failed", "error", err)

@ben-dz ben-dz Jul 23, 2026

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

The page-cache worker refreshes publisher_check every 30s, so any environment where the table is absent (local dev without remote tables) or the grant is missing logs this warning every 30s indefinitely. Consider skipping the lookup when !isMainnet(ctx) and/or probing table availability once at startup.

} else {
for i := range publishers {
if status, ok := lagging[publishers[i].VotePubkey]; ok {
publishers[i].Lagging = true
publishers[i].LaggingStatus = status
}
}
}

var totalNetworkStake int64
err = a.envDB(ctx).QueryRow(ctx,
`SELECT COALESCE(SUM(activated_stake_lamports), 0)
Expand Down Expand Up @@ -358,3 +374,34 @@ func (a *API) FetchPublisherCheckData(ctx context.Context, q string, epochsParam
Publishers: publishers,
}, nil
}

// fetchLaggedValidators returns a map of validator vote pubkey to its lagging
// status for every validator currently reported as not Healthy in
// dzf_data.lagged_validators.
func (a *API) fetchLaggedValidators(ctx context.Context) (map[string]string, error) {

@ben-dz ben-dz Jul 23, 2026

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

The success path of this function is never verified anywhere. The test harness (api/testing/api.go) never sets DZFDataDB, so in every existing test this builds FROM ` `.lagged_validators with an empty identifier and always errors — the tests cited in the PR only exercise the warn-and-degrade branch. Fix: set DZFDataDB in the harness constructors (e.g. to the per-test dbName), seed a lagged_validators fixture following the createPublisherShredStatsTable pattern with a Healthy row, a non-Healthy row matched to a fixture vote pubkey, and an unmatched pubkey, and assert lagging/lagging_status on the response — this also pins the status-string contract. Separately, the read-only prod ClickHouse user has no SELECT grant on dzf_data.lagged_validators (verified), so please confirm the deployed API user's grant before merge; if it's missing, the column ships permanently green with only a server-side warn every 30s.

start := time.Now()
query := fmt.Sprintf(`
SELECT validator_vote_public_key, status
FROM `+"`%s`"+`.lagged_validators
WHERE status != 'Healthy' AND validator_vote_public_key != ''`, a.DZFDataDB)

@ben-dz ben-dz Jul 23, 2026

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This assumes the table holds only current state: no recency bound, no FINAL/argMax dedup, and arbitrary last-write-wins in the map when a pubkey has multiple rows. The table is written by an external process and its schema/engine exists nowhere in this repo. If it's append-mode (or a pre-merge ReplacingMergeTree), a recovered validator stays flagged; if the writer stops, flags freeze silently. Confirm the write contract (full replace per cycle?) and document it here, or dedup to latest state per pubkey with a recency bound.


rows, err := a.envDB(ctx).Query(ctx, query)
metrics.RecordClickHouseQuery("lagged_validators", time.Since(start), err)
if err != nil {
return nil, err
}
defer rows.Close()

lagging := make(map[string]string)
for rows.Next() {
var votePubkey, status string
if err := rows.Scan(&votePubkey, &status); err != nil {
return nil, fmt.Errorf("scan: %w", err)
}
lagging[votePubkey] = status
}
if err := rows.Err(); err != nil {
return nil, fmt.Errorf("rows: %w", err)
}
return lagging, nil
}
1 change: 1 addition & 0 deletions api/main.go
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -362,6 +362,7 @@ func main() {
ShredderDB: config.GetShredderDB(),
PublisherDB: config.GetPublisherDB(),
DZDPDB: config.GetDZDPDB(),
DZFDataDB: config.GetDZFDataDB(),
PgPool: config.PgPool,
Neo4jClient: config.Neo4jClient,
Neo4jDatabase: config.Neo4jDatabase,
Expand Down
3 changes: 3 additions & 0 deletions scripts/setup-remote-tables.sh
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -178,6 +178,9 @@ echo ""
EXTERNAL_TABLES=(
"shredder:publisher_shred_stats"
"shredder_qa:publisher_shred_stats"
# lagged_validators is filled by an external process and only exists in prod;
# proxy just this table (validator_rewards is ingested locally by the indexer).
"dzf_data:lagged_validators"
)

if [[ ${#EXTERNAL_TABLES[@]} -gt 0 ]]; then
Expand Down
17 changes: 15 additions & 2 deletions web/src/components/publisher-check-page.tsx
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -33,7 +33,8 @@ type SortField =
| "publishing_leader_shreds"
| "publishing_retransmitted"
| "leader_slots"
| "validator_client";
| "validator_client"
| "lagging";

type SortDirection = "asc" | "desc";

Expand Down Expand Up @@ -412,6 +413,9 @@ export function PublisherCheckPage() {
`${b.validator_client} ${b.validator_version}`,
);
break;
case "lagging":
cmp = Number(a.lagging) - Number(b.lagging);
break;
default:
cmp = 0;
}
Expand Down Expand Up @@ -867,6 +871,10 @@ export function PublisherCheckPage() {
No Retransmit Shreds
<SortIcon field="publishing_retransmitted" />
</th>
<th className={thCenter} onClick={() => handleSort("lagging")}>
Fast Shreds
<SortIcon field="lagging" />
</th>
<th
className={thCenter}
onClick={() => handleSort("leader_slots")}
Expand All @@ -887,7 +895,7 @@ export function PublisherCheckPage() {
{pagedPublishers.length === 0 ? (
<tr>
<td
colSpan={13}
colSpan={14}
className="px-4 py-12 text-center text-muted-foreground"
>
{activeFilter
Expand Down Expand Up @@ -974,6 +982,11 @@ export function PublisherCheckPage() {
<td className="px-4 py-3 text-center">
<StatusIcon ok={!pub.publishing_retransmitted} />
</td>
<td className="px-4 py-3 text-center">
<StatusIcon

@ben-dz ben-dz Jul 23, 2026

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

lagging_status is never shown: the PR description says the status text appears in the cell and on hover, but StatusIcon accepts only ok and the cell has no title, so the field is shipped to the client and only feeds the string comparison. The legend box above the table also explains the other two icon columns but not "Fast Shreds". Add title={pub.lagging_status || undefined} to the cell and a legend entry (and update the PR description) — or drop the field if it has no UI consumer after unifying the predicate.

ok={pub.lagging_status !== "Action Needed"}

@ben-dz ben-dz Jul 23, 2026

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This predicate disagrees with the backend and the sort. The backend sets lagging=true for any status != 'Healthy' (publisher_check.go:386) and the sort comparator uses that boolean (line 417), but the icon is red only for the exact string "Action Needed" — a value that appears nowhere in the backend or the repo, from a table whose status vocabulary is undocumented. Any other non-Healthy value (or a casing change by the external writer) yields a row that sorts as lagging while showing a green check; rows with status='' are also flagged lagging with a green icon, and an undefined lagging_status during deploy skew renders green too. Use one predicate: ok={!pub.lagging} (also handles undefined), or filter the SQL to status = 'Action Needed' so the flag, sort, and icon agree.

/>
</td>
<td className="px-4 py-3 text-sm tabular-nums text-center">
{pub.leader_slots.toLocaleString()}
</td>
Expand Down
2 changes: 2 additions & 0 deletions web/src/lib/api.ts
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -5908,6 +5908,8 @@ export interface PublisherCheckItem {
validator_name: string
validator_version_ok: boolean
is_backup: boolean
lagging: boolean
lagging_status: string
}

export interface PublisherCheckResponse {
Expand Down
Loading