Skip to content

Telegram sink: limited interactivity, human-readable titles, and crash finding rate-limits #2137

Description

@RaviTharuma

Summary

Classic telegram_sink works well for getting Robusta findings into Telegram, but several UX gaps make it hard for non-SRE / on-call-adjacent humans to act on messages without Slack. Docs already note that 2-way interactivity is missing; this issue asks for a small, intentional set of improvements focused on Telegram as a primary chat destination.

Related (not duplicates):

HolmesGPT is great for AI triage; this request is specifically about deterministic classic Telegram notifications remaining usable when Telegram (not Slack) is the team’s day-to-day channel.


1) Limited CallbackBlock / inline buttons on Telegram

Today: Telegram sink does not implement CallbackBlock / inline keyboards. Links (e.g. Investigate / UI) may appear as text, but there is no first-class button UX comparable to Slack.

Ask: Support a minimal interactivity surface for Telegram, without requiring Slack:

Action Why
Open UI / finding link One-tap jump to Robusta UI or investigation URL
Silence / snooze (optional) Temporary mute of this alert fingerprint / finding, if Alertmanager or Robusta silence APIs are available

Out of scope (for this request): full general-purpose playbook callback UI, arbitrary multi-step bots, or rebuilding Slack’s full interactive surface.

Telegram Bot API inline keyboards are a natural fit; even 1–2 fixed buttons would close most of the “must use Slack for buttons” gap for small teams.

Example (generic config intent):

sinksConfig:
  - telegram_sink:
      name: team_telegram
      bot_token: "<BOT_TOKEN>"
      chat_id: "<CHAT_ID>"
      # hypothetical future knobs — names illustrative only
      # inline_actions: [open_ui, silence]

2) Optional human-readable title / summary mode for crash-style findings

Today: Default crashloop-style notifications often look roughly like:

  • Title: Crashing pod <pod-name>
  • Body: raw/enricher dumps (logs, tables as files, stack-ish text)

That is fine for SREs living in cluster context; it is hard for product engineers or ops-adjacent humans who only see Telegram.

Ask: Optional human-readable title templates and/or a compact what / where / layer summary mode for telegram_sink (or for findings that render there), so the first line of the Telegram message answers:

  1. What broke (symptom in plain language)
  2. Where (namespace / workload / container — not only pod name)
  3. Layer (optional tag: runtime / capacity / networking / app — if known from playbook or labels)
  4. Do this (optional one-liner from annotation or playbook description)

Example shape (illustrative, not a fixed format):

Needs attention · CrashLoop · demo/checkout-api
What: container restarting (exit 1)
Where: ns=demo workload=checkout-api container=app
Layer: runtime
Do this: check recent deploy / logs for startup errors

vs default:

Crashing pod checkout-api-7d9f8c6b5-xxxxx
[large log / file dump]

Possible levers (any of these would help):

  • Sink option: message_style: compact | full (or human_summary: true)
  • Title template on sink or playbook: e.g. title_template: "CrashLoop · {{ namespace }}/{{ owner_name }}"
  • Prefer Deployment/owner name over ephemeral pod name in titles when available
  • Keep current full dump behind send_files: true / expand path so power users lose nothing

This is complementary to #260 (container detail) but oriented at Telegram-first, human-first presentation, not only richer SRE detail.


3) Rate-limit / dedupe identical crash findings per namespace (values-configurable)

Today: A noisy CrashLoopBackOff in one namespace can flood Telegram with near-identical findings (same workload, new pod hash each time). Trigger-level rate limits (#1173) and sink scope/activity help routing, but a clear values-level knob to throttle identical crash findings per namespace (or per fingerprint key) is either missing or easy to miss in docs.

Ask: Document and/or add sink- or playbook-level controls such as:

# illustrative only
sinksConfig:
  - telegram_sink:
      name: team_telegram
      bot_token: "<BOT_TOKEN>"
      chat_id: "<CHAT_ID>"
      # e.g. collapse repeats for same finding "kind" in a namespace
      # finding_rate_limit:
      #   window_seconds: 900
      #   key: [namespace, name_prefix]   # or finding aggregation key
      #   max_per_window: 1

Or an official documented pattern using existing matchers + aggregation if this is already possible.

Desired behavior:

  • First crash finding for demo/checkout-api → notify immediately
  • Further identical crashes in the same namespace/workload within N minutes → suppress or send a single “still crashing (count=K)” update
  • Different namespaces / different workloads still notify independently

If this already exists (e.g. via finding aggregation, dedup, or playbook params), a short doc example under the Telegram sink page would satisfy this item.


Why classic Telegram (not “just use HolmesGPT / Slack”)

  • Many homelab and small-team setups use Telegram as the only chat sink
  • HolmesGPT triage is valuable, but classic sinks remain documented and widely deployed
  • Requiring Slack solely for silence/UI buttons is a high bar for Telegram-native teams
  • Improving presentation + light interactivity keeps classic notifications actionable without building a custom bot

Acceptance criteria (suggested)

  • Docs updated: current Telegram interactivity limits + any new options
  • At least one of: Telegram inline button for open-UI link; optional silence/snooze
  • Optional compact / human title or summary mode for crash-style findings on Telegram
  • Values-configurable rate-limit/dedupe for repeated crash findings per namespace (or documented existing equivalent)
  • No breaking change to default message format for existing users

Environment (generic)

  • Robusta classic with telegram_sink
  • Default crashloop / pod-issue playbooks enabled
  • Telegram as primary notification channel (no Slack)

Non-goals

  • Full parity with Slack Block Kit
  • Custom user bots / third-party bridges
  • Changing HolmesGPT product direction

Thanks for considering — happy to refine scope if maintainers prefer splitting into separate issues.

Metadata

Metadata

Assignees

No one assigned

    Labels

    No labels
    No labels

    Type

    No type

    Projects

    No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions