Skip to content

feat: enable StuckDetector for running task auto-recovery#1067

Open
sh1nj1 wants to merge 2 commits into
mainfrom
feat/enable-stuck-detector
Open

feat: enable StuckDetector for running task auto-recovery#1067
sh1nj1 wants to merge 2 commits into
mainfrom
feat/enable-stuck-detector

Conversation

@sh1nj1

@sh1nj1 sh1nj1 commented Mar 27, 2026

Copy link
Copy Markdown
Owner

Problem

StuckDetector was implemented but disabled by default (enabled: false). This meant stuck tasks in running state were never auto-recovered, requiring manual intervention via Kamal console.

Additionally, pending_approval tasks had no timeout — they could remain stuck indefinitely.

Solution

1. Enable StuckDetector by default

# policy_resolver.rb
"stuck_detection" => {
  "enabled" => true,  # was: false
  ...
}

The StuckDetectorJob already runs every 10 minutes in production (configured in recurring.yml). It just wasn't doing anything because the policy was disabled.

2. Add pending_approval timeout detection

New config: approval_timeout_minutes (default: 60)

  • Detects pending_approval tasks older than threshold
  • Auto-recovers: marks as cancelled (vs failed for running tasks)
  • Releases ResourceTracker reservation
  • Dequeues next task for the topic
  • Sends inbox notification with specific i18n message

Changes

  • policy_resolver.rb: Enable stuck_detection, add approval_timeout_minutes
  • stuck_detector.rb: Detect + auto-recover pending_approval tasks
  • ai_agent.en.yml / ai_agent.ko.yml: Add approval_timeout i18n
  • stuck_detector_test.rb: 3 new tests (detect, auto-recover, skip recent)

Tests

  • ✅ 14 stuck_detector tests, 0 failures
  • ✅ i18n completeness check passed
  • ✅ Rubocop clean

@chatgpt-codex-connector

Copy link
Copy Markdown

You have reached your Codex usage limits for code reviews. You can see your limits in the Codex usage dashboard.

sh1nj1 added 2 commits April 15, 2026 15:58
Enable StuckDetector by default (was disabled) to auto-recover stuck
tasks and prevent resource accumulation.

Changes:
- Enable stuck_detection in PolicyResolver defaults (enabled: true)
- Add approval_timeout_minutes config (default: 60 min)
- Detect pending_approval tasks exceeding timeout threshold
- Auto-recover: running→failed, pending_approval→cancelled
- Release ResourceTracker and dequeue topic on recovery
- Add i18n for approval_timeout notifications (en, ko)
- Add 3 tests for pending_approval detection and recovery
pending_approval was not the root cause of the resource tracker leak.
Keep StuckDetector enabled for running tasks only.
@sh1nj1
sh1nj1 force-pushed the feat/enable-stuck-detector branch from 2e18954 to 86e4513 Compare April 15, 2026 07:01
@sh1nj1 sh1nj1 changed the title feat: enable StuckDetector and add pending_approval timeout feat: enable StuckDetector for running task auto-recovery Apr 15, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant