Skip to content
Merged
Show file tree
Hide file tree
Changes from 5 commits
Commits
Show all changes
15 commits
Select commit Hold shift + click to select a range
fe7b357
feat(desktop): deliver background-agent completions into live voice s…
skanderkaroui Jul 22, 2026
6202601
test(desktop): gate DEBUG-seam AgentSync recovery tests for release c…
skanderkaroui Jul 22, 2026
7bb8d24
test(desktop): gate DEBUG-seam VoiceTurnCoordinator tests for release…
skanderkaroui Jul 22, 2026
4f91bd3
fix(desktop): only ack a voice completion after it reaches the provider
skanderkaroui Jul 22, 2026
d63fb3d
fix(desktop): keep voice completion service inert until app start
skanderkaroui Jul 22, 2026
4471936
fix(desktop): confirm background-context delivery before advancing th…
skanderkaroui Jul 23, 2026
8a0d87d
fix(desktop): retry a refused voice completion when the input window …
skanderkaroui Jul 23, 2026
d110a34
merge: origin/main into voice-completion-delivery
skanderkaroui Jul 23, 2026
3c7fad5
fix(desktop): keep input-window completion retry inert until service …
skanderkaroui Jul 23, 2026
20cfa0f
merge: origin/main into voice-completion-delivery
skanderkaroui Jul 24, 2026
7348c68
fix(desktop): make SingleFlight test Gate an actor for strict concurr…
skanderkaroui Jul 24, 2026
7a91e5d
merge: origin/main into voice-completion-delivery
skanderkaroui Jul 24, 2026
6c2436b
Merge remote-tracking branch 'origin/main' into claude/voice-completi…
skanderkaroui Jul 25, 2026
baa9c08
style(desktop): drop stray blank line from merge resolution
skanderkaroui Jul 25, 2026
60fa15a
test(desktop): gate DEBUG-seam handoff policy tests for release compiles
skanderkaroui Jul 25, 2026
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
Original file line number Diff line number Diff line change
Expand Up @@ -4,14 +4,14 @@
"desktop/macos/Desktop/Sources/FloatingControlBar/FloatingControlBarView.swift": 2853,
"desktop/macos/Desktop/Sources/FloatingControlBar/FloatingControlBarWindow.swift": 4842,
"desktop/macos/Desktop/Sources/FloatingControlBar/PushToTalkManager.swift": 2423,
"desktop/macos/Desktop/Sources/FloatingControlBar/RealtimeHubSession.swift": 1621
"desktop/macos/Desktop/Sources/FloatingControlBar/RealtimeHubSession.swift": 1647
},
"raise_justifications": {
"desktop/macos/Desktop/Sources/FloatingControlBar/AgentPill.swift": "Agent completion presentation now follows the canonical terminal lifecycle used by PTT and chat.",
"desktop/macos/Desktop/Sources/FloatingControlBar/FloatingControlBarView.swift": "Onboarding bar glow, the reach-error card (Retry/Skip), and the notch hover/voice surface-state boundary render on the shared bar surface.",
"desktop/macos/Desktop/Sources/FloatingControlBar/FloatingControlBarWindow.swift": "Reach-error state lives with the owning manager; savePreChatCenterIfNeeded snaps to the full stored restore frame so the pill no longer drifts per re-open cycle.",
"desktop/macos/Desktop/Sources/FloatingControlBar/PushToTalkManager.swift": "Strict concurrency annotations (Ticket 14) add Sendable conformances and isolation qualifiers without changing runtime behavior.",
"desktop/macos/Desktop/Sources/FloatingControlBar/RealtimeHubSession.swift": "Strict concurrency annotations (Ticket 14) add Sendable conformances and isolation qualifiers without changing runtime behavior."
"desktop/macos/Desktop/Sources/FloatingControlBar/RealtimeHubSession.swift": "sendBackgroundAgentContext is a non-buffering delivery variant (returns false instead of buffering) so voice completion acks only after confirmed provider send; it lives beside the sendTextInput path it deliberately diverges from."
},
"threshold": 1500
}
Original file line number Diff line number Diff line change
Expand Up @@ -4,13 +4,13 @@
"desktop/macos/Desktop/Sources/CloudConnectorFormAutomation.swift": 1678,
"desktop/macos/Desktop/Sources/DesktopAutomationBridge.swift": 4238,
"desktop/macos/Desktop/Sources/MemoryExportService.swift": 1578,
"desktop/macos/Desktop/Sources/OmiApp.swift": 1646
"desktop/macos/Desktop/Sources/OmiApp.swift": 1649
},
"raise_justifications": {
"desktop/macos/Desktop/Sources/AuthService.swift": "Apple first-auth name capture signals observers (authNameDidUpdate) and persists the name to Firebase so it survives reinstalls.",
"desktop/macos/Desktop/Sources/DesktopAutomationBridge.swift": "Reach-error card gains a debug bridge action so the actionable failure surface is verifiable in-process.",
"desktop/macos/Desktop/Sources/MemoryExportService.swift": "Pinned swift-format normalizes line layout without changing this source's runtime behavior.",
"desktop/macos/Desktop/Sources/OmiApp.swift": "openMainAppChat() gives the floating bar Continue-in-Omi affordances a chat-landing entry next to openMainAppWindow()."
"desktop/macos/Desktop/Sources/OmiApp.swift": "openMainAppChat() adds a chat-landing entry beside openMainAppWindow(), and one startup call wires AgentCompletionVoiceDelivery into launch beside the other singleton starts; the service is a separate file."
},
"threshold": 1500
}
158 changes: 158 additions & 0 deletions desktop/macos/Desktop/Sources/Chat/AgentCompletionVoiceDelivery.swift
Original file line number Diff line number Diff line change
@@ -0,0 +1,158 @@
import Combine
import Foundation

/// Bridges completed background-agent runs into the live realtime voice
/// conversation.
///
/// The kernel journals a completed run and the floating pill shows it, but a
/// live voice session has no eyes: nothing tells the realtime model that the
/// agent it spawned has finished. This service watches run projections for
/// transitions into a terminal state on background surfaces, reads the
/// canonical completion delta from the kernel (`peekCompletedAgentDelta`,
/// exactly-once via the per-surface checkpoint), and injects the delta prompt
/// into the live provider session as untrusted context. No synthetic response
/// is requested — the voice turn coordinator's output authority is untouched;
/// the model mentions the completion at the next natural turn boundary and can
/// answer follow-ups. The checkpoint advances only after the session confirms
/// the send, so an undelivered completion is retried instead of lost.
@MainActor
final class AgentCompletionVoiceDelivery {
static let shared = AgentCompletionVoiceDelivery()

/// Background surfaces whose terminal transitions can carry a user-facing
/// completion. Primary conversational surfaces (main_chat, realtime_voice,
/// task_chat, …) reach a terminal state on every ordinary answer and must
/// not trigger kernel delta reads.
static let triggerSurfaceKinds: Set<String> = ["floating_bar", "service", "workstream"]

struct Delta {
let ids: [String]
let prompt: String
let completedAtHighWaterMs: Int?
}

private let isVoiceSessionLive: @MainActor () -> Bool
private let peekDelta: @MainActor () async -> Delta?
private let injectContext: @MainActor (String) async -> Bool
private let acknowledge: @MainActor (Delta) -> Void
private let scheduleWork: @MainActor (@escaping @MainActor () async -> Void) -> Void

private var cancellable: AnyCancellable?
private var lastStatusBySurface: [String: AgentRunProjectionStatus] = [:]
private var deliveryInFlight = false
private var deliveryQueued = false
/// The shared instance stays completely inert — no status subscription, no
/// delivery, no reach into the agent runtime — until `start()` runs at app
/// launch. This keeps `voiceSessionDidConnect()`, which the realtime hub calls
/// from `hubDidConnect` (exercised by RealtimeHub unit tests), a no-op in any
/// context that never started the service.
private var hasStarted: Bool

init(
isVoiceSessionLive: (@MainActor () -> Bool)? = nil,
peekDelta: (@MainActor () async -> Delta?)? = nil,
injectContext: (@MainActor (String) async -> Bool)? = nil,
acknowledge: (@MainActor (Delta) -> Void)? = nil,
scheduleWork: (@MainActor (@escaping @MainActor () async -> Void) -> Void)? = nil,
hasStarted: Bool = false
) {
self.hasStarted = hasStarted
self.isVoiceSessionLive =
isVoiceSessionLive ?? { RealtimeHubController.shared.hasLiveVoiceSession }
self.peekDelta =
peekDelta
?? {
guard
let delta = await DesktopCoordinatorService.shared.peekCompletedAgentDelta(
surface: .realtimeVoice())
else { return nil }
return Delta(
ids: delta.ids,
prompt: delta.prompt,
completedAtHighWaterMs: delta.completedAtHighWaterMs)
}
self.injectContext =
injectContext
?? { text in
await RealtimeHubController.shared.injectBackgroundAgentCompletionContext(text)
}
self.acknowledge =
acknowledge
?? { delta in
DesktopCoordinatorService.shared.acknowledgeCompletedAgentDelta(
surface: .realtimeVoice(),
ids: delta.ids,
completedAtHighWaterMs: delta.completedAtHighWaterMs)
}
self.scheduleWork = scheduleWork ?? { work in Task { await work() } }
}

/// Idempotent; called once from app launch.
func start() {
guard cancellable == nil else { return }
hasStarted = true
cancellable = AgentRuntimeStatusStore.shared.$projectionsBySurface
.sink { [weak self] projections in
self?.observe(projections)
}
}

/// A newly connected voice session drains completions that finished while no
/// session was live (their checkpoint was deliberately left unadvanced). Inert
/// until `start()` so the realtime hub's `hubDidConnect` hook never drives the
/// unstarted shared instance into the agent runtime during tests.
func voiceSessionDidConnect() {
guard hasStarted else { return }
scheduleDelivery()
}

func observe(_ projections: [String: AgentRunProjection]) {
var fired = false
for (key, projection) in projections {
let previous = lastStatusBySurface[key]
lastStatusBySurface[key] = projection.status
guard
projection.status.isTerminal,
previous != projection.status,
previous?.isTerminal != true,
Self.triggerSurfaceKinds.contains(projection.surface.surfaceKind)
else { continue }
fired = true
}
// Prune surfaces that disappeared so a cleared-and-recreated surface is a
// fresh transition rather than a suppressed repeat.
lastStatusBySurface = lastStatusBySurface.filter { projections[$0.key] != nil }
if fired {
scheduleDelivery()
}
}

private func scheduleDelivery() {
if deliveryInFlight {
deliveryQueued = true
return
}
deliveryInFlight = true
scheduleWork { [weak self] in
await self?.runDelivery()
}
}

private func runDelivery() async {
defer {
deliveryInFlight = false
if deliveryQueued {
deliveryQueued = false
scheduleDelivery()
}
}
guard isVoiceSessionLive() else { return }
guard let delta = await peekDelta() else { return }
guard await injectContext(delta.prompt) else {
log("AgentCompletionVoiceDelivery: completion context not delivered; checkpoint unadvanced")
return
}
acknowledge(delta)
log("AgentCompletionVoiceDelivery: delivered \(delta.ids.count) completion(s) to live voice session")
}
}
Original file line number Diff line number Diff line change
@@ -0,0 +1,17 @@
import Foundation

/// Seam for `AgentCompletionVoiceDelivery`: lets completed background-agent
/// results reach the live voice conversation as silent context, without
/// touching the voice turn coordinator's output authority.
extension RealtimeHubController {
/// True when a physical provider session exists (including warm-idle).
var hasLiveVoiceSession: Bool { session != nil }

/// Adds completed background-agent context to the live conversation without
/// requesting a response. Returns false when no session is live so the
/// caller leaves the completion checkpoint unadvanced and retries later.
func injectBackgroundAgentCompletionContext(_ text: String) async -> Bool {
guard let session else { return false }
return await session.sendBackgroundAgentContext(text)
}
}
Original file line number Diff line number Diff line change
@@ -1,4 +1,4 @@
import AppKit

Check warning on line 1 in desktop/macos/Desktop/Sources/FloatingControlBar/RealtimeHubController+SessionDelegate.swift

View workflow job for this annotation

GitHub Actions / Hygiene

Large changed file

desktop/macos/Desktop/Sources/FloatingControlBar/RealtimeHubController+SessionDelegate.swift is 1419 lines; consider splitting files over 800 lines.

Check warning on line 1 in desktop/macos/Desktop/Sources/FloatingControlBar/RealtimeHubController+SessionDelegate.swift

View workflow job for this annotation

GitHub Actions / Desktop Swift Static Contracts

Large changed file

desktop/macos/Desktop/Sources/FloatingControlBar/RealtimeHubController+SessionDelegate.swift is 1419 lines; consider splitting files over 800 lines.
import CoreGraphics
import Foundation
import OmiSupport
Expand Down Expand Up @@ -608,6 +608,7 @@
guard isCurrentSession(source) else { return }
lastWarmAt = Date()
hubConnected = true // authenticated + ready — PTT may now route turns to the hub
AgentCompletionVoiceDelivery.shared.voiceSessionDidConnect()
let replayedReconnectTurn = reconnectAudioBuffer != nil
let replayedReplacementTurn = replacementAudioBuffer != nil
if replayedReplacementTurn {
Expand Down Expand Up @@ -1171,7 +1172,7 @@
return false
}

func hubDidError(_ message: String, source: RealtimeHubSession) {

Check warning on line 1175 in desktop/macos/Desktop/Sources/FloatingControlBar/RealtimeHubController+SessionDelegate.swift

View workflow job for this annotation

GitHub Actions / Hygiene

Long function

func hubDidError(_ message: String, source: RealtimeHubSession) is 200 lines; consider extracting focused helpers over 150 lines.

Check warning on line 1175 in desktop/macos/Desktop/Sources/FloatingControlBar/RealtimeHubController+SessionDelegate.swift

View workflow job for this annotation

GitHub Actions / Desktop Swift Static Contracts

Long function

func hubDidError(_ message: String, source: RealtimeHubSession) is 200 lines; consider extracting focused helpers over 150 lines.
guard isCurrentSession(source) else { return }
if reconnectAudioBuffer == nil {
_ = beginTransportRebindForActiveInputIfNeeded()
Expand Down
Original file line number Diff line number Diff line change
@@ -1,4 +1,4 @@
import Foundation

Check warning on line 1 in desktop/macos/Desktop/Sources/FloatingControlBar/RealtimeHubSession.swift

View workflow job for this annotation

GitHub Actions / Hygiene

Large changed file

desktop/macos/Desktop/Sources/FloatingControlBar/RealtimeHubSession.swift is 1647 lines; consider splitting files over 800 lines.

Check warning on line 1 in desktop/macos/Desktop/Sources/FloatingControlBar/RealtimeHubSession.swift

View workflow job for this annotation

GitHub Actions / Desktop Swift Static Contracts

Large changed file

desktop/macos/Desktop/Sources/FloatingControlBar/RealtimeHubSession.swift is 1647 lines; consider splitting files over 800 lines.
import Network
import VoiceTurnDomain

Expand Down Expand Up @@ -118,6 +118,7 @@
struct RealtimeHubInputLifecycleSnapshot: Equatable {
let isOpen: Bool
let activityOpen: Bool
let pendingTextInputCount: Int
let pendingAudioChunkCount: Int
let pendingVideoFrameCount: Int
let pendingCommit: Bool
Expand Down Expand Up @@ -404,6 +405,7 @@
returning: RealtimeHubInputLifecycleSnapshot(
isOpen: self.isOpen,
activityOpen: self.activityOpen,
pendingTextInputCount: self.pendingTextInputs.count,
pendingAudioChunkCount: self.pendingAudio.count,
pendingVideoFrameCount: self.pendingVideo.count,
pendingCommit: self.pendingCommit,
Expand Down Expand Up @@ -500,6 +502,30 @@
await sendTextInput(text, logLabel: "test text input")
}

/// Silently appends completed background-agent context to the conversation
/// without requesting a response — the model uses it on its next turn.
///
/// Unlike `sendTextInput`, this deliberately does NOT buffer: it returns
/// `true` only when the text is written to the live provider this call, and
/// `false` when the session cannot accept it right now (socket closed, or
/// Gemini has no open activity window). Buffering would let the completion be
/// dropped by `stop()`/`close()` (which clears `pendingTextInputs`) after the
/// caller already advanced its exactly-once checkpoint. Returning `false`
/// keeps the checkpoint unadvanced so the caller retries on the next terminal
/// transition or session connect.
func sendBackgroundAgentContext(_ text: String) async -> Bool {
await withCheckedContinuation { continuation in
q.async { [weak self] in
guard let self, self.isOpen, self.provider == .openai || self.activityOpen else {
continuation.resume(returning: false)
return
}
self.sendTextInputNow(text, logLabel: "background agent completion context")
continuation.resume(returning: true)
}
}
}

/// A provider can complete a tool-only response after accepting the final tool
/// result without emitting a user-facing reply. Continue the same physical turn
/// once, never as a synthetic user request. The continuation is bounded here so
Expand Down
3 changes: 3 additions & 0 deletions desktop/macos/Desktop/Sources/OmiApp.swift
Original file line number Diff line number Diff line change
@@ -1,4 +1,4 @@
import FirebaseAuth

Check warning on line 1 in desktop/macos/Desktop/Sources/OmiApp.swift

View workflow job for this annotation

GitHub Actions / Hygiene

Large changed file

desktop/macos/Desktop/Sources/OmiApp.swift is 1649 lines; consider splitting files over 800 lines.

Check warning on line 1 in desktop/macos/Desktop/Sources/OmiApp.swift

View workflow job for this annotation

GitHub Actions / Desktop Swift Static Contracts

Large changed file

desktop/macos/Desktop/Sources/OmiApp.swift is 1649 lines; consider splitting files over 800 lines.
import FirebaseCore
import OmiSupport
import OmiTheme
Expand Down Expand Up @@ -263,7 +263,7 @@
isExporting: ViewExporter.shouldExport())
}

func applicationDidFinishLaunching(_ notification: Notification) {

Check warning on line 266 in desktop/macos/Desktop/Sources/OmiApp.swift

View workflow job for this annotation

GitHub Actions / Hygiene

Long function

func applicationDidFinishLaunching(_ notification: Notification) is 350 lines; consider extracting focused helpers over 150 lines.

Check warning on line 266 in desktop/macos/Desktop/Sources/OmiApp.swift

View workflow job for this annotation

GitHub Actions / Desktop Swift Static Contracts

Long function

func applicationDidFinishLaunching(_ notification: Notification) is 350 lines; consider extracting focused helpers over 150 lines.
if ViewExporter.shouldExport() {
ViewExporter.run()
return
Expand Down Expand Up @@ -484,6 +484,9 @@
// Start resource monitoring (memory, CPU, disk)
ResourceMonitor.shared.start()

// Route completed background-agent results into live voice sessions.
AgentCompletionVoiceDelivery.shared.start()

scheduleAppLifecycleMaintenance()

// Identify user if already signed in
Expand Down
Loading
Loading