Skip to content

fix(deployment): close managed deployments once every lease on them has closed - #3939

Merged
baktun14 merged 1 commit into
mainfrom
fix/deployment-close-all-leases-closed
Sep 14, 2026
Merged

baktun14 merged 1 commit into
mainfrom
fix/deployment-close-all-leases-closed

Conversation

@baktun14

Copy link
Copy Markdown
Contributor

Why

A managed deployment whose leases have all been closed by the provider stays open on chain with no lease, no spend, and whatever deposit is left locked in escrow, and nothing in Console ever closes it. One user has three such deployments from June 22: each got a lease that the provider closed within minutes to hours (one with lease_closed_reason_unstable), and all three are still active today.

The cleanup-stale-deployments cron is the only closer that does not need a lease, but findStaleDeployments selected deployments with COUNT(leases) = 0, counting closed lease rows too. Every other closer (unreachable-provider sweeps, runtime-limit job, trial closers) starts from an active lease, so a deployment whose leases all closed is out of reach for all of them.

What

  • findStaleDeployments now counts only active leases, and also requires the last lease close height to be below the cutoff, so a lease closed seconds ago is not acted on until the same ten-minute window that already applies to the creation height has passed. Bids expire well inside that window, so a group without a lease past the cutoff cannot get one through Console anyway.
  • The option is renamed from createdHeight to staleBeforeHeight since it now bounds both heights. The cleaner's private method is renamed to match what it does.
  • New chain-DB integration spec for findStaleDeployments covering never-leased, all-leases-closed, lease closed after the cutoff, active lease, created after the cutoff, already closed, and other owners.

Before merging it is worth sizing the first run: count open deployments on managed wallets that have lease rows but no active lease. The twice-daily cron closes each wallet's batch in one derived tx and already handles the unsettleable-escrow and already-closed cases.

…as closed

The stale-deployment sweep only ever selected deployments with no lease row at all. A deployment
whose provider closed its lease kept an open, lease-less deployment on chain for good: no other
closer applies, since the unreachable-provider sweeps and the runtime-limit job both key off an
active lease.

Three deployments from June are still open this way for one user, each with a lease the provider
closed within minutes to hours of creation.

The sweep now treats a deployment as stale when it has no active lease and its last lease closed
before the same ten-minute cutoff that already applies to its creation height. Bids expire well
inside that window, so a group left without a lease past the cutoff cannot get one through
Console anyway, and closing returns whatever escrow remains to the wallet.
@baktun14
baktun14 requested a review from a team as a code owner September 13, 2026 13:45
@coderabbitai

coderabbitai Bot commented Sep 13, 2026

Copy link
Copy Markdown
Contributor

Warning

Review limit reached

  • Run on-demand review

On-demand reviews are free for the next 7 days. After that, they cost $0.25 per reviewed file.

Or wait 48 minutes for your next included review.

Check out review usage here.

View limit details

Limit details: You’ve used all 2 included reviews currently available. Your 50 included PR review attempts over the past 7 days set your current allowance at 2 reviews per hour.

Learn how review limits work.

Review configuration:

⚙️ Run configuration

Configuration used: Path: .coderabbit.yaml

Review profile: CHILL

Plan: Essentials

Run ID: 525efad9-d188-4ff8-920a-a596bf17ad68

📥 Commits

Reviewing files that changed from the base of the PR and between ec579eb and df262dd.

📒 Files selected for processing (4)
  • apps/api/src/deployment/repositories/deployment/deployment.repository.integration.ts
  • apps/api/src/deployment/repositories/deployment/deployment.repository.ts
  • apps/api/src/deployment/services/stale-managed-deployments-cleaner/stale-managed-deployments-cleaner.service.spec.ts
  • apps/api/src/deployment/services/stale-managed-deployments-cleaner/stale-managed-deployments-cleaner.service.ts

Comment @coderabbitai help to get the list of available commands.

@codecov

codecov Bot commented Sep 13, 2026

Copy link
Copy Markdown

Codecov Report

❌ Patch coverage is 85.71429% with 1 line in your changes missing coverage. Please review.
✅ Project coverage is 81.78%. Comparing base (ec579eb) to head (df262dd).
✅ All tests successful. No failed tests found.

Files with missing lines Patch % Lines
...t/repositories/deployment/deployment.repository.ts 75.00% 1 Missing ⚠️
Additional details and impacted files
@@            Coverage Diff             @@
##             main    #3939      +/-   ##
==========================================
- Coverage   82.35%   81.78%   -0.58%     
==========================================
  Files        1280     1180     -100     
  Lines       35404    32799    -2605     
  Branches     8566     8046     -520     
==========================================
- Hits        29158    26824    -2334     
+ Misses       5515     5262     -253     
+ Partials      731      713      -18     
Flag Coverage Δ *Carryforward flag
api 92.73% <85.71%> (-0.03%) ⬇️
deploy-web 72.42% <ø> (ø) Carriedforward from ec579eb
log-collector ?
notifications 94.35% <ø> (ø) Carriedforward from ec579eb
provider-console 81.68% <ø> (ø) Carriedforward from ec579eb
provider-inventory ?
provider-proxy 88.61% <ø> (ø) Carriedforward from ec579eb
tx-signer ?

*This pull request uses carry forward flags. Click here to find out more.

Files with missing lines Coverage Δ
...eaner/stale-managed-deployments-cleaner.service.ts 100.00% <100.00%> (ø)
...t/repositories/deployment/deployment.repository.ts 91.66% <75.00%> (-1.20%) ⬇️

... and 101 files with indirect coverage changes

🚀 New features to boost your workflow:
  • 📦 JS Bundle Analysis: Save yourself from yourself by tracking and limiting bundle sizes in JS merges.

@claude claude Bot left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Beyond the reported (zero) findings, I also traced the widened staleness definition into its other call site: cleanUpForWallet(wallet, 0) is invoked from reclaimTrialOrphanedDeployments before every trial-wallet deployment create, so this fix now also closes trial deployments whose lease closed very recently (not just never-leased ones) — this matches the PR's stated intent rather than being a side effect, but it's worth a human confirming that broadened blast radius is acceptable.

Extended reasoning...

The diff renames createdHeight to staleBeforeHeight and rewrites the HAVING clause in findStaleDeployments so a deployment counts as stale only when it has zero currently-open leases (COUNT(...) FILTER (WHERE closedHeight IS NULL) = 0) and its last closed lease closed before the cutoff (COALESCE(MAX(closedHeight), 0) < staleBeforeHeight), replacing the old buggy COUNT(leases) = 0 check that ignored closed leases entirely. A new asHeight() guard validates the height is a safe integer before it is string-interpolated into the raw SQL literal, mitigating injection risk from that one non-parameterized value (the other height usage stays parameterized via Op.lt). New integration tests cover never-leased, all-leases-closed-before-cutoff, last-lease-closed-at/after-cutoff, still-active-lease, created-at/after-cutoff, already-closed, and other-owner scenarios, and correctly use the repo's setup() convention.

I independently re-derived the SQL logic and it matches the stated semantics, including the "never leased" case falling out of COALESCE(..., 0) < staleBeforeHeight. I also checked the one behavior-widening consequence flagged by the bug hunt: cleanUpForWallet(wallet, 0) is called synchronously from deployment-writer.service.ts's reclaimTrialOrphanedDeployments before every trial-wallet deployment create, so the broadened staleness definition (any deployment with no active lease and last-closed-lease height below the near-current cutoff) now applies there too, not just in the twice-daily managed-wallet cron. This looks like the intended fix rather than an unintended side effect, but it does mean closes are triggered synchronously on a hot path (deployment creation) for a larger population than before.

Given this touches on-chain fund-closing logic (escrow/deposit release) and a subtle SQL correctness fix that is easy to get subtly wrong, plus a real (if likely intended) widening of behavior on a synchronous hot path, I'm choosing to defer with a brief note on what else was checked, rather than stay fully silent or approve outright.

@baktun14
baktun14 added this pull request to the merge queue Sep 14, 2026
Merged via the queue into main with commit f61c718 Sep 14, 2026
59 checks passed
@baktun14
baktun14 deleted the fix/deployment-close-all-leases-closed branch September 14, 2026 12:26
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant