Add probing service#815
Conversation
|
👋 Thanks for assigning @tnull as a reviewer! |
|
🔔 1st Reminder Hey @tnull! This PR has been waiting for your review. |
|
🔔 2nd Reminder Hey @tnull! This PR has been waiting for your review. |
|
🔔 3rd Reminder Hey @tnull! This PR has been waiting for your review. |
|
🔔 4th Reminder Hey @tnull! This PR has been waiting for your review. |
|
🔔 5th Reminder Hey @tnull! This PR has been waiting for your review. |
|
🔔 6th Reminder Hey @tnull! This PR has been waiting for your review. |
There was a problem hiding this comment.
Hi @randomlogin, thanks for the work on this! I've reviewed the first two commits:
I've left a bunch of inline comments addressing configuration and public API, commit hygiene, testing infrastructure, and test flakiness.
In summary:
- A couple of items are exposed publicly that seem like they should be scoped to probing or gated for tests only (see
scoring_fee_paramsinConfigandscorer_channel_liquidityonNode). - The probing tests duplicate existing test helpers (
setup_node,MockLogFacadeLogger). Reusing and extending what's already intests/common/would reduce duplication and keep the test file focused on the tests themselves. test_probe_budget_blocks_when_node_offlinehas a race condition where the prober dispatches probes before the baseline capacity is measured, causing the assertion between the baseline and stuck capacities to fail. Details in the inline comment.- A few nits about commit hygiene, import structure, and suggestions for renaming stuff.
Also needs to be rebased.
|
🔔 7th Reminder Hey @tnull! This PR has been waiting for your review. |
|
@enigbe, thanks for a review, the updates are incoming soon. |
randomlogin
force-pushed
the
add-probing-service
branch
3 times, most recently
from
436e4a3 to
07dfde4
Compare
randomlogin
force-pushed
the
add-probing-service
branch
from
ff741c2 to
c31f1ce
Compare
tnull
left a comment
tnull
left a comment
There was a problem hiding this comment.
Thanks for taking this on and excuse the delay here!
Did a first review pass and this already looks great! Here are some relatively minor comments, mostly concerning the API design.
|
🔔 4th Reminder Hey @enigbe! This PR has been waiting for your review. |
tnull
left a comment
tnull
left a comment
There was a problem hiding this comment.
Seems tests are failing right now:
thread 'exhausted_probe_budget_blocks_new_probes' (167312) panicked at tests/probing_tests.rs:381:5:
no probe dispatched within 15 s
failures:
exhausted_probe_budget_blocks_new_probes
probe_budget_increments_and_decrements
f99786b to
1e73e6e
Compare
|
As discussed above this is now blocked on lightningdevkit/rust-lightning#4603 becoming available. Will revisit once earlier LDK bumps have landed. |
Introduce a background probing service that periodically sends payment probes to discover liquidity along Lightning routes. Probes update the local scorer with channel liquidity information, improving pathfinding for subsequent real payments. The service supports three strategies: - HighDegree: probes nodes with the most channels in the network graph - Random: walks random paths from the local node - Custom: user-supplied strategy via the `ProbingStrategy` trait A dedicated `ProbingConfigBuilder` exposes amount bounds, locked-msat caps, probing intervals, and per-node cooldowns, with sensible defaults. The service runs as a cancellable background task driven by the existing `Runtime`, and budget accounting tracks both in-flight and locked amounts to bound outbound liquidity exposure. UniFFI bindings expose the probing service to the Swift, Kotlin, and Python language bindings. Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
Add integration tests that verify the probing service fires probes on the configured interval and respects the locked-msat budget cap. Shared helpers in tests/common are extended with probing-aware setup. Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
Changed the exhaust test to be statistical (locked amount never exceeds the cap) instead of trying to turn intermediary routing node offline when it hasn't yet forwarded the probe htlc.
Previously when calculated currently locked amount, we didn't account for preflight probes sent on a payment which could result in an incorrect value of probe locked_msat. Now Prober saves the PaymentId of probes it sent and tracks them on release, ignoring the user-sent ones.
Previously we tried to store the total amount of funds locked and/or exact probes in flight which was difficult to persist and restore after restart. Now it is completely deduced from the channel manager.
randomlogin
force-pushed
the
add-probing-service
branch
from
9e00279 to
73882a8
Compare
|
After the dependency update we have everything needed for this PR. Also in the last commit I renamed |
|
🔔 1st Reminder Hey @tnull! This PR has been waiting for your review. |
|
🔔 2nd Reminder Hey @tnull! This PR has been waiting for your review. |
tnull
left a comment
tnull
left a comment
There was a problem hiding this comment.
Excuse the delay here! This mostly looks good, I think the most important thing that still needs to be addressed is not setting the diversity multiplier globally, as it could unexpected consequences for users. This unfortunately also needs a rebase, and while you rebase please squash the fixup commits so that you don't touch the same codepaths multiple times (e.g. squash the rename commit)
| } | ||
| } | ||
|
|
||
| /// A UniFFI-compatible wrapper around [`ProbingConfigBuilder`] that uses interior mutability |
There was a problem hiding this comment.
nit: Lets drop the mention of UniFFI from these docs, they should just be identical to the ones on ProbingConfigBuilder as the bindings users surely don't care about UniFFI or whatever bindings generator we use. They only care about how the API works, so the UniFFI objects should just be treated as normal objects and explain the API.
While we're at it, we should probably also do the typedef trick we do on Builder to expose this as ProbingConfigBuilder rather than ArcedProbingConfigBuilder
| let scoring_fee_params = ProbabilisticScoringFeeParameters::default(); | ||
| let mut scoring_fee_params = ProbabilisticScoringFeeParameters::default(); | ||
| if let Some(penalty) = probing_config.and_then(|c| c.diversity_penalty_msat) { | ||
| scoring_fee_params.probing_diversity_penalty_msat = penalty; |
There was a problem hiding this comment.
Hmm, this will install the probing_diversity_penalty_msat globally, which is probably not what we want when a node enables the background probing but still wants to send regular payments.
This probably means that we can't use the default router for probing, but rather need to call https://docs.rs/lightning/latest/lightning/routing/router/fn.find_route.html manually (as it allows us to pass the score params directly per call) to create a route and then hand the pre-built route to https://docs.rs/lightning/latest/lightning/ln/channelmanager/struct.ChannelManager.html#method.send_payment_with_route
| .list_recent_payments() | ||
| .into_iter() | ||
| .filter_map(|p| match p { | ||
| RecentPaymentDetails::Pending { is_probe: true, total_msat, .. } => { |
There was a problem hiding this comment.
Codex:
- [P2] Include route fees in pending probe budget — /home/tnull/workspace/ldk-node/src/probing.rs:762-763
For pending probes this uses RecentPaymentDetails::Pending.total_msat, but LDK defines that value as excluding routing fees while the first-hop HTLC locks
the final amount plus all route fees. On routes or custom strategies with non-trivial fees, locked_msat() under-reports in-flight liquidity, so subsequent
budget checks can allow max_locked_msat to be exceeded and the public locked amount is misleading.
5 participants
Added a probing service which is used to send probes to estimate channels' capacities.
Related issue: #765.
Probing is intended to be used in two ways:
For probing a new abstraction
Proberis defined and is (optionally) created during node building.Prober periodically sends probes to feed the data to the scorer.
Prober sends probes using a ProbingStrategy.
ProbingStrategy trait has only one method:
fn next_probe(&self) -> Option<Probe>; every tick it generates a probe, whereProberepresents how to send a probe.To accommodate two different ways the probing is used, we either construct a probing route manually (
Probe::PrebuiltRoute) or rely on the router/scorer (Probe::Destination).Prober tracks how much liquidity is locked in-flight in probes, prevents the new probes from firing if the cap is reached.
There are two probing strategies implemented:
Random probing strategy, it picks a random route from the current node, the route is probed via
send_probe, thus ignores scoring parameters (what hops to pick), it also ignoresliquidity_limit_multiplierwhich prohibits taking a hop if its capacity is too small. It is a true random route.High degree probing strategy, it examines the graph and finds the nodes with the biggest number of (public) channels and probes routes to them using
send_spontaneous_preflight_probeswhich uses the current router/scorer.The former is meant to be used on payment nodes, while the latter on probing nodes. For the HighDegreeStrategy to work it is recommended to set
probing_diversity_penalty_msatto some nonzero value to prevent routes reuse, however it may fail to find any available routes.There are three tests added:
Example output (runs for ~1 minute, needs
--nocaptureflag):For performance testing I had to expose the scoring data (
scorer_channel_liquidity).Also exposed
scoring_fee_params: ProbabilisticScoringFeeParameterstoConfig.TODOs: