server, config: support updating leader lease online#10631
Conversation
ti-chi-bot
Bot
added
release-note
|
Note Reviews pausedIt looks like this branch is under active development. To avoid overwhelming you with review comments due to an influx of new commits, CodeRabbit has automatically paused this review. You can configure this behavior by changing the Use the following commands to manage reviews:
Use the checkboxes below for quick actions:
📝 WalkthroughWalkthroughPersist leader-lease into PersistOptions, expose it via the top-level ChangesLeader-lease persistence and runtime control
Sequence DiagramsequenceDiagram
participant Client
participant API as Config API
participant Server as PD Server
participant Persist as PersistOptions
participant Storage as Config Storage
Client->>API: POST /pd/api/v1/config {"lease": 3000}
API->>API: Detect top-level "lease"
API->>Server: SetLeaderLease(3000)
Server->>Server: Validate lease (>0)
Server->>Persist: SetLeaderLease(3000)
Persist->>Persist: Update atomic leaderLease
Server->>Storage: Persist updated config
Storage-->>Server: OK
Server-->>API: success
API-->>Client: 200 OK
Note over Client,Storage: On restart / reload
Server->>Storage: Load persisted config
Storage-->>Server: Config{..., lease:3000}
Server->>Persist: loadPersistedConfig(...)
Persist->>Persist: Validate lease -> true
Persist-->>Server: leaderLease=3000
Server->>Server: Use persisted leaderLease for campaigning
Estimated Code Review Effort🎯 4 (Complex) | ⏱️ ~45 minutes Possibly related issues
Possibly related PRs
Suggested labels
Suggested reviewers
Poem
🚥 Pre-merge checks | ✅ 4 | ❌ 1❌ Failed checks (1 warning)
✅ Passed checks (4 passed)
✏️ Tip: You can configure your own custom pre-merge checks in the settings. ✨ Finishing Touches🧪 Generate unit tests (beta)
Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out. Comment |
ti-chi-bot
Bot
added
the
size/L
![coderabbitai[bot]](https://avatars.githubusercontent.com/in/347564?s=60&v=4){kind=small}
There was a problem hiding this comment.
Actionable comments posted: 2
🤖 Prompt for all review comments with AI agents
Verify each finding against the current code and only fix it if needed.
Inline comments:
In `@server/config/persist_options.go`:
- Around line 829-831: Reload currently treats a default-initialized
cfg.LeaderLease as a present, valid value and overwrites any startup lease; fix
this by making the persisted presence explicit (either change cfg.LeaderLease to
a pointer type or add a boolean flag set by LoadConfig that indicates the field
was present) and only call IsValidLeaderLease and o.SetLeaderLease when that
presence indicator shows the field existed in the persisted blob; apply the same
change/guard for the other lease-related block at the 838-840 region so you only
apply persisted leases if the field was actually present in the loaded config.
🪄 Autofix (Beta)
Fix all unresolved CodeRabbit comments on this PR:
- Push a commit to this branch (recommended)
- Create a new PR with the fixes
ℹ️ Review info
⚙️ Run configuration
Configuration used: defaults
Review profile: CHILL
Plan: Pro
Run ID: b9f77e78-96a8-454f-a2b0-f0b11648ce41
📥 Commits
Reviewing files that changed from the base of the PR and between f941a7e and c8a100b984c363cae21ad8bec4ea51029b6b2175.
📒 Files selected for processing (5)
server/api/config.goserver/config/config_test.goserver/config/persist_options.goserver/server.gotests/server/api/api_test.go
JmPotato
force-pushed
the
bob/leader-lease-online-config
branch
from
c8a100b to
f66c676
Compare
There was a problem hiding this comment.
♻️ Duplicate comments (2)
server/config/persist_options.go (1)
829-831:⚠️ Potential issue | 🟠 MajorDo not apply
LeaderLeaseunless the persistedleasefield actually exists.On Line 829,
IsValidLeaderLease(cfg.LeaderLease)is not enough to distinguish “field missing” vs “field present.” For older stored blobs withoutlease, reload can still overwrite a custom startup lease.💡 Suggested fix (presence check before applying)
func (o *PersistOptions) Reload(storage endpoint.ConfigStorage) error { cfg := &persistedConfig{Config: &Config{}} + leaseProbe := struct { + LeaderLease *int64 `json:"lease"` + }{} // Pass nil to initialize cfg to default values (all items undefined) if err := cfg.Adjust(nil, true); err != nil { return err } isExist, err := storage.LoadConfig(cfg) if err != nil { return err } + _, _ = storage.LoadConfig(&leaseProbe) adjustScheduleCfg(&cfg.Schedule) ... if isExist { ... - if IsValidLeaderLease(cfg.LeaderLease) { - o.SetLeaderLease(cfg.LeaderLease) + if leaseProbe.LeaderLease != nil && IsValidLeaderLease(*leaseProbe.LeaderLease) { + o.SetLeaderLease(*leaseProbe.LeaderLease) } ... } return nil }🤖 Prompt for AI Agents
Verify each finding against the current code and only fix it if needed. In `@server/config/persist_options.go` around lines 829 - 831, The current code calls o.SetLeaderLease when IsValidLeaderLease(cfg.LeaderLease) is true, which overwrites a startup lease even if the persisted blob lacked a `lease` field; change the condition to require the persisted field's presence as well (e.g., check the config wrapper/flag that indicates the `lease` field was present or use a nil/presence check on cfg.LeaderLease) before calling o.SetLeaderLease; update the if around IsValidLeaderLease(cfg.LeaderLease) to something like "if cfg.HasLeaderLease() && IsValidLeaderLease(cfg.LeaderLease) { o.SetLeaderLease(cfg.LeaderLease) }" (replace HasLeaderLease with the actual presence indicator in the config struct).server/api/config.go (1)
218-219:⚠️ Potential issue | 🟠 Major
leaseis updateable, but followerGET /configcan still serve stalelease.With Line 218 enabling dynamic
lease, follower reads need to merge lease from leader too. Today followerGetConfigonly syncs schedule/replication (Lines 81-82), soleasemay lag until reload/leadership change.💡 Suggested fix in follower merge path
mergedCfg := localCfg mergedCfg.Replication = leaderCfg.Replication mergedCfg.Schedule = leaderCfg.Schedule + mergedCfg.LeaderLease = leaderCfg.LeaderLease h.rd.JSON(w, http.StatusOK, mergedCfg)🤖 Prompt for AI Agents
Verify each finding against the current code and only fix it if needed. In `@server/api/config.go` around lines 218 - 219, GetConfig on followers currently syncs only schedule/replication and can return a stale lease; update the follower merge path to include the leader's lease value as well. In the follower GET /config handling (where schedule/replication are merged) add logic to fetch and merge the leader's lease into the response so that the value updated via updateLeaderLease is reflected to followers; ensure you reuse the same merge/order semantics as schedule/replication so updateLeaderLease (case "lease") changes propagate immediately to follower GetConfig responses.
🧹 Nitpick comments (1)
server/config/config_test.go (1)
131-148: Add one backward-compat test for missingleasekey.Please add a case where stored config omits
leaseentirely and assert reload keeps the in-memory startup lease unchanged.🤖 Prompt for AI Agents
Verify each finding against the current code and only fix it if needed. In `@server/config/config_test.go` around lines 131 - 148, Extend TestLeaderLeaseReloadIgnoresNonPositivePersistedValues to include a case where the persisted config omits the lease key entirely: call storage.SaveConfig with an empty value (e.g., struct{}{} or a map with no "lease" key), then run the same reload flow (NewConfig, cfg.Adjust, set cfg.LeaderLease to 7, NewPersistOptions, opt.Reload) and assert opt.GetLeaderLease() remains 7; reference TestLeaderLeaseReloadIgnoresNonPositivePersistedValues, storage.SaveConfig, NewConfig, NewPersistOptions, opt.Reload, and opt.GetLeaderLease to locate the code to change.
🤖 Prompt for all review comments with AI agents
Verify each finding against the current code and only fix it if needed.
Duplicate comments:
In `@server/api/config.go`:
- Around line 218-219: GetConfig on followers currently syncs only
schedule/replication and can return a stale lease; update the follower merge
path to include the leader's lease value as well. In the follower GET /config
handling (where schedule/replication are merged) add logic to fetch and merge
the leader's lease into the response so that the value updated via
updateLeaderLease is reflected to followers; ensure you reuse the same
merge/order semantics as schedule/replication so updateLeaderLease (case
"lease") changes propagate immediately to follower GetConfig responses.
In `@server/config/persist_options.go`:
- Around line 829-831: The current code calls o.SetLeaderLease when
IsValidLeaderLease(cfg.LeaderLease) is true, which overwrites a startup lease
even if the persisted blob lacked a `lease` field; change the condition to
require the persisted field's presence as well (e.g., check the config
wrapper/flag that indicates the `lease` field was present or use a nil/presence
check on cfg.LeaderLease) before calling o.SetLeaderLease; update the if around
IsValidLeaderLease(cfg.LeaderLease) to something like "if cfg.HasLeaderLease()
&& IsValidLeaderLease(cfg.LeaderLease) { o.SetLeaderLease(cfg.LeaderLease) }"
(replace HasLeaderLease with the actual presence indicator in the config
struct).
---
Nitpick comments:
In `@server/config/config_test.go`:
- Around line 131-148: Extend
TestLeaderLeaseReloadIgnoresNonPositivePersistedValues to include a case where
the persisted config omits the lease key entirely: call storage.SaveConfig with
an empty value (e.g., struct{}{} or a map with no "lease" key), then run the
same reload flow (NewConfig, cfg.Adjust, set cfg.LeaderLease to 7,
NewPersistOptions, opt.Reload) and assert opt.GetLeaderLease() remains 7;
reference TestLeaderLeaseReloadIgnoresNonPositivePersistedValues,
storage.SaveConfig, NewConfig, NewPersistOptions, opt.Reload, and
opt.GetLeaderLease to locate the code to change.
ℹ️ Review info
⚙️ Run configuration
Configuration used: defaults
Review profile: CHILL
Plan: Pro
Run ID: 5e0a21ab-1f69-44ae-a4af-914af870c79f
📥 Commits
Reviewing files that changed from the base of the PR and between c8a100b984c363cae21ad8bec4ea51029b6b2175 and f66c67600cffe2e472032a0970f6964761f65450.
📒 Files selected for processing (5)
server/api/config.goserver/config/config_test.goserver/config/persist_options.goserver/server.gotests/server/api/api_test.go
JmPotato
force-pushed
the
bob/leader-lease-online-config
branch
from
f66c676 to
6168127
Compare
There was a problem hiding this comment.
♻️ Duplicate comments (1)
server/server.go (1)
1013-1025:⚠️ Potential issue | 🟠 MajorKeep
leasein the follower merge path.
GetConfig()still only overlaysScheduleandReplicationfromleaderCfg. After a runtime lease update, a follower can keep serving an oldleaseuntil its local cache reloads, so/pd/api/v1/configcan disagree with the leader. Please copyLeaderLeasefromleaderCfghere too.Suggested patch
mergedCfg := localCfg mergedCfg.Replication = leaderCfg.Replication mergedCfg.Schedule = leaderCfg.Schedule +mergedCfg.LeaderLease = leaderCfg.LeaderLease🤖 Prompt for AI Agents
Verify each finding against the current code and only fix it if needed. In `@server/server.go` around lines 1013 - 1025, GetConfig currently doesn't copy the leader's runtime lease into the returned cfg causing followers to serve stale LeaderLease; in GetConfig ensure you overlay LeaderLease from the persisted leader config (use the value returned by persistOptions.GetLeaderLease / the leaderCfg's lease) into cfg so the returned config reflects the leader's lease update (update the LeaderLease field in GetConfig to copy/clone the leader's lease value rather than leaving a stale local one).
🤖 Prompt for all review comments with AI agents
Verify each finding against the current code and only fix it if needed.
Duplicate comments:
In `@server/server.go`:
- Around line 1013-1025: GetConfig currently doesn't copy the leader's runtime
lease into the returned cfg causing followers to serve stale LeaderLease; in
GetConfig ensure you overlay LeaderLease from the persisted leader config (use
the value returned by persistOptions.GetLeaderLease / the leaderCfg's lease)
into cfg so the returned config reflects the leader's lease update (update the
LeaderLease field in GetConfig to copy/clone the leader's lease value rather
than leaving a stale local one).
ℹ️ Review info
⚙️ Run configuration
Configuration used: defaults
Review profile: CHILL
Plan: Pro
Run ID: 4124abaa-3025-491c-8fe5-0b2d872988cd
📥 Commits
Reviewing files that changed from the base of the PR and between f66c67600cffe2e472032a0970f6964761f65450 and 61681277780107c2387e22f0fc8429e0ad14bb45.
📒 Files selected for processing (5)
server/api/config.goserver/config/config_test.goserver/config/persist_options.goserver/server.gotests/server/api/api_test.go
🚧 Files skipped from review as they are similar to previous changes (1)
- server/config/persist_options.go
| @@ -1013,6 +1013,7 @@ func (s *Server) GetServiceMiddlewareConfig() *config.ServiceMiddlewareConfig { | |||
| // GetConfig gets the config information. | |||
| func (s *Server) GetConfig() *config.Config { | |||
| cfg := s.cfg.Clone() | |||
| cfg.LeaderLease = s.persistOptions.GetLeaderLease() | |||
There was a problem hiding this comment.
This keeps the leader response fresh, but follower GET /config still only merges schedule/replication from the leader. Can we also copy LeaderLease there so followers don't return the old lease after a POST?
There was a problem hiding this comment.
Fixed in c334f2830 by copying LeaderLease from the leader config in the follower GET /config merge path.
| @@ -813,12 +820,42 @@ func (o *PersistOptions) Reload(storage endpoint.ConfigStorage) error { | |||
| o.labelProperty.Store(cfg.LabelProperty) | |||
| o.keyspace.Store(&cfg.Keyspace) | |||
| o.microservice.Store(&cfg.Microservice) | |||
| if IsValidLeaderLease(cfg.LeaderLease) { | |||
There was a problem hiding this comment.
This still can't tell "missing lease field" from "lease was persisted". For old config blobs without lease, Adjust fills the default 5 and this can override a custom startup lease.
There was a problem hiding this comment.
Fixed in c334f2830. persistedConfig now records whether the top-level lease key was actually present during JSON unmarshal, and Reload only applies a present positive lease.
| if err != nil { | ||
| return err | ||
| } | ||
| if isExist && IsValidLeaderLease(cfg.LeaderLease) { |
There was a problem hiding this comment.
Same here: before applying this value, we should first know the lease key was actually present in storage, not just filled by defaults.
There was a problem hiding this comment.
Fixed in c334f2830 with the same presence check in ReloadLeaderLease, so narrow reload keeps the startup lease when old config blobs do not contain lease.
JmPotato
force-pushed
the
bob/leader-lease-online-config
branch
from
6168127 to
c334f28
Compare
ti-chi-bot
Bot
added
size/XL
There was a problem hiding this comment.
🧹 Nitpick comments (1)
tests/server/api/api_test.go (1)
128-156: ⚡ Quick winReset
cfgper request and verify all followers.Reusing the same
cfgmap across multipleReadGetJSONcalls is fragile:json.Unmarshalinto an existing map merges rather than replaces keys, so if a future regression causesleaseto be omitted from a response, the residual value from the previous call would mask the bug. Additionally,breakafter the first follower only validates one of the two non-leader members.♻️ Proposed fix
- var cfg map[string]any - err = testutil.ReadGetJSON(re, tests.TestDialClient, configURL, &cfg) + cfg := map[string]any{} + err = testutil.ReadGetJSON(re, tests.TestDialClient, configURL, &cfg) re.NoError(err) re.Equal(float64(7), cfg["lease"]) _, hasLeader := cfg["leader"] re.False(hasLeader) @@ - err = testutil.ReadGetJSON(re, tests.TestDialClient, configURL, &cfg) + cfg = map[string]any{} + err = testutil.ReadGetJSON(re, tests.TestDialClient, configURL, &cfg) re.NoError(err) re.Equal(float64(10), cfg["lease"]) _, hasLeader = cfg["leader"] re.False(hasLeader) @@ - for _, s := range cluster.GetServers() { - if s.GetServer().Name() == leader.GetServer().Name() { - continue - } - followerConfigURL := s.GetAddr() + "/pd/api/v1/config" - err = testutil.ReadGetJSON(re, tests.TestDialClient, followerConfigURL, &cfg) - re.NoError(err) - re.Equal(float64(10), cfg["lease"]) - break - } + for _, s := range cluster.GetServers() { + if s.GetServer().Name() == leader.GetServer().Name() { + continue + } + followerConfigURL := s.GetAddr() + "/pd/api/v1/config" + followerCfg := map[string]any{} + err = testutil.ReadGetJSON(re, tests.TestDialClient, followerConfigURL, &followerCfg) + re.NoError(err) + re.Equal(float64(10), followerCfg["lease"]) + }🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the rest with a brief reason, keep changes minimal, and validate. In `@tests/server/api/api_test.go` around lines 128 - 156, The test reuses the same cfg map across multiple testutil.ReadGetJSON calls which can cause stale keys to persist because json.Unmarshal merges into existing maps; change each ReadGetJSON call to use a fresh map (redeclare or make a new map[string]any before each call to testutil.ReadGetJSON) so responses replace prior state, and in the follower loop remove the break so you assert the lease value for all non-leader servers (identify servers via cluster.GetServers() and compare against leader.GetServer().Name()).
🤖 Prompt for all review comments with AI agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.
Nitpick comments:
In `@tests/server/api/api_test.go`:
- Around line 128-156: The test reuses the same cfg map across multiple
testutil.ReadGetJSON calls which can cause stale keys to persist because
json.Unmarshal merges into existing maps; change each ReadGetJSON call to use a
fresh map (redeclare or make a new map[string]any before each call to
testutil.ReadGetJSON) so responses replace prior state, and in the follower loop
remove the break so you assert the lease value for all non-leader servers
(identify servers via cluster.GetServers() and compare against
leader.GetServer().Name()).
ℹ️ Review info
⚙️ Run configuration
Configuration used: defaults
Review profile: CHILL
Plan: Pro
Run ID: 093b3c9c-8c7c-4f03-a3ed-bf02d043ad33
📥 Commits
Reviewing files that changed from the base of the PR and between 61681277780107c2387e22f0fc8429e0ad14bb45 and c334f2830db2eb70524bf7954eaa6e156dea4847.
📒 Files selected for processing (5)
server/api/config.goserver/config/config_test.goserver/config/persist_options.goserver/server.gotests/server/api/api_test.go
JmPotato
force-pushed
the
bob/leader-lease-online-config
branch
from
c334f28 to
75cda2d
Compare
ti-chi-bot
Bot
added
size/L
JmPotato
force-pushed
the
bob/leader-lease-online-config
branch
from
75cda2d to
93d6178
Compare
There was a problem hiding this comment.
🧹 Nitpick comments (1)
tests/server/api/api_test.go (1)
147-156: 💤 Low valueOptional: replace the single-iteration loop with a direct follower lookup (or check all followers).
The
for ... { continue ... break }pattern only validates one non-leader server but reads like an iteration over all servers. Either drop thebreakto assert lease propagation on every follower (stronger guarantee), or usecluster.GetServer(cluster.GetFollower())to make the single-follower intent explicit.♻️ Suggested refactor (assert on every follower)
- for _, s := range cluster.GetServers() { - if s.GetServer().Name() == leader.GetServer().Name() { - continue - } - followerConfigURL := s.GetAddr() + "/pd/api/v1/config" - err = testutil.ReadGetJSON(re, tests.TestDialClient, followerConfigURL, &cfg) - re.NoError(err) - re.Equal(float64(10), cfg["lease"]) - _, hasLeader = cfg["leader"] - re.False(hasLeader) - break - } + for _, s := range cluster.GetServers() { + if s.GetServer().Name() == leader.GetServer().Name() { + continue + } + followerConfigURL := s.GetAddr() + "/pd/api/v1/config" + err = testutil.ReadGetJSON(re, tests.TestDialClient, followerConfigURL, &cfg) + re.NoError(err) + re.Equal(float64(10), cfg["lease"]) + _, hasLeader = cfg["leader"] + re.False(hasLeader) + }🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the rest with a brief reason, keep changes minimal, and validate. In `@tests/server/api/api_test.go` around lines 147 - 156, The test currently iterates servers but uses continue+break which only checks a single non-leader; either (A) assert on every follower by removing the break so the loop checks lease propagation for all servers returned by cluster.GetServers() (keep the existing comparison against leader.GetServer().Name(), use s.GetAddr() + "/pd/api/v1/config", testutil.ReadGetJSON, and re.Equal(float64(10), cfg["lease"]) for each), or (B) make the single-follower intent explicit by replacing the loop with a direct lookup like cluster.GetServer(cluster.GetFollower()) and then call testutil.ReadGetJSON against that server address and assert cfg["lease"] equals 10. Choose one approach and update the test accordingly.
🤖 Prompt for all review comments with AI agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.
Nitpick comments:
In `@tests/server/api/api_test.go`:
- Around line 147-156: The test currently iterates servers but uses
continue+break which only checks a single non-leader; either (A) assert on every
follower by removing the break so the loop checks lease propagation for all
servers returned by cluster.GetServers() (keep the existing comparison against
leader.GetServer().Name(), use s.GetAddr() + "/pd/api/v1/config",
testutil.ReadGetJSON, and re.Equal(float64(10), cfg["lease"]) for each), or (B)
make the single-follower intent explicit by replacing the loop with a direct
lookup like cluster.GetServer(cluster.GetFollower()) and then call
testutil.ReadGetJSON against that server address and assert cfg["lease"] equals
10. Choose one approach and update the test accordingly.
ℹ️ Review info
⚙️ Run configuration
Configuration used: defaults
Review profile: CHILL
Plan: Pro
Run ID: 43b7378c-d7f7-4313-b380-bfe93cba0972
📥 Commits
Reviewing files that changed from the base of the PR and between c334f2830db2eb70524bf7954eaa6e156dea4847 and 75cda2dc509ce9a990702aca9ffc41b4bae9d8a7.
📒 Files selected for processing (5)
server/api/config.goserver/config/config_test.goserver/config/persist_options.goserver/server.gotests/server/api/api_test.go
Signed-off-by: JmPotato <[email protected]>
JmPotato
force-pushed
the
bob/leader-lease-online-config
branch
from
93d6178 to
8834eea
Compare
|
/test pull-unit-test-next-gen-2 |
|
/test pull-unit-test-next-gen-3 |
1 similar comment
|
/test pull-unit-test-next-gen-3 |
|
@YuhaoZhang00: adding LGTM is restricted to approvers and reviewers in OWNERS files. DetailsIn response to this: Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes-sigs/prow repository. |
|
[APPROVALNOTIFIER] This PR is NOT APPROVED This pull-request has been approved by: YuhaoZhang00 The full list of commands accepted by this bot can be found here. DetailsNeeds approval from an approver in each of these files:Approvers can indicate their approval by writing |
Signed-off-by: JmPotato <[email protected]>
ti-chi-bot
Bot
added
size/XL
Signed-off-by: JmPotato <[email protected]>
|
/test pull-unit-test-next-gen-2 |
Signed-off-by: JmPotato <[email protected]>
|
@JmPotato: The following tests failed, say
Full PR test history. Your PR dashboard. DetailsInstructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes-sigs/prow repository. I understand the commands that are listed here. |
| return h.updateMicroserviceConfig(cfg, kp[len(kp)-1], value) | ||
| case "controller": | ||
| return h.updateControllerConfig(kp[len(kp)-1], value) | ||
| case "lease": |
There was a problem hiding this comment.
Please consider version-gating this API until all PD members understand the persisted lease. During rolling upgrade, an old PD can still persist lease: 0 in the config blob, which makes the new code ignore the online value and fall back to local toml on the next campaign.
| // the online update API. Keep the accepted range aligned with file config | ||
| // behavior: reject explicit non-positive updates, but do not add an online-only | ||
| // upper bound. | ||
| func ValidateLeaderLease(lease int64) error { |
There was a problem hiding this comment.
Could we add an upper bound here? Values larger than math.MaxInt64 / int64(time.Second) can be persisted successfully, but time.Duration(lease) * time.Second overflows later in election keepalive and may break the next campaign.
| failpoint.Inject("loadLeaderLeaseFail", func() { | ||
| failpoint.Return(int64(0), errors.New("failpoint: fail to load persisted leader lease")) | ||
| }) | ||
| cfg := &persistedConfig{Config: &Config{}} |
There was a problem hiding this comment.
This still unmarshals the full persisted config on the election path. Could we decode only a small {lease} struct here? That avoids extra work and prevents unrelated config decode issues from changing campaign behavior.
3 participants
What problem does this PR solve?
Issue Number: ref #10630
Production clusters can see PD leader lease keepalive p99 latency approach the
5s default lease timeout, leaving little safety margin before leader churn.
Operators need a way to tune the PD leader lease online between releases without
editing the local config file and restarting PD.
What is changed and how does it work?
Implementation notes:
GET /pd/api/v1/configreturns the effective top-levelleasevalue.POST /pd/api/v1/configaccepts{"lease": <seconds>}and persists it through the existing config blob.Campaign, then falls back to the current in-memory value if the load fails. The existing full config reload after leadership is kept is unchanged.leasekeep the startup toml value; legacy non-positive persisted values are ignored.0and negative values because they are explicit invalid updates. File configlease = 0keeps the existing startup behavior and defaults to5throughAdjustInt64.Check List
Tests
Code changes
Side effects
Related changes
pingcap/docs/pingcap/docs-cn: TBDManual test
go test ./server/config -run 'TestLeaderLease|TestReloadKeepsCustomLeaderLeaseWhenPersistedConfigMissesLease|TestValidateLeaderLease|TestLoadPersistedLeaderLease' -count=1go test -tags without_dashboard ./tests/server/api -run 'TestLeaderLeaseConfigAPI|TestLeaderLeaseTakesEffectAfterTransfer|TestLeaderLeaseCampaignSurvivesLoadFailure' -count=1go vet ./server/configgofmt -l server/config/config.go server/config/config_test.go server/config/persist_options.go server/server.go tests/server/api/api_test.gogit diff --checkRelease note
Summary by CodeRabbit
New Features
Bug Fixes
Tests