Repository navigation
Refactor: Migrate optional in-memory context storage to pkg/cache - #262
Conversation
Back MemStore with the shared kubevela/pkg cache instead of a plain map. Entries are stored without expiry since this holds live workflow context. Also take the lock across GetOrCreateInMemoryContext so the lookup and create can't race, and use the same namespace defaulting for reads and writes. Signed-off-by: Nachiket <nachiketpatil2604@gmail.com>
There was a problem hiding this comment.
All reported issues were addressed across 2 files
Reply with feedback, questions, or to request a fix.
Fix all with cubic | Re-trigger cubic
Codecov Report❌ Patch coverage is
Additional details and impacted files@@ Coverage Diff @@
## main #262 +/- ##
==========================================
+ Coverage 62.49% 70.48% +7.99%
==========================================
Files 62 73 +11
Lines 4415 5852 +1437
==========================================
+ Hits 2759 4125 +1366
- Misses 1324 1341 +17
- Partials 332 386 +54
Flags with carried forward coverage won't be shown. Click here to find out more. ☔ View full report in Codecov by Harness. 🚀 New features to boost your workflow:
|
The previous concurrency test only checked that a key existed afterwards, so it would pass even if GetOrCreateInMemoryContext were not atomic. Its DeleteInMemoryContext calls also never matched the stored keys. Race workers on a single key and assert every caller ends up with the stored context. Signed-off-by: Nachiket <nachiketpatil2604@gmail.com>
|
Hi, CI is green and I tightened the concurrency test based on the bot review. Would someone have time to take a look? |
|
@briankane any chance you could review this? It moves the in-memory context store over to kubevela/pkg's cache (#254), and CI is green. |
briankane
left a comment
There was a problem hiding this comment.
Thanks for picking this up, and sorry it sat so long. It's a tidy change and does what #254 asked: no expiry on the entries, flag untouched, and moving off the bare map also gets rid of the unlocked read in GetInMemoryContext. Happy to approve. A few things worth a look, none of them blocking:
- The concurrency test only trips occasionally. I swapped the old get-then-create back in and it failed in roughly a handful of 300 runs, and never under
-race, which is how CI runs it. If you can force the interleaving (a test hook, or a slow create) it'd actually guard the lock; otherwise I'd keep it but not lean on it. - We store the caller's pointer, and
newContext/writeToStorekeep mutating it outside the lock, so a concurrentDeepCopyIntocan still race on the maps. Same as before this PR, but storingcm.DeepCopy()in Put/Update would make the locking mean what it says. - Not yours, but since you're in here:
DeleteInMemoryContextbuilds a key with no namespace, so it never matches anything, and nothing calls it anyway. Contexts pile up for the life of the process. I'll open a follow-up issue for it. - In
TestWorkflowContextInMemory,sync()still does acli.Patchwhen the flag is on, which would NotFound against a real API server. The mock returning nil hides that. Also one for a follow-up issue.
Description of your changes
Moves
MemStoreontogithub.com/kubevela/pkg/cache, same approach as the email change in #259. Entries go in with no expiry since this is live workflow context, and the exported methods stay the same because vela-core calls them.GetOrCreateInMemoryContextnow holds the lock for the whole lookup + create, and reads use the same namespace defaulting as writes (empty namespace ->default).Fixes #254
I have:
make reviewableto ensure this PR is ready for review. (fmt, vet, golangci-lint and staticcheck; no manifest or chart changes)backport release-x.ylabels to auto-backport this PR if necessary.How has this code been tested
Added
pkg/context/storage_test.go: create/get/update, empty namespace, get-or-create returning a copy, concurrent access, andNewContext->Commit->LoadContextwithEnableInMemoryContexton.go test -race ./pkg/context/...passes.Special notes for your reviewer
Two existing bugs I left out to keep this a straight refactor:
DeleteInMemoryContext(appName)builds the key without the namespace, so the delete vela-core does on app deletion never matches anything (details on Refactor: Migrate optional in-memory context storage to pkg/cache #254).sync()still falls through tocli.Patchafter updating the memory store. Looks like that came in with 9eda5ba.I can send follow-ups for either if you want.
Summary by cubic
Migrates in-memory workflow context storage from a plain map to
github.com/kubevela/pkg/cachewithout changing exported methods for vela-core callers. Entries are stored without expiry since they hold live workflow context. Fixes #254.GetOrCreateInMemoryContextnow holds the lock across the whole lookup and create, so concurrent calls can't race.default).pkg/context/storage_test.gocovering create/update/get, empty namespace handling, concurrent get-or-create asserting every caller ends up with the stored context, andNewContext/Commit/LoadContextwith in-memory storage enabled.DeleteInMemoryContextbuilds its key without the namespace, andsync()still falls through tocli.Patchwhen in-memory context is on. Can be sent as follow-ups.Written for commit 76bfa23. Summary will update on new commits.