s3: match the SymLink metadata key case-insensitively in ReadLink - #1625
Open
barnumbirr wants to merge 2 commits into
Open
s3: match the SymLink metadata key case-insensitively in ReadLink#1625barnumbirr wants to merge 2 commits into
barnumbirr wants to merge 2 commits into
Conversation
S3 lowercases user metadata keys. SymLink writes "SymLink" and ReadLink
looks up output.Metadata["SymLink"], but the value comes back under
"symlink", so the lookup misses and ReadLink returns an empty string with
a nil error. To the caller that reads as "this object is not a link"
rather than as a failure.
deb.packageIndexByHash uses ReadLink to find the physical file its .old
pointer refers to, and removes it before rotating:
linkTarget, err = publishedStorage.ReadLink(oldIndexPath)
if err == nil {
_ = publishedStorage.Remove(linkTarget)
}
With "" returned the Remove is a no-op and its error is discarded, so the
superseded by-hash index file is never deleted. The pointers rotate
correctly and the physical files accumulate: one per index file per hash
algorithm per publish, indefinitely. Filesystem backends are unaffected
because they have real symlinks.
Measured on a live R2-backed archive publishing three suites a few times
a day: dists/ held 711 objects, of which 672 were by-hash and only 39
were current indexes, growing by about 48 objects per publish since
Acquire-By-Hash was enabled.
The existing TestSymLink cannot catch this, because it ends in
c.Skip("copy not available in s3test") and never runs. The regression
test added here writes the metadata with PutObject instead, so it needs
no CopyObject support, and fails before this change with
obtained string = "".
4 tasks
Codecov Report✅ All modified and coverable lines are covered by tests. Additional details and impacted files@@ Coverage Diff @@
## master #1625 +/- ##
==========================================
- Coverage 77.37% 77.08% -0.30%
==========================================
Files 165 165
Lines 15747 15750 +3
==========================================
- Hits 12185 12141 -44
- Misses 2356 2408 +52
+ Partials 1206 1201 -5 ☔ View full report in Codecov by Harness. 🚀 New features to boost your workflow:
|
codecov/patch flagged the change at 75% of diff hit. The uncovered lines were the two the existing test could not reach: the fallthrough when no metadata key matches, and the HeadObject error return. Both are worth pinning regardless of the coverage gate. packageIndexByHash calls ReadLink on a path it has only just confirmed exists and treats an error as "leave it alone", so "this is not a link" and "I could not find out" have to stay distinguishable: the first is an empty string with a nil error, the second is an error. ReadLink goes to 100%.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Description of the Change
S3 lowercases user metadata keys.
SymLinkwrites"SymLink", butReadLinklooks up
output.Metadata["SymLink"]and the value comes back under"symlink"— so the lookup misses andReadLinkreturns an empty string with anil error. To the caller that reads as "this object is not a link" rather
than as a failure.
deb.packageIndexByHashusesReadLinkto find the physical file its.oldpointer refers to, and removes it before rotating:
With
""returned, theRemoveis a no-op and its error is discarded. Thepointers rotate correctly, so the bug is invisible in the published metadata —
but every superseded by-hash index file is left behind, one per index file per
hash algorithm per publish, indefinitely.
Filesystem backends are unaffected: they have real symlinks and never go
through this path.
Evidence
Measured on a live R2-backed archive publishing three suites a few times a day,
with
Acquire-By-Hashenabled on 2026-08-24:Only 39 index files are current. The other 672 are orphaned by-hash copies,
accruing at roughly 48 per publish. Reading the metadata back from R2 directly
shows the casing:
Why the existing test does not catch it
TestSymLinkround-tripsSymLink→ReadLink, but ends withc.Skip("copy not available in s3test"), so it never runs — the test server hasno
CopyObject. This path has therefore never been covered.The regression test added here writes the metadata with
PutObjectinstead, soit needs no
CopyObjectsupport. It fails before the change withobtained string = ""and passes after.Checklist
suite has no S3 backend; the unit test covers the decoding directly
go build ./s3/...,go vet ./s3/...,gofmt -l s3/andgo test ./s3/...all clean