ci: STEP 2: BitBake S3 cache as tar.zst with fingerprint skip#349
ci: STEP 2: BitBake S3 cache as tar.zst with fingerprint skip#349neo-jesse wants to merge 3 commits into
Conversation
Surface AWS errors, check object existence before download, install or fall back from zstd, and avoid silent cold builds when cache is missing. Co-authored-by: Cursor <cursoragent@cursor.com>
| ensure_tools() { | ||
| if command -v zstd >/dev/null 2>&1; then | ||
| ARCHIVE_FORMAT=zstd | ||
| log "Using zstd ($(command -v zstd))" | ||
| return 0 | ||
| fi | ||
|
|
||
| log "zstd not found; attempting install..." | ||
| if command -v dnf >/dev/null 2>&1; then | ||
| sudo dnf install -y zstd || true | ||
| elif command -v yum >/dev/null 2>&1; then | ||
| sudo yum install -y zstd || true | ||
| elif command -v apt-get >/dev/null 2>&1; then | ||
| sudo apt-get update -qq || true | ||
| sudo DEBIAN_FRONTEND=noninteractive apt-get install -y -qq zstd || true | ||
| fi | ||
|
|
||
| # Refresh command hash in case the package just landed on PATH. | ||
| hash -r 2>/dev/null || true | ||
|
|
||
| if command -v zstd >/dev/null 2>&1; then | ||
| ARCHIVE_FORMAT=zstd | ||
| log "Using zstd after install ($(command -v zstd))" | ||
| return 0 | ||
| fi | ||
|
|
||
| if command -v zip >/dev/null 2>&1 && command -v unzip >/dev/null 2>&1; then | ||
| ARCHIVE_FORMAT=zip | ||
| log "WARNING: zstd unavailable; falling back to zip for cache archives" | ||
| return 0 | ||
| fi | ||
|
|
||
| log "ERROR: neither zstd nor zip/unzip is available" >&2 | ||
| return 1 | ||
| } |
There was a problem hiding this comment.
I do not mind a simple check here but I don't like a catch all installer. I think we should understand if our ephemeral custom runners will or will not have zstd and reconcile them to have it where we control that reconciliation. From a quick check it looks like almost all standard github runners do have it, just FYI.
This check can then simply be print the version or fail if not present.
I also think we don't need any backwards compatibility. When this merges I expect no cache hit and new caches built and pushed according to this new pattern.
We want the code in this file as simple and straightforward as possible to make it clear to all maintainers exactly how we build, bust, write, and use the caches for this job.
Require zstd on the runner (print version or fail), drop zip/legacy fallback and installers, and document pull/push/fingerprint clearly. Co-authored-by: Cursor <cursoragent@cursor.com>
neo-jesse
changed the title
2 participants
Summary
.github/scripts/s3-bitbake-cache.shfordownloads/sstate/gitastar.zst(legacy zip fallback on pull).*.manifest.Test plan