feat: add verify_integrity() for full-file checksum verification

.github/copilot-instructions.md

@@ -0,0 +1,70 @@
+# GitHub Copilot Instructions for lsm-tree (structured-world fork)
+
+## Project Overview
+
+This is a **maintained fork** of [fjall-rs/lsm-tree](https://github.com/fjall-rs/lsm-tree) — a K.I.S.S. LSM-tree implementation in Rust. We maintain additional features and hardening for the [CoordiNode](https://github.com/structured-world/coordinode) database engine while contributing patches upstream.
+
+## Review Scope Rules (CRITICAL)
+
+**Review ONLY code within the PR's diff.** For issues found in code outside the diff:
+- Do NOT suggest inline code fixes for unchanged lines
+- Instead, suggest creating a **separate issue** with the finding (e.g., "Consider opening an issue to add size validation in `from_reader` — this is outside the scope of this clippy-fix PR")
+
+**Each PR has a defined scope in its description.** Read the "out of scope" section before reviewing. If something is listed as out of scope, do not flag it — it is tracked in another PR.
+
+**Cross-PR awareness:** This fork has multiple feature branches in parallel. If a hardening or feature seems missing, check whether it exists in another open PR before suggesting it. Reference the other PR number if known.
+
+**Prefer issue suggestions over code suggestions for out-of-scope findings.** This keeps PRs focused and reviewable.
+
+## Rust Code Standards
+
+- **Unsafe code:** Prefer safe alternatives. If `unsafe` is required, it must have a `// SAFETY:` comment explaining the invariant.
+- **Error handling:** No `unwrap()` or `expect()` on I/O paths. Use `Result<T, E>` propagation. `expect()` is acceptable for programmer invariants (e.g., lock poisoning) with `#[expect(clippy::expect_used, reason = "...")]`.
+- **Clippy:** Code must pass `cargo clippy --all-features -- -D warnings`. Use `#[expect(...)]` (not `#[allow(...)]`) for justified suppressions — `#[expect]` warns if the suppression becomes unnecessary.
+- **Casts:** Prefer `TryFrom`/`TryInto` for fallible conversions. `as` casts are acceptable for infallible cases (e.g., `u32` to `u64`) with `#[expect(clippy::cast_possible_truncation)]` and a reason.
+- **Feature gates:** Code behind `#[cfg(feature = "...")]` must compile with any combination of features. Variables used only in feature-gated branches must also be feature-gated.
+
+## Testing Standards
+
+- **Corruption tests:** When adding validation for on-disk data, add a test that tampers the relevant field and asserts the error. Use the same serialization path as production (e.g., `lz4_flex::compress` not `compress_prepend_size`).
+- **No mocks for storage:** Tests use real on-disk files via `tempfile::tempdir()`.
+- **Test naming:** `fn <what>_<condition>_<expected>()` — e.g., `fn lz4_corrupted_header_triggers_decompress_error()`.
+
+## Commit Message Format
+
+```
+<type>(scope): <description>
+
+- Detail 1
+- Detail 2
+```
+
+Types: `feat`, `fix`, `refactor`, `test`, `docs`, `style`, `chore`, `perf`, `ci`, `build`, `revert`
+
+**Forbidden patterns:** "address review", "fix PR comments", "WIP", "temporary"
+
+## Build and Test
+
+```bash
+cargo clippy --all-features -- -D warnings  # Lint (strict)
+cargo test --features lz4                    # Tests with lz4
+cargo test --all-features                    # Tests with all features
+cargo fmt --all -- --check                   # Format check
+```
+
+## Feature Flags
+
+| Flag | Description |
+|------|-------------|
+| `lz4` | LZ4 compression (enabled by default in fjall) |
+| `zstd` | Zstd compression (PR #1) |
+| `bytes_1` | Use `bytes` crate for Slice type |
+| `metrics` | Expose prometheus metrics |
+
+## Architecture Notes
+
+- `src/table/block/` — On-disk block format (header + compressed payload)
+- `src/vlog/blob_file/` — Value log for large values (separate from LSM blocks)
+- `src/compaction/` — Compaction strategies (leveled, FIFO, tiered)
+- `src/seqno.rs` — Sequence number generator (MVCC versioning)
+- Compression is pluggable via `CompressionType` enum with `#[cfg(feature)]` variants

.github/dependabot.yml

@@ -0,0 +1,22 @@
+version: 2
+updates:
+  - package-ecosystem: "cargo"
+    directory: "/"
+    schedule:
+      interval: "weekly"
+    open-pull-requests-limit: 10
+    groups:
+      minor-and-patch:
+        patterns:
+          - "*"
+        update-types:
+          - "minor"
+          - "patch"
+    commit-message:
+      prefix: "chore(deps)"
+  - package-ecosystem: "github-actions"
+    directory: "/"
+    schedule:
+      interval: "weekly"
+    commit-message:
+      prefix: "ci(deps)"

.github/instructions/code-review.instructions.md

@@ -0,0 +1,27 @@
+---
+applyTo: "**/*.rs"
+---
+
+# Code Review Instructions for lsm-tree (structured-world fork)
+
+## Scope Rules (CRITICAL)
+
+- **Review ONLY code within the PR's diff.** Do not suggest inline fixes for unchanged lines.
+- For issues in code **outside the diff**, suggest creating a **separate issue** instead of proposing code changes. Example: "Consider opening an issue to add size validation here — this is outside this PR's scope."
+- **Read the PR description carefully.** If the PR body has an "out of scope" section listing items handled by other PRs, do NOT flag those items.
+- This fork has **multiple feature branches in parallel**. A hardening that seems missing in one PR may already exist in another open PR. Check the "out of scope" section for cross-references.
+
+## Rust Standards
+
+- `unsafe` blocks require `// SAFETY:` comments explaining the invariant
+- Prefer `#[expect(lint)]` over `#[allow(lint)]` — `#[expect]` warns when suppression becomes unnecessary
+- Use `TryFrom`/`TryInto` for fallible conversions; `as` casts need `#[expect(clippy::cast_possible_truncation)]` with reason
+- No `unwrap()` / `expect()` on I/O paths — use `Result` propagation
+- `expect()` is acceptable for programmer invariants (lock poisoning) with `#[expect(clippy::expect_used, reason = "...")]`
+- Code must pass `cargo clippy --all-features -- -D warnings`
+
+## Testing
+
+- Corruption/validation tests: tamper the relevant on-disk field and assert the error
+- Use same serialization as production (e.g., `lz4_flex::compress` not `compress_prepend_size`)
+- Test naming: `fn <what>_<condition>_<expected>()`

.github/workflows/coordinode-ci.yml

@@ -0,0 +1,92 @@
+name: CoordiNode CI
+
+on:
+  push:
+    branches:
+      - main
+      - "feat/#*"
+      - "fix/#*"
+  pull_request:
+    branches:
+      - main
+
+env:
+  CARGO_TERM_COLOR: always
+
+jobs:
+  lint:
+    # Fast gate — runs on every push and PR
+    timeout-minutes: 10
+    runs-on: ubuntu-latest
+    steps:
+      - uses: actions/checkout@v6
+      - uses: dtolnay/rust-toolchain@stable
+        with:
+          components: rustfmt, clippy
+      - uses: Swatinem/rust-cache@v2
+      - name: Format
+        run: cargo fmt --all -- --check
+      - name: Clippy (strict)
+        run: cargo clippy --all-features -- -D warnings
+
+  test:
+    # Full matrix — only on push to main/feature branches (not on PRs)
+    if: github.event_name == 'push'
+    needs: lint
+    timeout-minutes: 20
+    strategy:
+      matrix:
+        rust_version: [stable, "1.90.0"]
+        os: [ubuntu-latest, windows-latest, macos-latest]
+    runs-on: ${{ matrix.os }}
+    steps:
+      - uses: actions/checkout@v6
+      - uses: dtolnay/rust-toolchain@stable
+        with:
+          toolchain: ${{ matrix.rust_version }}
+      - uses: Swatinem/rust-cache@v2
+        with:
+          prefix-key: ${{ runner.os }}-cargo
+      - uses: taiki-e/install-action@nextest
+      - name: Run tests
+        run: cargo nextest run --all-features
+      - name: Run doc tests
+        run: cargo test --doc --features lz4
+
+  test-pr:
+    # Lightweight — only on PRs (ubuntu stable)
+    if: github.event_name == 'pull_request'
+    needs: lint
+    timeout-minutes: 15
+    runs-on: ubuntu-latest
+    steps:
+      - uses: actions/checkout@v6
+      - uses: dtolnay/rust-toolchain@stable
+      - uses: Swatinem/rust-cache@v2
+      - uses: taiki-e/install-action@nextest
+      - name: Run tests
+        run: cargo nextest run --all-features
+      - name: Run doc tests
+        run: cargo test --doc --features lz4
+
+  codecov:
+    if: github.event_name == 'push'
+    needs: lint
+    timeout-minutes: 20
+    runs-on: ubuntu-latest
+    steps:
+      - uses: actions/checkout@v6
+      - uses: dtolnay/rust-toolchain@nightly
+        with:
+          components: llvm-tools-preview
+      - uses: Swatinem/rust-cache@v2
+      - uses: taiki-e/install-action@cargo-llvm-cov
+      - uses: taiki-e/install-action@nextest
+      - run: cargo llvm-cov --no-report nextest --all-features
+      - run: cargo llvm-cov --no-report --doc --features lz4
+      - run: cargo llvm-cov report --doctests --lcov --output-path lcov.info
+      - uses: codecov/codecov-action@v5
+        with:
+          files: lcov.info
+        env:
+          CODECOV_TOKEN: ${{ secrets.CODECOV_TOKEN }}

.github/workflows/coordinode-release.yml

@@ -0,0 +1,31 @@
+name: Release
+
+on:
+  push:
+    branches:
+      - main
+
+permissions:
+  contents: write
+  pull-requests: write
+
+jobs:
+  release-plz:
+    runs-on: ubuntu-latest
+    steps:
+      - name: Checkout
+        uses: actions/checkout@v6
+        with:
+          fetch-depth: 0
+
+      - name: Install Rust toolchain
+        uses: dtolnay/rust-toolchain@stable
+
+      - name: Run release-plz
+        uses: release-plz/[email protected]
+        with:
+          command: release-pr
+          config: .release-plz.toml
+        env:
+          GITHUB_TOKEN: ${{ secrets.GITHUB_TOKEN }}
+          CARGO_REGISTRY_TOKEN: ${{ secrets.CARGO_REGISTRY_TOKEN }}

.github/workflows/upstream-monitor.yml

@@ -0,0 +1,130 @@
+name: Upstream Monitor
+
+on:
+  schedule:
+    - cron: "0 8 * * 1,4"
+  workflow_dispatch:
+
+permissions:
+  contents: write
+  pull-requests: write
+
+jobs:
+  check-upstream:
+    timeout-minutes: 10
+    runs-on: ubuntu-latest
+    steps:
+      - name: Checkout
+        uses: actions/checkout@v6
+        with:
+          fetch-depth: 0
+
+      - name: Add upstream remote
+        run: git remote add upstream https://github.com/fjall-rs/lsm-tree.git
+
+      - name: Fetch upstream and origin
+        run: |
+          git fetch upstream main
+          git fetch origin main
+
+      - name: Check for new upstream commits
+        id: check
+        run: |
+          BEHIND=$(git rev-list origin/main..upstream/main --count)
+          echo "behind=$BEHIND" >> "$GITHUB_OUTPUT"
+          echo "Commits behind upstream: $BEHIND"
+
+      - name: Try merge and create PR or issue
+        if: steps.check.outputs.behind > 0
+        env:
+          GH_TOKEN: ${{ secrets.GITHUB_TOKEN }}
+          BEHIND: ${{ steps.check.outputs.behind }}
+        run: |
+          git config user.name "github-actions[bot]"
+          git config user.email "github-actions[bot]@users.noreply.github.com"
+
+          SYNC_BRANCH="chore/upstream-sync-$(date +%Y%m%d)"
+          git checkout -b "$SYNC_BRANCH" origin/main
+
+          if git merge --no-commit --no-ff upstream/main 2>&1; then
+            git commit -m "chore: sync upstream ($BEHIND new commits)"
+            git push origin "$SYNC_BRANCH"
+
+            gh pr create \
+              --title "chore: sync upstream ($BEHIND new commits)" \
+              --body "$(cat <<'EOF'
+          ## Upstream Sync
+
+          Automated sync from [fjall-rs/lsm-tree](https://github.com/fjall-rs/lsm-tree) main branch.
+
+          **Commits behind:** ${{ steps.check.outputs.behind }}
+
+          ### Review checklist
+          - [ ] Review upstream changes for breaking modifications
+          - [ ] Verify our patches still apply cleanly
+          - [ ] Run full test suite
+          EOF
+          )" \
+              --base main \
+              --head "$SYNC_BRANCH"
+          else
+            CONFLICTS=$(git diff --name-only --diff-filter=U 2>/dev/null || true)
+            git merge --abort
+
+            gh issue create \
+              --title "Upstream sync conflict ($BEHIND new commits)" \
+              --body "$(cat <<EOF
+          ## Upstream Sync Conflict
+
+          Automated merge from [fjall-rs/lsm-tree](https://github.com/fjall-rs/lsm-tree) main branch failed due to conflicts.
+
+          **Commits behind:** $BEHIND
+
+          ### Conflicting files
+          \`\`\`
+          $CONFLICTS
+          \`\`\`
+
+          ### Resolution
+          Manual merge required:
+          \`\`\`bash
+          git remote add upstream https://github.com/fjall-rs/lsm-tree.git
+          git fetch upstream main
+          git checkout -b chore/upstream-sync main
+          git merge upstream/main
+          # Resolve conflicts, then push
+          \`\`\`
+          EOF
+          )" \
+              --label "upstream-sync"
+          fi
+
+      - name: Check for merged feature branches
+        env:
+          GH_TOKEN: ${{ secrets.GITHUB_TOKEN }}
+        run: |
+          git fetch upstream main
+
+          for ref in $(git for-each-ref --format='%(refname:short)' refs/remotes/origin/feat/#* refs/remotes/origin/fix/#*); do
+            BRANCH="${ref#origin/}"
+            BRANCH_TIP=$(git rev-parse "$ref")
+
+            if git merge-base --is-ancestor "$BRANCH_TIP" upstream/main 2>/dev/null; then
+              echo "Branch '$BRANCH' is fully merged into upstream/main"
+
+              EXISTING=$(gh issue list --search "Upstream merged: $BRANCH" --state open --json number --jq 'length')
+              if [ "$EXISTING" = "0" ]; then
+                gh issue create \
+                  --title "Upstream merged: $BRANCH" \
+                  --body "$(cat <<EOF
+          Branch \`$BRANCH\` has been fully merged into upstream's main branch.
+
+          This branch can likely be deleted. Please verify and clean up.
+
+          **Branch tip:** \`$BRANCH_TIP\`
+          EOF
+          )" \
+                  --label "upstream-sync"
+              fi
+            fi
+          done

.release-plz.toml

@@ -0,0 +1,7 @@
+[workspace]
+# Don't publish to crates.io — we're a maintained fork using git dependencies
+publish = false
+# Create GitHub releases with changelog
+git_release_enable = true
+# Use conventional commits for changelog
+changelog_update = true