feat: add range tombstones (delete_range / delete_prefix)

.github/copilot-instructions.md

@@ -62,10 +62,24 @@ cargo fmt --all -- --check                   # Format check
 | `bytes_1` | Use `bytes` crate for Slice type |
 | `metrics` | Expose prometheus metrics |
 
+## Design Decision Analysis (CRITICAL)
+
+**Before flagging any potential issue, trace the call chain visible in the PR diff:**
+
+1. **Read the caller** — does the caller already handle the case you're about to flag? If a function returns a value that looks problematic in isolation, check call sites visible in the diff. The caller may handle the edge case explicitly (e.g., empty collections, `None` vs `Some(empty)`).
+2. **Check type-level guarantees** — does the type system prevent the issue? Different enum variants, wrapper types, or visibility modifiers may make a "collision" or "misuse" structurally impossible.
+3. **Read adjacent comments** — comments starting with "NOTE:", "Use X instead of Y because...", or explaining WHY a specific approach was chosen document deliberate design decisions. If the comment accurately describes the code behavior, the design is intentional.
+
+**Only suppress Tier 3/Tier 4 findings if they fail these checks.** If the pattern still indicates a Tier 1 (logic/correctness) or Tier 2 (safety/crash) issue, flag it regardless of documented rationale or caller handling.
+
 ## Architecture Notes
 
 - `src/table/block/` — On-disk block format (header + compressed payload)
 - `src/vlog/blob_file/` — Value log for large values (separate from LSM blocks)
 - `src/compaction/` — Compaction strategies (leveled, FIFO, tiered)
 - `src/seqno.rs` — Sequence number generator (MVCC versioning)
+- `src/range_tombstone.rs` — Range tombstone data model and serialization
+- `src/range_tombstone_filter.rs` — MVCC-aware range tombstone filtering for iterators
+- `src/active_tombstone_set.rs` — Tracks active range tombstones during compaction
+- `src/memtable/interval_tree.rs` — Interval tree for memtable range tombstone queries
 - Compression is pluggable via `CompressionType` enum with `#[cfg(feature)]` variants

.github/instructions/rust.instructions.md

@@ -43,6 +43,10 @@ Focus review effort on real bugs, not cosmetics. Stop after finding issues in hi
 
 These are not actionable review findings. Do not raise them:
 
+- **Caller-handled edge cases**: Before flagging a function for not handling an edge case (empty collection, `None` vs `Some(empty)`, missing guard), check call sites visible in the PR diff. If all visible callers already handle the case, the function's behavior is part of a deliberate contract — not a bug. Only flag if the edge case is truly unhandled end-to-end within the scope of the PR.
+- **Type-system-prevented issues**: Before flagging a potential collision, overlap, or misuse, check whether distinct enum variants, wrapper types, or visibility modifiers make the issue structurally impossible. A `WeakTombstone` variant that never appears in user-facing merge paths cannot collide with user data regardless of key/seqno overlap.
+- **Documented design decisions** (Tier 3-4 only): When code has a comment explaining WHY a specific approach was chosen, trust the documented reasoning for style and API design choices. Flag only if the comment contradicts the actual code behavior — not if you would have chosen a different approach. This exclusion does NOT apply to Tier 1 (logic bugs, data corruption) or Tier 2 (safety, crash recovery) — always flag those regardless of documentation.
+
 - **Comment wording vs code behavior**: If a comment says "flush when full" but the threshold is checked with `>=` not `>`, the intent is clear — the boundary condition is a design choice. Do not suggest rewording comments to match exact comparison operators.
 - **Comment precision**: "returns the block" when it technically returns `Result<Block>` — the comment conveys meaning, not type signature.
 - **Magic numbers with context**: `4` in `assert_eq!(header.len(), 4, "expected u32 checksum")` — the assertion message provides the context. Do not suggest a named constant when the value is used once in a test with an explanatory message.

README.md

@@ -11,6 +11,11 @@
 > **Maintained fork** by [Structured World Foundation](https://sw.foundation) for the [CoordiNode](https://github.com/structured-world/coordinode) database engine.
 > Based on [fjall-rs/lsm-tree](https://github.com/fjall-rs/lsm-tree). We contribute patches upstream and maintain additional features needed for CoordiNode (zstd compression, custom sequence number generators, batch get, intra-L0 compaction, security hardening).
 
+> [!IMPORTANT]
+> This fork now introduces a fork-specific **disk format V4** compatibility boundary.
+> `V4` is a breaking on-disk change relative to `V3` because the fork persists new semantics such as range tombstones and merge operands.
+> New code may continue reading supported `V3` databases, but databases written with these `V4` semantics must not be opened by older `V3` binaries.
+
 A K.I.S.S. implementation of log-structured merge trees (LSM-trees/LSMTs) in Rust.
 
 > [!NOTE]

src/abstract_tree.rs

@@ -15,9 +15,22 @@ pub type RangeItem = crate::Result<KvPair>;
 
 type FlushToTablesResult = (Vec<Table>, Option<Vec<BlobFile>>);
 
+// Sealed on purpose: this trait is still public as a consumer-side bound
+// (`&impl AbstractTree`), but external implementations are no longer part of
+// the supported extension surface. Internal flush/version hooks keep evolving
+// with crate-owned tree types and must not create downstream semver traps.
+//
+// `sealed` stays `pub` only so sibling modules in this crate can write
+// `crate::abstract_tree::sealed::Sealed` in their impls. The parent module
+// `abstract_tree` is not publicly exported from the crate root, so downstream
+// crates still cannot name or implement this trait.
+pub mod sealed {
+    pub trait Sealed {}
+}
+
 /// Generic Tree API
 #[enum_dispatch::enum_dispatch]
-pub trait AbstractTree {
+pub trait AbstractTree: sealed::Sealed {
     /// Debug method for tracing the MVCC history of a key.
     #[doc(hidden)]
     fn print_trace(&self, key: &[u8]) -> crate::Result<()>;
@@ -76,7 +89,9 @@ pub trait AbstractTree {
         _lock: &MutexGuard<'_, ()>,
         seqno_threshold: SeqNo,
     ) -> crate::Result<Option<u64>> {
-        use crate::{compaction::stream::CompactionStream, merge::Merger};
+        use crate::{
+            compaction::stream::CompactionStream, merge::Merger, range_tombstone::RangeTombstone,
+        };
 
         let version_history = self.get_version_history_lock();
         let latest = version_history.latest_version();
@@ -93,6 +108,14 @@ pub trait AbstractTree {
 
         let flushed_size = latest.sealed_memtables.iter().map(|mt| mt.size()).sum();
 
+        // Collect range tombstones from sealed memtables
+        let mut range_tombstones: Vec<RangeTombstone> = Vec::new();
+        for mt in latest.sealed_memtables.iter() {
+            range_tombstones.extend(mt.range_tombstones_sorted());
+        }
+        range_tombstones.sort();
+        range_tombstones.dedup();
+
         let merger = Merger::new(
             latest
                 .sealed_memtables
@@ -104,7 +127,22 @@ pub trait AbstractTree {
 
         drop(version_history);
 
-        if let Some((tables, blob_files)) = self.flush_to_tables(stream)? {
+        // Clone needed: flush_to_tables_with_rt consumes the Vec, but on the
+        // RT-only path (no KV data, tables.is_empty()) we re-insert RTs into the
+        // active memtable. Flush is infrequent and RT count is small.
+        if let Some((tables, blob_files)) =
+            self.flush_to_tables_with_rt(stream, range_tombstones.clone())?
+        {
+            // If no tables were produced (RT-only memtable), re-insert RTs
+            // into active memtable so they aren't lost
+            if tables.is_empty() && !range_tombstones.is_empty() {
+                let active = self.active_memtable();
+                for rt in &range_tombstones {
+                    let _ =
+                        active.insert_range_tombstone(rt.start.clone(), rt.end.clone(), rt.seqno);
+                }
+            }
+
             self.register_tables(
                 &tables,
                 blob_files.as_deref(),
@@ -216,10 +254,26 @@ pub trait AbstractTree {
     /// # Errors
     ///
     /// Will return `Err` if an IO error occurs.
-    #[warn(clippy::type_complexity)]
     fn flush_to_tables(
         &self,
         stream: impl Iterator<Item = crate::Result<InternalValue>>,
+    ) -> crate::Result<Option<FlushToTablesResult>> {
+        self.flush_to_tables_with_rt(stream, Vec::new())
+    }
+
+    /// Like [`AbstractTree::flush_to_tables`], but also writes range tombstones.
+    ///
+    /// This is an internal extension hook on the crate's sealed tree types and
+    /// is hidden from generated documentation.
+    ///
+    /// # Errors
+    ///
+    /// Will return `Err` if an IO error occurs.
+    #[doc(hidden)]
+    fn flush_to_tables_with_rt(
+        &self,
+        stream: impl Iterator<Item = crate::Result<InternalValue>>,
+        range_tombstones: Vec<crate::range_tombstone::RangeTombstone>,
     ) -> crate::Result<Option<FlushToTablesResult>>;
 
     /// Atomically registers flushed tables into the tree, removing their associated sealed memtables.
@@ -680,4 +734,36 @@ pub trait AbstractTree {
     /// Will return `Err` if an IO error occurs.
     #[doc(hidden)]
     fn remove_weak<K: Into<UserKey>>(&self, key: K, seqno: SeqNo) -> (u64, u64);
+
+    /// Deletes all keys in the range `[start, end)` by inserting a range tombstone.
+    ///
+    /// This is much more efficient than deleting keys individually when
+    /// removing a contiguous range of keys.
+    ///
+    /// Returns the approximate size added to the memtable.
+    /// Returns 0 if `start >= end` (invalid interval is silently ignored).
+    ///
+    /// This is a required method on the crate's sealed tree types.
+    fn remove_range<K: Into<UserKey>>(&self, start: K, end: K, seqno: SeqNo) -> u64;
+
+    /// Deletes all keys with the given prefix by inserting a range tombstone.
+    ///
+    /// This is sugar over [`AbstractTree::remove_range`] using prefix bounds.
+    ///
+    /// Returns the approximate size added to the memtable.
+    /// Returns 0 for empty prefixes or all-`0xFF` prefixes (cannot form valid half-open range).
+    fn remove_prefix<K: AsRef<[u8]>>(&self, prefix: K, seqno: SeqNo) -> u64 {
+        use crate::range::prefix_to_range;
+        use std::ops::Bound;
+
+        let (lo, hi) = prefix_to_range(prefix.as_ref());
+
+        let Bound::Included(start) = lo else { return 0 };
+
+        // Bound::Unbounded means the prefix is all 0xFF — no representable
+        // exclusive upper bound exists, so we cannot form a valid range tombstone.
+        let Bound::Excluded(end) = hi else { return 0 };
+
+        self.remove_range(start, end, seqno)
+    }
 }