Skip to content

Commit 1bc4970

Browse files
test(index): add the A/B script the builder benchmark was quoted from
CONTRIBUTING asks for benchmark numbers reproducible from the branch with a command anyone can run. The three-genome table in the PR was produced by hand, so nobody else could rerun it. The script drives the two arms through --limitGenomeGenerateRAM, which is the flag that actually selects the builder, so the comparison exercises the dispatch rather than bypassing it. It reports wall clock and peak RSS and then compares SA and SAindex with cmp, failing if they differ. It refuses to run above a load average of 2. A genomeGenerate run is minutes long and uses every core, so an unrelated job does not add noise evenly: it lands on whichever arm overlaps it, and nothing in the resulting numbers shows that it happened. I made exactly that mistake benchmarking the BAM backends on this machine today.
1 parent aaa7b69 commit 1bc4970

1 file changed

Lines changed: 53 additions & 0 deletions

File tree

‎test/bench_sa_builders.sh‎

Lines changed: 53 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,53 @@
1+
#!/usr/bin/env bash
2+
# Compare the two suffix-array builders on one genome: wall clock, peak RSS,
3+
# and whether the SA and SAindex they write are byte-identical.
4+
#
5+
# test/bench_sa_builders.sh <genome.fa> <outdir> [saIndexNbases] [threads]
6+
#
7+
# Which builder runs is decided by --limitGenomeGenerateRAM, so the two arms
8+
# differ only in that flag: 0 means "no limit" and always picks libsais, and a
9+
# limit small enough that the estimate never fits always picks caps-sa.
10+
set -euo pipefail
11+
12+
FASTA=${1:?usage: bench_sa_builders.sh <genome.fa> <outdir> [saIndexNbases] [threads]}
13+
OUT=${2:?usage: bench_sa_builders.sh <genome.fa> <outdir> [saIndexNbases] [threads]}
14+
NBASES=${3:-14}
15+
THREADS=${4:-8}
16+
17+
ROOT="$(cd "$(dirname "${BASH_SOURCE[0]}")/.." && pwd)"
18+
BIN="$ROOT/target/release/rustar-aligner"
19+
20+
# A genomeGenerate run is minutes long and uses every core, so an unrelated job
21+
# does not merely add noise: it lands squarely on the arm unlucky enough to
22+
# overlap it, and nothing in the resulting numbers shows that it happened.
23+
LOAD=$(uptime | sed -E 's/.*load averages?: *([0-9.]+).*/\1/')
24+
if awk -v l="$LOAD" 'BEGIN{exit !(l > 2.0)}'; then
25+
echo "load average is $LOAD; other work is running and the numbers would not mean anything." >&2
26+
echo "wait for the machine to be idle, or set BENCH_IGNORE_LOAD=1 to override." >&2
27+
[ "${BENCH_IGNORE_LOAD:-0}" = "1" ] || exit 1
28+
fi
29+
30+
cargo build --release --manifest-path "$ROOT/Cargo.toml" >&2
31+
mkdir -p "$OUT"
32+
33+
arm() { # $1 = label, $2 = --limitGenomeGenerateRAM value
34+
local dir="$OUT/$1"
35+
rm -rf "$dir"
36+
mkdir -p "$dir"
37+
/usr/bin/time -l "$BIN" --runMode genomeGenerate --runThreadN "$THREADS" \
38+
--genomeDir "$dir" --genomeFastaFiles "$FASTA" \
39+
--genomeSAindexNbases "$NBASES" --limitGenomeGenerateRAM "$2" \
40+
> "$OUT/$1.log" 2> "$OUT/$1.time"
41+
local wall rss
42+
wall=$(grep ' real' "$OUT/$1.time" | awk '{print $1}')
43+
rss=$(grep 'maximum resident set size' "$OUT/$1.time" | awk '{print $1}')
44+
printf "%-8s wall=%ss peakRSS=%.2f GB (%s)\n" "$1" "$wall" \
45+
"$(echo "$rss/1073741824" | bc -l)" \
46+
"$(grep -o 'Building suffix array with [a-z-]*' "$OUT/$1.time" | head -1)"
47+
}
48+
49+
arm libsais 0
50+
arm capssa 1000000
51+
52+
if cmp -s "$OUT/libsais/SA" "$OUT/capssa/SA"; then echo " SA identical"; else echo " SA DIFFERS"; exit 1; fi
53+
if cmp -s "$OUT/libsais/SAindex" "$OUT/capssa/SAindex"; then echo " SAindex identical"; else echo " SAindex DIFFERS"; exit 1; fi

0 commit comments

Comments
 (0)