[jmarrec]

Does array alignment matter, with and without native optimizations + fast math?

Context: EnergyPlus builds ObjexxFCL arrays with OBJEXXFCL_ALIGN=64 (first element aligned to 64 bytes), a setting from ObjexxFCL 4.1 (NatLabRockies/EnergyPlus#5395) meant to help vectorization, at a time when -Ofast was also proposed (and rejected). Replacing ObjexxFCL arrays by std::vector / std::array drops that to the default allocator alignment. And NatLabRockies/EnergyPlus#11816 adds an opt-in -march/-mcpu=native + -ffast-math build, which is when vectorization (and so maybe alignment) can actually kick in.

What's in here

• bench_alignment_vectorization.cpp: five EnergyPlus-like kernels (Sum, Dot, Axpy, Cubic curve evaluation, 5-point Stencil on a row-major grid) taking a std::span (so the compiler cannot know the alignment, like EnergyPlus going through Array::operator()), plus std::assume_aligned<64> variants of Sum and Axpy.
  • Layouts: Default (std::vector<double>), Aligned64 (like OBJEXXFCL_ALIGN=64), Aligned128 (Apple Silicon cache line), and Offset8 / Offset16 / Offset32 (a 128-byte aligned buffer viewed from +8/+16/+32 bytes, so the alignment does not depend on the allocator; Offset16 is what std::vector<double> gets on glibc/MSVC).
  • Sizes: 37, 251, 1003, 30011, 4000037 doubles. Deliberately not powers of two (those hid vector-loop tails and caused cache set aliasing) and mostly small, like real EnergyPlus arrays.
  • Per-buffer *_offset_mod_128 counters report the alignment each buffer actually got.
• CMakeLists.txt: the file is built twice:
  • bench_alignment_vectorization: EnergyPlus-like default flags -O3 -ffp-contract=off
  • bench_alignment_vectorization_native: -O3 -march=native (-mcpu=native on ARM) -ffast-math -fno-finite-math-only, like Fix #5451 - Allow enabling compiler native optimizations and fast math  NatLabRockies/EnergyPlus#11816
• compare_alignment_results.py: Markdown tables relative to Aligned64 from one or two (default, native) result files, with --offsets to list observed buffer alignments.

Products/bench_alignment_vectorization        --benchmark_repetitions=5 --benchmark_report_aggregates_only=true --benchmark_out=../results/alignment_default.json --benchmark_out_format=json
Products/bench_alignment_vectorization_native --benchmark_repetitions=5 --benchmark_report_aggregates_only=true --benchmark_out=../results/alignment_native.json --benchmark_out_format=json
python3 compare_alignment_results.py results/alignment_default.json results/alignment_native.json --offsets

A full run takes ~10 min per executable; --benchmark_repetitions=3 --benchmark_min_time=0.1s is enough for a quick look.

Results

Two machines: macOS M1 Max (Apple clang, NEON 16-byte vectors) and Linux Ryzen 7 1800X (Zen 1, GCC, AVX2 32-byte vectors). Median of 5, max coefficient of variation 8.7% (M1) / 5.2% (Ryzen).

What the allocator actually gives std::vector<double>

• __STDCPP_DEFAULT_NEW_ALIGNMENT__ is 16 on Apple clang, GCC and MSVC: that's all std::allocator<double> guarantees.
• glibc (Ryzen): 16-byte aligned at all sizes (observed offsets 0/16/32/48/80/96/112 mod 128).
• macOS: 16 bytes for small blocks, page-aligned (16 KiB pages) for blocks >= ~32 KiB, so on a Mac Default is often "luckily" as aligned as Aligned128, hence the explicit OffsetN layouts.

Alignment, % vs Aligned64 (ie OBJEXXFCL_ALIGN=64)

Layout	M1 Max	Ryzen 1800X
Default / Aligned128	within noise	within noise
Offset16 (glibc/MSVC std::vector)	within noise, both builds	within noise, both builds (*)
Offset32 (AVX2-aligned, not 64)	within noise	within noise (*)
Offset8, Axpy, default / native build	+12 to +16% / +11 to +22% (n 251-30011)	+17 to +28% / +14 to +34% (n 37-30011)
Offset8, Cubic	up to +8%	+6 to +10% (native)

(*) Two tiny-size anomalies on the Ryzen that are not alignment driven: Cubic native n=37 is +60/+70% for Offset16 and Default, but also +58% for Aligned128 and only +1% for Offset8; Stencil default n=37 Offset32 +17%. Most likely relative placement of input/output buffers (store/load address aliasing), not 16 vs 64.

Native + fast math speedup vs default build (Aligned64)

Kernel	M1 Max	Ryzen 1800X
Sum (n >= 251)	7.2-8.5x (1.1x at n=37)	2.0x (1.8x at 4M, 1.5x at n=37)
Dot (n >= 251)	4.5-6.3x (1.7x at n=37)	2.0x (1.5x at 4M, 1.9x at n=37)
Cubic (in cache)	1.26-1.38x	1.41-1.48x
Axpy	1.0-1.07x	1.0-1.08x
Stencil	1.0x	0.89-0.97x (slower at mid sizes)

std::assume_aligned<64>

No effect on the M1. On the Ryzen it helps Sum in the native build (-27% at n=37, -11% at 251, -8% at 4M: GCC drops the alignment prologue), but the hinted Axpy is 30-35% slower in the default build (n 37-1003). EnergyPlus never used such hints (ASSUME_ALIGNED_OBJEXXFCL is unused), so OBJEXXFCL_ALIGN=64 never got that benefit.

Conclusions

• Going from ObjexxFCL's 64-byte alignment to std::vector's 16 bytes costs nothing measurable on either machine, in default or native builds. Alignment is not a reason to keep ObjexxFCL arrays.
• Only misalignment within an element (Offset8) costs, more so on Zen 1 AVX2, but no allocator produces that for double.
• Native + fast math gains depend heavily on the CPU (reductions 7-8x on the M1, 2x on Zen 1, which splits 256-bit ops in two) and on size (little at tens of elements, which is typical of EnergyPlus arrays). This supports keeping Fix #5451 - Allow enabling compiler native optimizations and fast math  NatLabRockies/EnergyPlus#11816 opt-in.

Excluded outliers, and the first (power-of-two) run

• Excluded (single block, not reproduced in the other build): M1 default Stencil Aligned128 4M +146% (buffer placement: Aligned64/Default have the same alignment at that size); M1 default Axpy Aligned64 4M reference 15% slow (cv 5.5%; AxpyAssumeAligned64 on the same layout was normal).
• The first commit used power-of-two sizes (1 Ki / 32 Ki / 4 Mi) and a single Misaligned8 layout. Same conclusions on the M1, but Default was page-aligned or lucky on macOS (so it never tested the 16-byte case), and Stencil showed a -27% "misaligned is faster" artifact at 4 Mi from cache set aliasing between the power-of-two buffers, which disappears with non-power-of-two sizes.

🤖 Generated with Claude Code