Fix trailing guard elimination for aligned large allocations

[t-h-i-n-k-er]

allocate_pages_aligned() laid out the mapping as [guard][lead_pad][usable][trail_pad][guard] and the trail unmap removed trail_size bytes starting at base+usable_size+guard_size. That address falls within the trailing guard rather than after it: when alignment exceeds guard_size, trail_size > guard_size and the munmap carves trail_size bytes out of the trailing guard region, fully removing the original guard page and replacing it with alignment padding that happens to be PROT_NONE. The guard_size stored in the region metadata no longer describes a real guard region. deallocate_pages() and the quarantine both assume [guard][usable][guard] with guard_size bytes on each side, but the trailing guard was eliminated and what sits after the usable area is just leftover alignment padding from the initial PROT_NONE mapping.

Change:
Restructure allocate_pages_aligned() so the trailing guard sits directly after the usable area with alignment padding beyond it: [guard][lead_pad][usable][guard][trail_pad]. When trail_size is nonzero, the old trail region (trail_size + guard_size bytes starting at base+usable_size) is unmapped and just the guard (guard_size bytes) is remapped at base+usable_size. The alignment padding beyond the guard is discarded. The total mapping size is unchanged, and the final layout is exactly [guard][usable][guard] with guard_size bytes of PROT_NONE on each side, which is what deallocate_pages() and the quarantine expect.

The mprotect of the usable area is moved after the unmap/remap so that the RW VMA is established after the VMA layout is finalized, avoiding a window where the usable pages sit between PROT_NONE fragments from the remap.

Why not the old approach:
The old code assumed that removing trail_size bytes from base+usable_size+guard_size trims the "end" of the mapping, but the trailing guard does not start at base+usable_size, it starts at base+usable_size+trail_size. So the munmap target is inside the guard when guard_size > trail_size (the common case for the default config) and inside the trail padding when guard_size <= trail_size (large alignment, small guard). In both cases the total PROT_NONE after the usable area happens to be guard_size bytes, which is why the bug is not immediately visible the guard region is functionally replaced by padding that is also PROT_NONE. However the guard_size in the region metadata is wrong: it describes a guard page that no longer exists at the recorded position, and any code that reasons about the guard's location (e.g. the in-place realloc shrink path, which installs a new guard at old+size and quarantines old+size+old_guard_size) operates on an incorrect layout.

RSS and VmSize are unchanged. VMA count goes from ~2.9 to exactly 3.0 per allocation (leading guard, usable, trailing guard), because the guard is no longer split by the trail unmap. Under CONFIG_LABEL_MEMORY the improvement is larger: the old code's trail unmap could split the trailing guard VMA, forcing prctl(PR_SET_VMA_ANON_NAME) to operate on a fragment; the new code produces a single clean guard VMA.

Verification
Builds clean under gcc -Werror, default and light configs, with and without CONFIG_LABEL_MEMORY. Test suite passes all 51 tests in every configuration.

Guard pages fault on overflow and underflow for aligned large allocations with alignments from 8 KiB to 2 MiB, verified by forking and accessing the page before and after the usable area. deallocate_pages() correctly unmaps the full [guard][usable][guard] range. The quarantine path correctly remaps the usable area as PROT_NONE and later unmaps the full guard-bounded region.

The memory_map_fixed for the remapped guard is MAP_FIXED|MAP_ANONYMOUS|MAP_PRIVATE with PROT_NONE, which is the same combination used by the existing memory_map_fixed() for quarantine, so kernel behavior is well-tested.

[thestinger]

Why wouldn't the guard size in the metadata work properly when it has that much PROT_NONE memory after the mapping?

Is there an issue beyond the kernel not merging the VMAs? It doesn't work as well as it should and we might need to change it to work around that but it should be able to merge them if they're identical. I'm aware it has limitations with what it can merge in practice which are annoying for mremap.

[t-h-i-n-k-er]

Why wouldn't the guard size in the metadata work properly when it has that much PROT_NONE memory after the mapping?

Is there an issue beyond the kernel not merging the VMAs? It doesn't work as well as it should and we might need to change it to work around that but it should be able to merge them if they're identical. I'm aware it has limitations with what it can merge in practice which are annoying for mremap.

Apologies about the initial bit, the metadata value is numerically correct in terms of how much PROT_NONE follows the allocation, even though the semantic guard page is gone. I was wrong about the mremap path though. The VMA at old + old_size after the trail unmap is actually guard_size bytes, not trail_size. The trail unmap always carves from offset guard_size to the end of the trailing PROT_NONE VMA, so it leaves exactly guard_size bytes. The mremap would work size-wise.

The problem is what those bytes are. The first trail_size bytes at old + old_size are trail padding, and only the remaining guard_size - trail_size bytes are a fragment of the actual guard (when trail_size < guard_size), or the guard is fully gone and it's all padding (when trail_size >= guard_size). The guard page that guard_size in the metadata is supposed to describe doesn't exist at that position anymore. Overflow still faults because it's all PROT_NONE, but anything that reasons about what specifically sits at old + old_size is operating on an incorrect layout.

Under CONFIG_LABEL_MEMORY this matters concretely. prctl(PR_SET_VMA_ANON_NAME) names VMAs, and the "guard" VMA after the trail unmap is actually trail padding. The named region in /proc/pid/maps doesn't correspond to the actual guard, which makes the labeling misleading for debugging. The fix produces a single clean guard VMA that's actually the guard.

The mprotect ordering is also a real improvement. Currently memory_protect_rw runs before the trail unmap, so the RW VMA sits between PROT_NONE fragments until the unmap finishes. Moving it after the unmap/remap means the RW VMA is only established once the layout is finalized, no window where usable pages sit between fragments from the pending remap.

[thestinger]

Note there's also #258 as an alternative to PROT_NONE. For regular non-aligned allocation it would require mmap + 2x madvise instead of mmap + mprotect along with madvise (DONTNEED) + madvise (GUARD_INSTALL) for the quarantine instead of only 1x mmap which is annoying. If we did that then there wouldn't be issues with VMA merging. However, it has other downsides. We could also use a similar approach for guard slabs by replacing the PROT_NONE memory with installed guards, perhaps always replacing the one before the latest slab but leaving the PROT_NONE region after it. We could also free slabs via MADV_DONTNEED + INSTALL_GUARD instead of replacing them with PROT_NONE memory. One of the downsides of all of this would be reducing support for not using overcommit but hardened_malloc is better for that than typical allocators right now.

[t-h-i-n-k-er]

MADV_GUARD_INSTALL would sidestep the VMA issue cleanly since guards wouldn't be separate VMAs in the first place. The fix I'm proposing is for the PROT_NONE path as it exists today, and it would still be needed if that path sticks around for kernel compatibility or as a fallback. Happy to rework or drop this if you'd rather go the GUARD_INSTALL route instead.