[wasm2c] Fix exception testing on macOS

[kevmoo]

This commit addresses the assertion failures during exception testing
in wasm2c on macOS:
Assertion failed: (!(ss.ss_flags & SS_ONSTACK) && "attempt to deallocate altstack while in use")

Exception handling in wasm2c uses wasm_rt_try(target) which mapped
to sigsetjmp(buf, 1). However, on macOS (XNU), performing nested
sigsetjmp(..., 1) and siglongjmp across threads with an allocated
alternate signal stack can erroneously cause the kernel to preserve the
SS_ONSTACK flag in the thread state, even when the exception did not
originate from the signal handler.

Since WebAssembly exception handling relies strictly on setjmp/longjmp
for control flow and does not need to restore signal masks (unlike stack
exhaustion traps, which do need the mask to unblock SIGSEGV), the fix
is to simply avoid saving the signal mask for exception boundaries.

We split WASM_RT_SETJMP_SETBUF into two variants:

• WASM_RT_SETJMP_TRAP_SETBUF: Uses sigsetjmp(buf, 1) for traps (safely handles SIGSEGV).
• WASM_RT_SETJMP_EXN_SETBUF: Uses sigsetjmp(buf, 0) for Wasm exceptions (bypasses SS_ONSTACK bug).

This also reverts commit 74cc83c
(which temporarily disabled macos-latest in CI), as the tests now
pass successfully.

Fixes #2654

[kevmoo]

CC @sbc100

This is 99% vibe coded. I know ~0 C++. I asked gemini to investigate the failure and see if we could resolve it carefully. Gemini wrote the commit message.

Zero offence will be taken if this is closed w/ prejudice.
Discovered while trying to help out with #2622 to see if I could get dart2wasm bits parsing.

[sbc100]

Thanks for working on this!

Is there an actual upstream macOS kernel bug for this maybe?

[kevmoo]

I saw zero info on anything upstream. Just the same issue being hit in Go.

[kevmoo]

I tried to leave enough breadcrumbs...

[kevmoo]

Here's the gemini-CLI dump of the explanation of this issue. Again, this is 99% vibes.

Core macOS Bug: XNU Kernel SS_ONSTACK Regression

The assertion failure (attempt to deallocate altstack while in use) is caused by a long-standing kernel-level bug in macOS (XNU), which has become significantly more prevalent in macOS 14 (Sonoma) and macOS 15 (Sequoia).

1. The "Sticky" SS_ONSTACK Flag

On macOS, when a signal handler is executed on an alternate stack (configured via sigaltstack), the kernel sets an internal SS_ONSTACK flag.

• The Bug: If the handler is exited via siglongjmp instead of a normal return, the kernel fails to clear this flag.
• The Result: The system "thinks" the thread is still executing on the alternate stack even after control has returned to the main stack.

2. Erroneous Setting via siglongjmp(..., 1)

More critically for wasm2c, recent versions of the XNU kernel appear to erroneously set the SS_ONSTACK flag when siglongjmp(env, 1) (with signal mask restoration enabled) is called outside of a signal handler, provided an
alternate stack is installed for that thread.

This happens because siglongjmp(..., 1) triggers a specific kernel path for signal mask restoration that incorrectly "commits" the thread's state to the alternate stack, even during normal control flow.

3. Impact on WABT / wasm2c

In wasm2c, WebAssembly exceptions use wasm_rt_try and wasm_rt_throw, which map to sigsetjmp and siglongjmp:

• Previous Behavior: Using sigsetjmp(buf, 1) caused the kernel to erroneously mark the thread as SS_ONSTACK.
• The Assertion: When the thread finished and attempted to clean up its alternate stack in os_disable_and_deallocate_altstack, it called sigaltstack(NULL, &ss) and found the SS_ONSTACK flag set, triggering the fatal
  assertion: "attempt to deallocate altstack while in use".

4. The Fix

By switching to sigsetjmp(buf, 0) for Wasm exceptions, we bypass the buggy signal mask restoration path in the macOS kernel. Since Wasm exceptions are purely for control flow and do not need to restore signal masks (unlike Wasm
trap handlers recovering from a SIGSEGV), this safely avoids the erroneous SS_ONSTACK persistence.

Technical References

• Go Issue #44501: Extensive documentation of this macOS kernel behavior and its impact on language runtimes.
• Apple Radar #FB13684124: Related regressions in macOS 14 signal handling (specifically regarding SIGKILL vs SIGSEGV on Apple Silicon).
• Wasmtime/LLVM: Similar workarounds have been implemented in other runtimes to avoid siglongjmp(..., 1) on macOS due to these kernel-level stack-tracking failures.

[shravanrn]

@sbc100 I don't know what the policy in this project is AI generated submissions, so I'll let you approve this PR if needed. I haven't checked the source of the bug, but if is as described, the spirit of this fix looks reasonable modulo comments

[sbc100]

Thanks @kevmoo and AI for the fix :)

Thanks @shravanrn for taking a second a look. Good to see the rather nasty OS bug fixed (worked around).