Fix extension handshake for third party Wii Remotes

[kglazier]

Third party Wii Remotes connect to the kernel's Bluetooth stack fine, but their extension is never identified, so the controller stays unusable in game and the link eventually dies with an HCI supervision timeout (reason 0x08), reconnects, and repeats. Official remotes are unaffected.

I traced this by adding temporary dbgprintf logging to BT.c and lwbt/hci.c and reading /ndebug.log off the SD card. The handshake stalls in TRANSFER_EXT1 and never reaches TRANSFER_GET_IDENT.

There are two separate causes.

1. The 0x11 LED ack is misread as an extension change

In the 0x22 acknowledge branch, buffer[3] is the report id being acknowledged, and the code tests it with & 0x02. That works for 0x12/0x16/0x17, which is what the existing //??message being acknowledged todo lucky all needed messages had 2 bit set comment is getting at.

Third party remotes also acknowledge the 0x11 player LED/rumble report that BTSetControllerState() sends. 0x11 & 0x02 == 0, so that ack fell through to the else branch, which treats it as the extension being unplugged and resets transferstate back to TRANSFER_EXT1. From a log:

DATA report=20 state=0 ctrl=0 b3=02   <- status report, handshake advances to EXT2
DATA report=22 state=2 ctrl=0 b3=11   <- ack of the LED report -> reset back to EXT1
DATA report=22 state=1 ctrl=0 b3=12
DATA report=22 state=1 ctrl=0 b3=16   <- stuck in EXT1 from here on

Fix: never take the reset path when the acknowledged report id is 0x11.

2. The status report arrives before it is requested

These remotes send their 0x20 status report on their own while still in TRANSFER_CONNECT, and do not send another one in response to the 0x15 request, so TRANSFER_EXT1 waits forever. Fix: accept the status report in TRANSFER_CONNECT as well, and if an acknowledgement turns up while still sitting in TRANSFER_EXT1, re-send the 0x15 request rather than waiting indefinitely.

Testing

Built with the toolchain versions in the readme (devkitARM r53, devkitPPC r35, libogc 1.8.23). Tested on a Wii (System Menu 4.3U) with two different third party remotes plus an official Classic Controller, on Mario Kart: Double Dash and Mario Party. Both remotes go from unusable to fully working. An official remote with the same Classic Controller behaves the same before and after, and the changes only add conditions to paths that third party remotes were already falling into.

I do not have a MotionPlus or a Wii U Pro Controller to hand, so those paths are untested beyond the fact that they are not touched.

[PedroRafael215]

I learned how to build Nintendont just to test this new fix, and it worked perfectly with my third-party 2-in-1 Wii Remote!

Thank you so much for fixing this! 😁🥳🥳🥳