NaCl platform qualification hangs on more armhf variants

[slipher]

I've had the the NaCl platform qualification hanging (perhaps as in #740) on some armhf systems:

• A QEMU emulated virtual machine (not sure which CPU variant)
• Raspberry Pi 2. This has a "Broadcom BCM2836" CPU based on ARM Cortex A7.

When I debugged the QEMU case, I found it to be caused by a breakpoint instruction: __asm__ __volatile__ ("BKPT");. At the time, I wasn't able to find any information about this instruction causing the program to hang (i.e. halt execution, but not terminate) and though it must be an emulator quirk.

[slipher]

I hypothesize that this instruction is only there for permitting breakpoint instructions in user code, to be used with the nacl-gdb. If this is true, then when rebuilding the NaCl runtime we can just remove that check from the platform qualification and forbid the instruction in the validator.

[slipher]

It's true that the breakpoint instruction is intended by user code, but unfortunately there is another use of a breakpoint instruction with a different encoding:

https://github.com/DaemonEngine/native_client/blob/d281e6a3f08a13a7c7ba6ffd7160b2267a3c8765/src/include/arm_sandbox.h#L26-L33

[slipher]

The behavior of BKPT on affected systems seems to be immediately resuming execution at the same instruction. A program that has executed the instruction starts using 100% CPU. This happens in a simple test program with just the asm snippet in main(), so it has nothing to with how NaCl sets up things. Also the BKPT instruction does not pause execution even running under GDB.

Anyway I think I've figured it out. It's a regression in Linux commit c3f89986fde7bb9ccc86a901bf28e1f7d69fc3b3 "ARM: 9391/2: hw_breakpoint: Handle CFI breakpoints". This commit defines some software-defined error code ARM_ENTRY_CFI_BREAKPOINT whose value (3) overlaps with a hardware-generated error code. https://www.jwhitham.org/2015/04/the-mystery-of-fifteen-millisecond.html explains how it works in older, working kernels: the value 3 is not handled in the switch, and the error handler passes on the exception to whatever the next thing in the chain is. The commit's poorly chosen fallback behavior in case CONFIG_CFI_CLANG is not enabled in the kernel build apparently causes the program to resume execution at the same point.

I confirmed that booting an older kernel fixes the bug. 6.10 should be the first broken Linux kernel version; it would affect any (actually 32-bit) armhf system. I wonder if 64-bit ARM systems running the loader in 32-bit mode could be affected too, but the platform qualification is already disabled there for some other reason.

I'll do a bit more verification and then try to report this to kernel devs.

[illwieckz]

Thanks for the investigation!