Automatically detecting violations of assembly calling convention

Question

Is there a way for the assembler (or a static analyzer) to warn if the hand-coded assembly code contains violations of the platform's assembly calling convention?

The platform I'm using is ARMv7A with the GNU GAS assembler. The reason for the question is a bug I wrote where my function did not push/pop the required registers (r4-r11 on ARM) upun entry/exit. The registers were trashed, causing the caller to crash (thankfully, the automated tests detectected the bug). Simplified program:

my_function:
    mov     r4, #42  @Trash register r4 in violation of calling convention
    bx      lr       @Return from function

caller:
    ...
    mov r4, #4        @Initialise register r4, to be used later
    bl my_function    @Call my_function with no arguments
    mov r0, r4        @Set argument r0 as r4 (== 42, but should be 4)
    bl other_function @Call other_function with (the now trashed) argument r0

ARM calling convention: http://infocenter.arm.com/help/topic/com.arm.doc.ihi0042d/IHI0042D_aapcs.pdf

This might be a bit naive because I'm not an assembly programmer, but what about a debugger like gdb? — Douglas B. Staple
@JaggedO'Neill: The Application Binary Interface (for any particular platform) is a specification, not convention. You cannot expect to alter registers that the calling routine relies on without consequences. — Eric Postpischil
It should be stated the requested task is impossible to do perfectly, because it is equivalent to the halting problem. Typical interface requirements state that certain registers should have the same value on exit that they did on entry. The called routine is not required to save and restore the register, merely not to leave it altered. If a routine to determine whether a register were left unaltered existed, we could construct a routine that altered a register if and only if the first routine indicated this routine did not alter the register. — Eric Postpischil

Alexey Frunze Alexey Frunze · Accepted Answer · 2012-10-02T01:09:06

I don't think there's such a tool. It is expected that assembly programmers know very well what they're doing.

However, if you're really after it, you can develop such a tool.

In it you'd need to parse the assembly source code and recognize instructions modifying the registers that have to be preserved by a subroutine. Ditto for the instructions saving/restoring those registers.

You will also need to figure out where subroutines begin and end. The beginning can be recognized by noting that the label, designating the entry point into the subroutine, is defined as public/global/whatever, basically, that it is supposed to visible from outside by the linker. Another possible heuristic is seeing that a label is used in call instructions, or whatever the equivalent of it is. Likewise you can determine subroutine ends, by noting the instructions or sequences of them that perform a return.

There may be cases where it's hard to figure out the beginnings and ends or whether a register is being truly trashed or preserved. Some of them can be resolved with extra heuristics. The rest can by default result in warnings. It's probably better to have false negatives, false alarms, then the other way around. If the code is small, figuring out whether the warning is meaningful or not should be easy.

For this task you should consider some scripting or scriptish languages that can handle well strings, support regular expressions and support "standard" containers and algorithms working with them (searching/sorting/etc). Perl and Python could do the job well. I'd not advise doing it in C or C++ because you'd need to rewrite and throw away numerous small pieces in the development process. Compiling is an extra hurdle, however small it may be. Debugging low-level code or templates isn't fun.

Automatically detecting violations of assembly calling convention

1 Answers