Register ESI causes RunTime-Check Failure #0 error

Question

I've spend lot of time trying to solve this problem and I don't understand, why it doesn't work. Problem's description is in comments below:

.386
.MODEL FLAT, STDCALL

OPTION CASEMAP:NONE
.NOLIST
.NOCREF
INCLUDE \masm32\include\windows.inc
.LIST

.CODE
DllEntry PROC hInstDLL:HINSTANCE, reason:DWORD, reserved1:DWORD
mov eax, TRUE
ret
DllEntry ENDP

caesarAsm proc string: DWORD, key: DWORD, stringLength : DWORD

        mov esi, 1 ; I cannot use this register, mov esi, (anything) causes Crash:
        ; Run-Time Check Failure #0 - The value of ESP was not properly saved across a function call.  This is usually a result of calling a function declared with one calling convention with a function pointer declared with a different calling convention
        mov eax, string 
        ret
caesarAsm endp
END DllEntry

I searched "whole" Internet, I found that the problem is connected with stack, but no operation on stacks helped me to solve it. I'm using Microsoft Visual Studio 2012

Jester Jester · Accepted Answer · 2015-10-22T12:55:16

I assume the error does not occur in this function, rather, it is triggered elsewhere. The esi register is a callee-saved register. You must make sure its value is the same at function exit as it was at entry. You can use it in your function, but you must save and restore its value. Such as:

    push esi
    mov esi, 1
    mov eax, string 
    pop esi
    ret

This is all well documented. You may only use eax, ecx and edx without saving.

Side note: you are using high-level features of your assembler, you might want to check the actual generated code or refrain from using them until you are confident in what the result is going to be. Incidentally masm has a USES keyword which would do the save/restore for you.

Register ESI causes RunTime-Check Failure #0 error

1 Answers