0
votes

I try to hook some functions, doesn't matter if __stdcall or __cdecl on x86.

I want to do the following things:
1. Preserve the stack
2. Preserve de registers
3. Do my things
4. Restore the registers
5. Restore the stack

I don't have problems with the stack, but I do have some problems with my registers backup: I cannot backup registers without modifying some of them (I can NOT use the stack because I cannot backup the stack this way)!

It is possible for me to backup them on the heap (I use a structure with some .EAX , .EBX and so on members), I acces that structure from ASM and here is the problem... I have to modify some registers to be able to do this!

However, this is my story. What I really want to understand is the answer to the following question:

Is there any "rule" about the registers that can be modified and the registers that are NOT modified during a function call?

I checked with a debugger some function calls. I add a breakpoint at "call SomeFunction", press F8 to "step over function call" and check the modified registers. I can see something like this: 1. ESP/EBP may be modified depending on calling convention (cdecl vs stdcall) 2. EAX, EBX, EDX - are almost always modified! 3. EBX, EDI, ESI seems to be ALWAYS preserved!

So here comes my "pseudo-solution": is is ok if I preserve only those registers (EBX, EDI, ESI)? I don't mess up the stack so EBP and ESP are not a problem. But I have to modify some registers (EAX, ECX, EDX).

Will I have any problems with some compiler optimizations? Is it possible to mess up the code by modifying those "innocent" registers: EAX, ECX, EDX?

Thank you

3
What do you mean by preserving the stack? You don't really preserve it, just restore SP after you finish.Andrey
Yes, I do not modify EBP and I modify but restore ESP. The problem is more complicated than I described. I just need to know if it is ok to preserve only those registers: EBX, EDI, ESI.Ionut
@lonut you must preserve all registers. But I don't get your problem, it sounds trivial.Andrey
A function looks like this: push param; call function; The call push next EIP on stack and goes to that function. 1. On that function I jump to my code (asm only function, naked) 2. I have to call some functions and jump to another function (callback) 3. The callback function must have the stack as before the call (same prototype) 4. After callback function is executed I must have unmodified registers and stackIonut
@lonut sorry, I still don't get this all. you want to do something custom or what? Wny not you just follow calling conventions and just keep it simple?Andrey

3 Answers

1
votes

It is unclear why you need anything other than standard techniques for saving registers. The usual scheme is:

    ; call site: push args into stack, move to registers as required for callee
          push  arg1
          push  arg2
          mov   reg, arg3
          call  subroutine
          lea   esp, sizeof(arg1)+sizeof(arg2)[esp]  ; pop arg1 and arg2
          ...

    subroutine:
          push  ebp

          push  reg1     ; save the registers that subroutine is documented to preserve
          ...
          push  regn     ; if you insist, save *all* the registers
          mov   ebp, esp ; now ebp points to stacked args/saved registers no matter what you do next
          ; note: we have not saved the ESP, but we do know how much we pushed on ESP
          ...
          body of subroutine, move ESP up and down
          ...
    subroutine_exit:
          mov    esp, ebp ; now points to saved registers
          pop    regn
          ...
          pop    reg1
          pop    ebp
          ret
0
votes

Yes, calling convention allows you to clobber EAX, ECX and EDX. I believe in microsoft c++ ECX is used for the this pointer though. Note this applies to functions following the convention only. If you try to hook an internal function which might have been optimized to not use the standard convention, it won't work.

You can just save stuff in the data section, with extra care about reentrancy and concurrency if applicable.

0
votes

Yes, I do the following things:
1. Place a jump to instead of first 5 function bytes (classic hook)
2. Jump to my naked asm only function
3. In my function I need to call some functions (memcpy - restore original bytes, FlushInstruction cache) and work with a structure so here I need to modify some registers
4. I add stuff on the stack but at the end the stack is the same as in the "call" moment
5. I JUMP to the callback function (jmp EAX) with correct stack! I cannot get back here because in the callback function I need to call the original function and return to "call function + 5" (return EIP put on the stack at call time). And I have to jump with the correct stack because the parameters are on the stack, the hook is generic and I don't know how many parameters are there!

So, in the callback function I do my stuff, call original function and restore the hook. What I need is exactly this: I need to restore the registers, or to make sure I do not modify them, but as I said I have to do it.

At this moment I'm trying to avoid using EBX, EDI and ESI. I hope it will work and I will not have problems with optimizations. If it does not work, I will try to backup them in my structure and restore them in the callback function.

Thank you all!