I am calling a NASM 64-bit DLL from ctypes. The dll takes five input parameters. In the Windows calling convention, the first four are passed in rcx, rdx, r8 and r9, and the fifth is passed on the stack.
The Overview of x64 Calling Conventions doc (https://docs.microsoft.com/en-us/cpp/build/overview-of-x64-calling-conventions) says "Any parameters beyond the first four must be stored on the stack, above the shadow store for the first four, prior to the call." So therefore the value can't be accessed with a pop, and I think it should be accessed with RSP. If the fifth (and later) parameters are above the shadow store, then I guessed it would be RSP minus 40 (mov rax,[rsp-40]), but it's not.
I tried "walking the stack" which means I tried at rsp-0, rsp-8, rsp-16, etc, all the way to rsp-56, but it did not return the value that I had passed as the fifth parameter (a single 64-bit double).
According to https://docs.microsoft.com/en-us/cpp/build/stack-allocation, the stack layout on entry is return address, rcx, rdx, r8, r9, and the stack parameter area, so I would expect to find my value at rsp-48, but it's not there, nor is it at rsp-56.
So my question is: how do I access a parameter passed on the stack on entry to the dll in the Windows calling convention?
EDIT: Here is the relevant ctypes code:
hDLL = ctypes.WinDLL("C:/Test_Projects/MultiCompare/py_descent.dll")
CallName = hDLL.Main_Entry_fn
CallName.argtypes = [ctypes.POINTER(ctypes.c_double),ctypes.POINTER (ctypes.c_double),ctypes.c_double,ctypes.POINTER(ctypes.c_double),ctypes.c_double]
CallName.restype = ctypes.c_double
ret_ptr = CallName(CA_x,CA_d,CA_mu,length_array_out,CA_N_epochs)
Data types:
CA_x: pointer to double(float) array
CA_d: pointer to double(float) array
CA_mu: double
length_array_out: pointer to double(float) array
CA_N_epochs: double
Here is the dll entry point where the vars are retrieved. I always push rdi and rbp on entry, so I take parameters passed on the stack first before I do that to prevent stack misalignment:
Main_Entry_fn:
; First the stack parameters
movsd xmm0,[rsp+40]
movsd [N_epochs],xmm0
; End stack section
push rdi
push rbp
mov [x_ptr],rcx
mov [d_ptr],rdx
movsd [mu],xmm2
mov [data_master_ptr],r9
; Now assign lengths
; (this part intentionally omitted for brevity)
call py_descent_fn
exit_label_for_Main_Entry_fn:
pop rbp
pop rdi
ret
func(int,int,double,int)would use RCX, RDX, XMM2, R9. A 5th parameter would always be on the stack, e.g. [RSP+40]. – Mark Tolonen