Unable to access array pointer passed to NASM 64 bit DLL

Question

I am passing pointers two arrays from a Python program (using ctypes) to a NASM 64-bit DLL. In the Windows calling convention, the pointers are passed in rcx and rdx. The two arrays are both Int64. The arrays are created in Python and passed to the DLL this way:

PassArrayType = ctypes.c_int64 * 10
PVarrNew = PassArrayType()
OutputArrayType = ctypes.c_int64 * 1000
arrNew = OutputArrayType()
retvar = SimpleTest(ctypes.byref(PVarrNew), ctypes.byref(arrNew))

In the DLL, I can read from the array pointer in rcx but I can't write to the array.

For example, reading a value from the array pointed to by rcx works:

push qword [rcx+32]
pop qword [tempvar]

But writing a value to the array pointed to by rcx does not work:

mov rax,1235
push rax
pop qword [rcx+32]

While writing the same value to a variable works:

mov rax,1235
push rax
pop qword [tempvar]

I cannot read from OR write to the array pointed to by rdx.

So my questions are:

Why can I read from the array pointed to by rcx but not write to it?
Why can't I read or write to the array pointed to by rdx?

Michael, thanks for your reply. I misunderstood minimal to mean only the code lines at issue. I'm fairly new to Stack Overflow, but now I understand how much code to post. Here's the full Python code and the full NASM code. Python is 3.6.2. NASM is 64-bit.

Python code:

OutputArrayType = ctypes.c_int64 * 1000
arrNew = OutputArrayType()

PassArrayType = ctypes.c_int64 * 10
PVarrNew = PassArrayType()
PVarrNew[0] = id(PVarrNew)
PVarrNew[1] = 2
PVarrNew[2] = len(PVarrNew)

ThisDll = ctypes.WinDLL(r"C:/Temp2/Std_Math_Formulas.dll")
SimpleTest = ThisDll.SimpleTest
SimpleTest.argtypes = [ctypes.c_void_p, ctypes.c_void_p]
SimpleTest.restype = ctypes.c_int64

retvar = SimpleTest(ctypes.byref(PVarrNew), ctypes.byref(arrNew))

NASM code:

; Header Section
[BITS 64]

export SimpleTest
section .data
tempvar:  dq 0

section .text
finit

SimpleTest:

push rdi
push rbp

mov rdi,rcx
mov rbp,rdx

push qword [rcx+32]
pop qword [tempvar]

; this works with rcx, but not rdx
mov rdi,rcx
push qword [rdi+32]
pop qword [tempvar]

; this works with rcx, but not rdx
mov rax,1235
push rax
pop qword [rcx+32]

mov rax,[tempvar]

pop rbp
pop rdi
ret

I assemble and link my DLL with:

nasm -Z myfile.err -f Win64 C:\Temp2\Std_Math_Formulas.asm -l myfile.lst -F cv8 -g -o C:\Temp2\Std_Math_Formulas.obj 
GoLink Std_Math_Formulas.obj /dll /entry SimpleTest msvcrt.dll

Michael, thanks for your reply. I have added both sets of code (Python and NASM) below the original question above. — RTC222
Using NASM for compiler and GoLink for linker: COMPILE: nasm -Z myfile.err -f Win64 C:\Temp2\Std_Math_Formulas.asm -l myfile.lst -F cv8 -g -o C:\Temp2\Std_Math_Formulas.obj LINK: GoLink Std_Math_Formulas.obj /dll /entry SimpleTest msvcrt.dll — RTC222
Now we are getting somewhere. /entry is a special entry point that exists for DLL initialization. It is a special function with its own parameters. By specifying /entry SimpleTest you tell Windows to have Windows initialize the DLL by calling that function. That's likely causing serious issues. Remove the /entry SimpleTest so there is no DLL entry point. since you aren't using the C runtime (yet) you don't need the msvcrt.dll either. Try it with just GoLink Std_Math_Formulas.obj /dll — Michael Petch
I'm curious. The code you are showing here. Is it exactly what are you using. Or are you assembling different code? Have you tried to use the exact code you have posted into the question (without alteration)? I have a standard Python 3.6.4 install, NASM, and GOLINK and with the exception of having to add a line import ctypes into the python code you showed it works if the /entry is left off.It also work if I change RCX to RDX. In that case arrNew[4] has the value 1235 in it after it returns (the return value will be 0 but that's because the array had 0s in it to begin with) — Michael Petch
Michael, you are right -- it was the GoLink command. I just posted this solution as an answer below. — RTC222

RTC222 RTC222 · Accepted Answer · 2018-01-06T18:06:43

The solution proposed above by Michael Petch solved the problem. I removed "/entry SimpleTest" from the GoLink linker command string, as Michael proposed, and now I can both read from and write to from the arrays pointed to by rcx and rdx. The correct command string is:

GoLink Std_Math_Formulas.obj /dll msvcrt.dll

Many thanks for your solution; I am extremely grateful for your help.

Unable to access array pointer passed to NASM 64 bit DLL

1 Answers