C++ Lua userdata messing with Mathfu float values

Question

I'm trying to use Mathfu::Vector from Lua to create vectors but I've got an issue when I try to retrieve the user data back from the stack.

here is the function I use to define the user data

static int vec4(lua_State *L)
{
  const float x = luaL_checknumber(L, 1);
  const float y = luaL_checknumber(L, 2);
  const float z = luaL_checknumber(L, 3);
  const float w = luaL_checknumber(L, 4);

  mathfu::Vector<float, 4> vec(x, y, z, w);
  *static_cast<mathfu::Vector<float, 4> **>(
      lua_newuserdata(L, sizeof(mathfu::Vector<float, 4> *))) = &vec;
  luaL_setmetatable(L, VECTOR_LIB_NAME);

  return 1;
}

And I have a function that gets the dot product of two vectors based on a self value and an argument (both userdata)

static int dotVector(lua_State *L)
{
  mathfu::Vector<float, 4> *vecA = *static_cast<mathfu::Vector<float, 4> **>(luaL_checkudata(L, 1, VECTOR_LIB_NAME));
  mathfu::Vector<float, 4> *vecB = *static_cast<mathfu::Vector<float, 4> **>(luaL_checkudata(L, 2, VECTOR_LIB_NAME));

  LOGI("A: %f, %f, %f", vecA->x, vecA->y, vecA->z);
  LOGI("B: %f, %f, %f", vecB->x, vecB->y, vecB->z);
  float dot = mathfu::Vector<float, 4>::DotProduct(*vecA, *vecB);
  lua_pushnumber(L, dot);

  return 1;
};

called from lua like

local vecA = vec4(1, 2, 3, 4)
local vecB = vec4(5, 6, 7, 8)
vecA:dot(vecB)

But the values that get printed are crazy like it's converting bad data to a float

A: -8055257885499399810754615288266752.000000, 0.000000, 0.000000
B: 0.000000, 3746318080.000000, 3573.204346

I've examined the stack and the first two elements are definitely UserData but I don't seem to be able to dig any deeper into the user data. The vars in C++ contain those values but I'm at a loss as to what might be causing this. Pretty sure the userdata isn't what I expect it to be where I expect it to be.

Alex Zaharov Alex Zaharov · Accepted Answer · 2020-02-18T16:45:27

No, you did memory leak. 1st version was pointer to stack allocated object (&vec) which is gets destroyed once function ends. 2nd version (new mathfu::Vector) creates properly on heap ... and i guess u never delete it later = memory leak.

Also try avoid static_cast of pointers. Or any forced cast. That will add problems, for example if g++ will use "strict-aliasing" (implicit by -O3) then any pointers casts will be broken. "strict-aliasing" assumes 1 memory block is never pointed by different type pointers. Or you should explicit configure compiler to stay away from that. General good idea is to use memcpy. Compilers understand & optimize it properly. For example if you want pointer as char to int32:

int32 a; 
char b[sizeof(a)]; 
memcpy(b, &a, sizeof(a));

and use b now instead

static_cast<char*>(&a);

C++ Lua userdata messing with Mathfu float values

2 Answers