Keep Lua state in a C++ environment to limit context switches

Question

I'm having fun coding simple OpenGL demos and I recently decided to use Lua with my C++ engine in order to change the rendering dynamically without having to recompile on and on my project. Thus I can tweak more easily the rendering algorithm. But I know that my current rendering update functions are probably far from being efficient.

For the moment, I'm transfering a matrix from C++ to Lua, modifying it in a Lua script and sending it back to my C++ rendering engine. But I'm reloading the Lua script each time I get an update call from the C++ engine, and I'm losing all of the variable context. That means I'm always starting from scratch and my rendering is far from being smooth. I include some code sample below to explain what I'm doing. I am currently learning Lua with C++ embedding, so I know I still don't have the best practices.

update.lua

function transform(m)
    amplitude = 1.5
    frequency = 500
    phase = 0.0
    r = {}

    for i = 1, #m do
        r[i] = {}
        for j = 1, #m[i] do
            if (i % 2) then
                r[i][j] = amplitude * math.sin(m[i][j] + phase)
            else
                r[i][j] = -amplitude * math.sin(m[i][j] + phase)
            end
            phase = phase + 0.001
        end
    end
    return r
end

-- called by c++
function update()
    m = pull()
    r  = transform(m)
    push(r)
end

matrix.cpp

// pull matrix from lua point of view
static int pull(lua_State * _L)
{
    _push(_L, &_m);

    return 1;
}

// push matrix from lua point of view
static int push(lua_State * _L)
{
    // get number of arguments
    int n = lua_gettop(_L);

    if(1 == n) {
        _pull(_L, 1, &_m);
    }

    return 1;
}

void matrix::load_file(char * file, char * function)
{
    int status;

    // load the file containing the script we are going to run
    status = luaL_loadfile(_L, file);
    switch (status) {
    case LUA_OK:
        break;
    case LUA_ERRFILE:
        std::cout << "LUA_ERRFILE: " << lua_error(_L) << std::endl;
        break;
    case LUA_ERRSYNTAX:
        std::cout << "LUA_ERRSYNTAX: " << lua_error(_L) << std::endl;
        break;
    default:
        std::cout << lua_error(_L) << std::endl;
    }

    lua_getglobal(_L, function);
    status = lua_pcall(_L, 1, 1, 0);
    if (status != LUA_OK) {
        std::cout << "error running file" << lua_error(_L) << std::endl;
    }
}

void matrix::update()
{
    load_file("lua/update.lua", "update");
}

I'm thinking of passing some arguments when calling the update() function, but I'm wondering if the C++ to Lua then back to C++ approach is correct and efficient. Especially considering the fact that I might transfer and modify huge matrix in Lua. I probably lack some embedded Lua knowledge to keep context while loading a script. Do you have some general advice on how I would improve my code ? I know that my current approach is overly complicated.

Answer 1

A quick fix would be to only load the file if it has been modified since the last frame:

static time_t last_modified = 0;
struct stat sbuf;
stat(file, &sbuf);
if (sbuf.st_mtime > last_modified) {
    last_modified = sbuf.st_mtime;
    status = luaL_loadfile(_L, file);
    // etc
}

// Now call the function
lua_getglobal(_L, function);
status = lua_pcall(_L, 1, 1, 0);

Answer 2

OK, loading the chunk of the update() function into a global variable and having a global parameter table in the Lua script is the way to go. I achieved this using the following guidelines, and I will post the detailed steps below. Basically, loading the script entirely first ensures that all global variables are stored in the C++ context. Then storing the wanted function as an index allows us to run it again, while keeping the global variables in the script evolving on their own.

Step 1

First call luaL_loadfile once at init

Step 2

Run the script once using lua_pcall(_L, 0, 0, 0);

This ensures that the global variables, which are used as parameters in the Lua script are in memory.

Step 3

Store the Lua function. I managed to do it with the following C++ code:

void matrix::store(char * function)
{
    lua_newtable(_L);  // create table for functions
    _idx = luaL_ref(_L, LUA_REGISTRYINDEX); // store said table in pseudo-registry
    lua_rawgeti(_L, LUA_REGISTRYINDEX, _idx); // retrieve table for functions

    lua_getglobal(_L, function); // retrieve function to store

    if (lua_isfunction(_L, -1)) {
        _f = luaL_ref(_L, -2); // store a function in the function table
    }
    else {
        lua_pop(_L, 1);
        std::cout << "can't find " << function << std::endl;
    }

    // table is two places up the current stack counter
    lua_pop(_L, 1); // we are done with the function table, so pop it

    std::cout << "idx: " << _idx << ", function: " << _f << std::endl;
}

Step 4

Call the stored function again when rendering using the following C++ function:

void matrix::run()
{
    int status;

    if (_f == -1) {
        std::cout << "invalid function index " << _f << std::endl;
    }
    else {
        lua_rawgeti(_L, LUA_REGISTRYINDEX, _idx); // retrieve function table
        lua_rawgeti(_L, -1, _f); // retrieve function
        //use function
        status = lua_pcall(_L, 0, 0, 0);  // 0 arguments, 0 results
        if (status != LUA_OK) {
            std::cout << "error running function" << lua_error(_L) << std::endl;
        }
        //don't forget to pop the function table from the stack
        lua_pop(_L, 1);
    }
}

Step 5 (optional)

If we set all the Lua parameters in a global table, we can retrieve them dynamically in C++ using the following piece of code:

void matrix::get_params(char * p)
{
    lua_getglobal(_L, p);
    lua_pushnil(_L);

    int i = 0;
    while(lua_next(_L,-2))
    {
        const char * key = lua_tostring(_L,-2);
        double value = lua_tonumber(_L,-1);
        lua_pop(_L,1);
        std::cout << key << " = " << value << std::endl;
        _h[i].key.assign(key);
        _h[i].value = value;
        i++;
    }
    lua_pop(_L, 1);
}

Where _his a simple dynamic structure defined as such:

typedef struct {
    std::string key;
    float value;
} hash;

I only use float, so this simple structure is convenient enough for my needs, and allows me to add lots of variables in my Lua script without bothering with a structure definition in C++. This way I can add as many parameters in my Lua table and do the maths when updating.

Step 6

Tweak the Lua script forever ! Et voila:

p = {
    amplitude = 1.5,
    frequency = 500,
    phase = 0.0
}

function transform(m)
    r = {}

    for i = 1, #m do
        r[i] = {}
        for j = 1, #m[i] do
            if (i % 2) then
                r[i][j] = p.amplitude * math.sin(m[i][j] + p.phase)
            else
                r[i][j] = -p.amplitude * math.sin(m[i][j] + p.phase)
            end
            p.phase = p.phase + 0.001
        end
    end
    return r
end

-- called by c++
function update()
    m = pull()
    r  = transform(m)
    push(r)
end

This solution fits my needs, but seems very complicated and inefficient. But it was a fine hacking session anyway.