Issues installing yaml for lua with luarocks during the c-compile

Question

I have lua, that I isntalled using luarocks.

Edit: on a windows system

I successfully installed luasocket by downloading the luasocket-2.0.2-3.win32-x86.rock file from v202-3 from here. Its an older version, but it works.

However now I need to install yaml so I can work with yaml files. Now I am forced to install using a *.src.rock file or a .rockspec file.

I have tried:

luarocks install yaml which gives:

D:\user\workspace\_lua>luarocks install yaml
Warning: Failed searching manifest: Failed fetching manifest for https://luarocks.org - Failed downloading https://luarocks.org/manifest - C:\Users\user\AppData\Local/LuaRocks/Cache/https___luarocks.org/manifest
Warning: Failed searching manifest: Failed fetching manifest for https://raw.githubusercontent.com/rocks-moonscript-org/moonrocks-mirror/master/ - Failed downloading https://raw.githubusercontent.com/rocks-moonscript-org/moonrocks-mirror/master/manifest - C:\Users\user\AppData\Local/LuaRocks/Cache/https___raw.githubusercontent.com_rocks-moonscript-org_moonrocks-mirror_master_/manifest
Warning: Failed searching manifest: Failed fetching manifest for http://luafr.org/moonrocks/ - Failed downloading http://luafr.org/moonrocks/manifest - C:\Users\user\AppData\Local/LuaRocks/Cache/http___luafr.org_moonrocks_/manifest
Warning: Failed searching manifest: Failed fetching manifest for http://luarocks.logiceditor.com/rocks - Failed downloading http://luarocks.logiceditor.com/rocks/manifest - C:\Users\user\AppData\Local/LuaRocks/Cache/http___luarocks.logiceditor.com_rocks/manifest

Error: No results matching query were found.

So, then I downloaded yaml-1.1.2-1.src.rock and ran the command: luarocks install yaml-1.1.2-1.src.rock

This told me that I had dependencies yaml --> lub --> luafilesystem. so then I downloaded:

luafilesystem-1.6.3-2.src.rock
lub-1.1.0-1.src.rock
yaml-1.1.2-1.src.rock

And I installed them one after the other with the result:

luarocks install luafilesystem-1.6.3-2.src.rock - OK
luarocks install lub-1.1.0-1.src.rock - OK
luarocks install yaml-1.1.2-1.src.rock - FAILED, see errors below

Error log:

D:\user\downloads\luarocks-2.3.0-win32\extras>luarocks install yaml-1.1.2-1.src.rock
Using yaml-1.1.2-1.src.rock... switching to 'build' mode
mingw32-gcc -O2 -c -o src/api.o -ID:/user/install/lua/luarocks/include src/api.c -Isrc
mingw32-gcc -O2 -c -o src/b64.o -ID:/user/install/lua/luarocks/include src/b64.c -Isrc
mingw32-gcc -O2 -c -o src/dumper.o -ID:/user/install/lua/luarocks/include src/dumper.c -Isrc
mingw32-gcc -O2 -c -o src/emitter.o -ID:/user/install/lua/luarocks/include src/emitter.c -Isrc
mingw32-gcc -O2 -c -o src/loader.o -ID:/user/install/lua/luarocks/include src/loader.c -Isrc
mingw32-gcc -O2 -c -o src/lyaml.o -ID:/user/install/lua/luarocks/include src/lyaml.c -Isrc
mingw32-gcc -O2 -c -o src/parser.o -ID:/user/install/lua/luarocks/include src/parser.c -Isrc
mingw32-gcc -O2 -c -o src/reader.o -ID:/user/install/lua/luarocks/include src/reader.c -Isrc
mingw32-gcc -O2 -c -o src/scanner.o -ID:/user/install/lua/luarocks/include src/scanner.c -Isrc
mingw32-gcc -O2 -c -o src/strtod.o -ID:/user/install/lua/luarocks/include src/strtod.c -Isrc
In file included from src/strtod.c:23:0:
src/strtod.c:24:17: warning: '_errno' redeclared without dllimport attribute: previous dllimport ignored [-Wattributes]
 extern  int     errno;
                 ^
mingw32-gcc -O2 -c -o src/writer.o -ID:/user/install/lua/luarocks/include src/writer.c -Isrc
mingw32-gcc -shared -o yaml/core.dll src/api.o src/b64.o src/dumper.o src/emitter.o src/loader.o src/lyaml.o src/parser.o src/reader.o src/scanner.o src/strtod.o src/writer.o D:/user/install/lua/luarocks/lua5.1.lib -lMSVCR80
src/api.o:api.c:(.text+0x395): undefined reference to '_imp__yaml_realloc'
src/api.o:api.c:(.text+0x412): undefined reference to '_imp__yaml_string_extend'
            :
    LOTS OF THESE
            :
src/scanner.o:scanner.c:(.text.unlikely+0x29d0): undefined reference to '_imp__yaml_queue_extend'
src/scanner.o:scanner.c:(.text.unlikely+0x29e8): undefined reference to '_imp__yaml_token_delete'
collect2.exe: error: ld returned 1 exit status

Error: Build error: Failed compiling module yaml/core.dll

Does anyone know why this build fails? - I got it directly from here: https://luarocks.org/modules/gaspard/yaml/1.1.2-1 which is the most downloaded version... so I guess it should work?

I mean, I can see what the error is... but I don't understand why there should be an error. How can I resolve this?... Any help appreciated :)

UPDATE

Thanks to siffiejoe's comment, I was able to compile the yaml rock. Here is the output:

D:\user\downloads\luarocks-2.3.0-win32\extras>luarocks install yaml-1.1.2-1.src.rock CC="gcc -DYAML_DECLARE_STATIC"
Using yaml-1.1.2-1.src.rock... switching to 'build' mode
gcc -DYAML_DECLARE_STATIC -O2 -c -o src/api.o -ID:/user/install/lua/luarocks/include src/api.c -Isrc
gcc -DYAML_DECLARE_STATIC -O2 -c -o src/b64.o -ID:/user/install/lua/luarocks/include src/b64.c -Isrc
gcc -DYAML_DECLARE_STATIC -O2 -c -o src/dumper.o -ID:/user/install/lua/luarocks/include src/dumper.c -Isrc
gcc -DYAML_DECLARE_STATIC -O2 -c -o src/emitter.o -ID:/user/install/lua/luarocks/include src/emitter.c -Isrc
gcc -DYAML_DECLARE_STATIC -O2 -c -o src/loader.o -ID:/user/install/lua/luarocks/include src/loader.c -Isrc
gcc -DYAML_DECLARE_STATIC -O2 -c -o src/lyaml.o -ID:/user/install/lua/luarocks/include src/lyaml.c -Isrc
gcc -DYAML_DECLARE_STATIC -O2 -c -o src/parser.o -ID:/user/install/lua/luarocks/include src/parser.c -Isrc
gcc -DYAML_DECLARE_STATIC -O2 -c -o src/reader.o -ID:/user/install/lua/luarocks/include src/reader.c -Isrc
gcc -DYAML_DECLARE_STATIC -O2 -c -o src/scanner.o -ID:/user/install/lua/luarocks/include src/scanner.c -Isrc
gcc -DYAML_DECLARE_STATIC -O2 -c -o src/strtod.o -ID:/user/install/lua/luarocks/include src/strtod.c -Isrc
In file included from src/strtod.c:23:0:
src/strtod.c:24:17: warning: '_errno' redeclared without dllimport attribute: previous dllimport ignored [-Wattributes]
 extern  int     errno;
                 ^
gcc -DYAML_DECLARE_STATIC -O2 -c -o src/writer.o -ID:/user/install/lua/luarocks/include src/writer.c -Isrc
mingw32-gcc -shared -o yaml/core.dll src/api.o src/b64.o src/dumper.o src/emitter.o src/loader.o src/lyaml.o src/parser.o src/reader.o src/scanner.o src/strtod.o src/writer.o D:/user/install/lua/luarocks/lua5.1.lib -lMSVCR80
Updating manifest for D:\user\install\lua\luarocks\systree/lib/luarocks/rocks
yaml 1.1.2-1 is now built and installed in D:\user\install\lua\luarocks\systree (license: MIT)

So it says it is installed. However when I try to use the yaml packge it does not find it:

local yaml = require("yaml")
local data = yaml.load("mytest.yaml")

The system reports that it can't find yaml.lua or yaml.dll. I then searched *.lua and *.dll in my luarocks folder I found:

D:\user\install\lua\luarocks\systree>dir *.lua *.dll /b /s
D:\user\install\lua\luarocks\systree\lib\lua\5.1\lfs.dll
D:\user\install\lua\luarocks\systree\lib\lua\5.1\mime\core.dll
D:\user\install\lua\luarocks\systree\lib\lua\5.1\socket\core.dll
D:\user\install\lua\luarocks\systree\lib\lua\5.1\yaml\core.dll
D:\user\install\lua\luarocks\systree\lib\luarocks\rocks\luafilesystem\1.6.3-2\tests\test.lua
D:\user\install\lua\luarocks\systree\share\lua\5.1\ltn12.lua
D:\user\install\lua\luarocks\systree\share\lua\5.1\mime.lua
D:\user\install\lua\luarocks\systree\share\lua\5.1\socket.lua
D:\user\install\lua\luarocks\systree\share\lua\5.1\lub\Autoload.lua
D:\user\install\lua\luarocks\systree\share\lua\5.1\lub\Dir.lua
D:\user\install\lua\luarocks\systree\share\lua\5.1\lub\init.lua
D:\user\install\lua\luarocks\systree\share\lua\5.1\lub\Param.lua
D:\user\install\lua\luarocks\systree\share\lua\5.1\lub\Template.lua
D:\user\install\lua\luarocks\systree\share\lua\5.1\socket\ftp.lua
D:\user\install\lua\luarocks\systree\share\lua\5.1\socket\http.lua
D:\user\install\lua\luarocks\systree\share\lua\5.1\socket\smtp.lua
D:\user\install\lua\luarocks\systree\share\lua\5.1\socket\tp.lua
D:\user\install\lua\luarocks\systree\share\lua\5.1\socket\url.lua
D:\user\install\lua\luarocks\systree\share\lua\5.1\yaml\init.lua

For example, I managed to install luasocket with luarocks install luasocket-2.0.2-3.win32-x86.rock because there was a .rock file available. And so you can see there is a socket.lua. For yaml, lub and luafilesystem I can see various install locations, but there is no .lua or .dll files... I searched more for yaml specifically:

D:\user\install\lua\luarocks\systree>dir *yaml* /b /s
D:\user\install\lua\luarocks\systree\lib\lua\5.1\yaml
D:\user\install\lua\luarocks\systree\lib\luarocks\rocks\yaml
D:\user\install\lua\luarocks\systree\lib\luarocks\rocks\yaml\1.1.2-1\yaml-1.1.2-1.rockspec
D:\user\install\lua\luarocks\systree\share\lua\5.1\yaml

ehh... so then I went and cloned the project from git (had to download the zip file because my stupid IT blocked my git clone ... (port 443).

So finally got that sorted, then in the yaml-master folder I tried:

luarocks make yaml-1.1.2-1.rockspec CC-"gcc -DYAML_DECLARE_STATIC"

This had the same effect as before. So now I have tried both .src.rock and .rockspec types...

I am stuck again :(

UPDATE 2

When I built from git, there was a file called yaml.lua in the top level. But all it contains is the text yaml/init.lua ... not sure what to do with that - tried copying it into the my ..install\lua\luarocks\systree\share\lua\5.1 dir, but when I run my test lua projam it gives an error.... what is going on, it can't be this hard to install :(

UPDATE 3

Following on from siffiejoe's great answer, I have more or less got this yaml installed. But I have a bug running it:

Here is my lua script:

package.path = "D:\\user\\install\\lua\\luarocks\\systree\\share\\lua\\5.1\\?\\init.lua;" .. package.path
print("package.path: " .. package.path)
local yaml = require("yaml")
local data = yaml.load("mytest.yaml")

I am manually adding the lua path for now while I am testing. When I run this I get the window:

And my log looks like:

D:\user\install\lua\luarocks\systree\share\lua\5.1\?\init.lua;D:\user\install\lua\luarocks\systree\share\lua\5.1\?.lua;D:\user\install\lua\luarocks\systree\share\lua\5.1\?\init.lua;./?.lua;./?/init.lua;./lua/?.lua;./lua/?/init.lua;D:\user\install\lua\zerobrane\lualibs/?/?.lua;D:\user\install\lua\zerobrane\lualibs/?.lua;D:\user\install\lua\zerobrane\lualibs/?/?/init.lua;D:\user\install\lua\zerobrane\lualibs/?/init.lua
D:\user\install\lua\zerobrane\bin\lua.exe: error loading module 'lfs' from file 'D:\user\install\lua\luarocks\systree\lib\lua\5.1\lfs.dll':
    The specified procedure could not be found.


stack traceback:
    [C]: at 0x66df0460
    [C]: in function 'require'
    ...\install\lua\luarocks\systree\share\lua\5.1\lub\init.lua:17: in main chunk
    [C]: in function 'require'
    ...install\lua\luarocks\systree\share\lua\5.1\yaml\init.lua:25: in main chunk
    [C]: in function 'require'
    D:\user\workspace\_lua\myluatest.lua:3: in main chunk
    [C]: at 0x00402a57
Program completed in 56.64 seconds (pid: 15856).

So it appears to come back to this MSVCR80.dll ... this is not making sense to me, I assume this is do with microsoft... Some things I have read about it could be a corrupt MSVCR80.dll that needs replacing or somthing. But I am not yet sure about why this is occuring. Any thoughts?

Probably because your LuaRocks was installed using mingw (usage of mingw-gcc in the traceback/logs), and the yaml installation is trying to use MS VCR library: -lMSVCR80 — hjpotter92
@hjpotter92 That's a good point, I had not spotted that was Microsoft file. What's that doing in there ;o .. his other libs all worked with mingw32-gcc. Any idea how to resolve this? I have MSVC2010 on my PC. edit also his instuction in his git repo say to run luarocks make (once you clone the project) - this also yields the same errors — code_fodder
@code_fodder: Try luarocks install yaml-1.1.2-1.src.rock CC="mingw32-gcc -DYAML_DECLARE_STATIC" — siffiejoe
@siffiejoe - hey that worked! ... but then when trying to use yaml I encountered further issues : (( ... can you take a look at my update to see if you can spot anything? Also since that does answer original question you can post this as an answer, I will mark it up. (btw - how did you know to use DYAML_DECLARE_STATIC?) — code_fodder

siffiejoe siffiejoe · Accepted Answer · 2016-08-18T10:20:29

The include file src/yaml.h from the github repository contains the following preprocessor code:

#ifdef WIN32
#   if defined(YAML_DECLARE_STATIC)
#       define  YAML_DECLARE(type)  type
#   elif defined(YAML_DECLARE_EXPORT)
#       define  YAML_DECLARE(type)  __declspec(dllexport) type
#   else
#       define  YAML_DECLARE(type)  __declspec(dllimport) type
#   endif
#else
#   define  YAML_DECLARE(type)  type
#endif

That means that on Windows all yaml-related functions by default are decorated with __declspec(dllimport) (the _imp__ prefixes in the linker error messages were hints). Because of this the linker expects those functions to be in an external DLL. But the yaml code is bundled and compiled together with the Lua binding into a single DLL yaml\core.dll, and so the linker complains. The correct way to fix this is to remove all import/export-related decorations from the yaml functions, and this can be achieved by defining YAML_DECLARE_STATIC. This macro should definitely be set in the rockspec (I'll post a bug report later), but a quick and dirty solution is to set it via the CC variable on the commandline:

luarocks install yaml-1.1.2-1.src.rock CC="mingw32-gcc -DYAML_DECLARE_STATIC"

After that there should be a yaml\core.dll file and a yaml\init.lua file somewhere in your LuaRocks directory. To use the Lua yaml binding you have to make sure that a require("yaml") loads the yaml\init.lua file, and that require("yaml.core") loads yaml\core.dll. This can be achieved by adding to package.path/package.cpath (from within your Lua script) or preferably by adding to LUA_PATH/LUA_CPATH. Since the installed LuaSocket already works, you probably have LUA_PATH/LUA_CPATH mostly set up correctly, but the yaml binding uses a different approach for loading its main Lua module than LuaSocket: LuaSocket has socket.lua for require("socket"), but yaml has yaml\init.lua for require("yaml"). This approach has the advantage that any Lua submodules (e.g. yaml.a) would be in the same directory as the code for the main Lua module. Anyway, you also have to add

D:\user\install\lua\luarocks\systree\share\lua\5.1\?\init.lua

to your LUA_PATH environment variable.

Btw., the yaml.lua file you found in the github repository is a symbolic link and won't work correctly on Windows anyway (but it could interfere, so you shouldn't put it into your rocks tree). It's only there for convenience during development.

Issues installing yaml for lua with luarocks during the c-compile

1 Answers