Issue while compiling Cython code with distutils

Question

I'm trying to compile Cython code and use it as Python module. I followed Cython's basic tutorial. I've made file hellowolrd.pyx:

print "Hello World"

I've made setup.py:

from distutils.core import setup
from Cython.Build import cythonize

setup(
    ext_modules = cythonize("helloworld.pyx")
)

and run:

python setup.py build_ext --inplace

But unfortunatly an error has occured:

running build_ext
building 'helloworld' extension
gcc -pthread -Wno-unused-result -Wsign-compare -DDYNAMIC_ANNOTATIONS_ENABLED=1 -DNDEBUG -O2 -g -pipe -Wformat -Werror=format-security -Wp,-D_FORTIFY_SOURCE=2 -fstack-protector --param=ssp-buffer-size=4 -g -O2 -g -pipe -Wformat -Werror=format-security -Wp,-D_FORTIFY_SOURCE=2 -fstack-protector --param=ssp-buffer-size=4 -I/usr/include/ncursesw -fPIC -I/usr/include/python3.5m -c helloworld.c -o build/temp.linux-x86_64-3.5/helloworld.o
gcc -pthread -shared -Wl,--as-needed -Wl,-z,relro -Wl,-O1 -Wl,--build-id -Wl,--enable-new-dtags build/temp.linux-x86_64-3.5/helloworld.o -L/usr/lib64 -lpython3.5m -o /home/dima/Projects/AntennaModeling/test/helloworld.cpython-35m-x86_64-linux-gnu.so
/bin/ld: skipping incompatible /usr/lib/gcc/x86_64-mageia-linux-gnu/5.4.0/../../../libpython3.5m.so when searching for -lpython3.5m
/bin/ld: skipping incompatible /lib/libpython3.5m.so when searching for -lpython3.5m
/bin/ld: skipping incompatible /usr/lib/libpython3.5m.so when searching for -lpython3.5m
/bin/ld: cannot find -lpython3.5m
collect2: error: command ld failed with exit status 1
error: command 'gcc' failed with exit status 1

Seems that gcc not satisfied with python3.5 library. I've tried to compile without distutils with following commands:

cython -a helloworld.pyx
gcc -shared -pthread -fPIC -fwrapv -O2 -Wall -fno-strict-aliasing -I/usr/include/python3.5m -o helloworld.so helloworld.c

And it worked. From know i can import helloworld from Python interpreter. But i can't figure out how to make Cython code compiled with distutils. I've tried to pass the same flags using extra_compile_args but it didn't worked.

UPD: It found out that gcc can't find 64-bit library of python so tried to explicitly show library directory. So i changed setup.py

from distutils.core import setup
from distutils.extension import Extension
from Cython.Build import cythonize

extensions = [
        Extension("helloworld", ["helloworld.pyx"],
        library_dirs = ['/usr/lib64']),
        ]
setup(
    name = 'Hello world app',
    ext_modules = cythonize(extensions),
)

and still the same output

running build_ext
building 'helloworld' extension
gcc -pthread -Wno-unused-result -Wsign-compare -DDYNAMIC_ANNOTATIONS_ENABLED=1 -DNDEBUG -O2 -g -pipe -Wformat -Werror=format-security -Wp,-D_FORTIFY_SOURCE=2 -fstack-protector --param=ssp-buffer-size=4 -g -O2 -g -pipe -Wformat -Werror=format-security -Wp,-D_FORTIFY_SOURCE=2 -fstack-protector --param=ssp-buffer-size=4 -I/usr/include/ncursesw -fPIC -I/usr/include/python3.5m -c helloworld.c -o build/temp.linux-x86_64-3.5/helloworld.o
gcc -pthread -shared -Wl,--as-needed -Wl,-z,relro -Wl,-O1 -Wl,--build-id -Wl,--enable-new-dtags build/temp.linux-x86_64-3.5/helloworld.o -L/usr/lib64 -L/usr/lib64 -lpython3.5m -o /home/dima/Projects/AntennaModeling/test/helloworld.cpython-35m-x86_64-linux-gnu.so
/bin/ld: skipping incompatible /usr/lib/gcc/x86_64-mageia-linux-gnu/5.4.0/../../../libpython3.5m.so when searching for -lpython3.5m
/bin/ld: skipping incompatible /lib/libpython3.5m.so when searching for -lpython3.5m
/bin/ld: skipping incompatible /usr/lib/libpython3.5m.so when searching for -lpython3.5m
/bin/ld: cannot find -lpython3.5m
collect2: error: command ld failed with exit status 1
error: command 'gcc' failed with exit status 1

the problem is linking (second gcc -pthread..) and not compiling (on the command line you are compiling, not linking) it looks like you have a wrong type of libraries (32bit, wrong architecture?) — ead
@ead, oh my, i thought i checked this, but i've mistaken. it seems my system has two python libraries for 32-bit architecture and 64-bit architecture. And by default it checks 32-bit librarry. I'll try to change path to lib in distutils. Thanks! — Zhu
@ead. There is path to 64-bit libarary but linker doesn't say anything about it in the output. — Zhu
I think cython uses something like /usr/bin/python-config --ldflags and this probably returns a wrong path — ead
@ead, in the output frome above there is gcc flags "-L/usr/lib64 -lpython3.5m" and it is exactly where library is. — Zhu

ead ead · Accepted Answer · 2017-09-22T09:41:39

Normally, the linker searches for a library with name libXXX.so and not libXXX.so.1.0 (and if not successful, then for library with name libXXX.a). Usually libXXX.so is a symlink to libXXX.so.Y.0 which is the real library.

You can watch the linker working by calling the link-command with -Wl,--verbose, e.g.:

gcc -Wl,--verbose -pthread -shared -Wl,--as-needed -Wl,-z,relro -Wl,-O1 -Wl,--build-id -Wl,--enable-new-dtags build/temp.linux-x86_64-3.5/helloworld.o -L/usr/lib64 -lpython3.5m -o /home/dima/Projects/AntennaModeling/test/helloworld.cpython-35m-x86_64-linux-gnu.so

and you will see something like:

...
attempt to open /usr/lib64/libpython3.5m.so failed
...

So there are two options:

A: introduce symlink /usr/lib64/libpython3.5m.so on your system.

Or less intrusive B: add libraries=[':libpython3.5m.so.1.0'] to your definition of extension:

extensions = [
        Extension("helloworld", ["helloworld.pyx"],
        library_dirs = ['/usr/lib64']),
        libraries=[':libpython3.5m.so.1.0']
        ]

Now linking command should look a little bit different:

 gcc ... -L/usr/lib64 -l:libpython3.5m.so.1.0 ...

And the linker will be looking for libpython3.5m.so.1.0 and not only libpython3.5m.so which is still provided by distutils.

Issue while compiling Cython code with distutils

1 Answers