Undefined Symbol in C++ When Loading a Python Shared Library

Question

I have been trying to get a project of mine to run but I have run into trouble. After much debugging I have narrowed down the problem but have no idea how to proceed.

Some background, I am using a python script inside C++ code. This is somewhat documented on Python, and I managed to get it running very well in my basic executable. #include and a -lpython2.6 and everything was grand.

However, difficulty has arisen when running this python script from a shared library(.so). This shared library is "loaded" as a "module" by a simulation system (OpenRAVE). The system interacts with this module using a virtual method for "modules" called SendCommand. The module then starts a boost::thread, giving python its own thread, and returns to the simulation system. However, when python begins importing its modules and thus loading its dynamic libraries it fails, I assume due to the following error:

 ImportError: /usr/lib/python2.6/dist-packages/numpy/core/multiarray.so: undefined symbol: _Py_ZeroStruct 

I have run ldd on my executable and the shared library, there doesn't some to be a difference. I have also run nm -D on the file above, the _Py_ZeroStruct is indeed undefined. If you guys would like print outs of the commands I would be glad to supply them. Any advice would be greatly appreciated, thank you.

Here is the full python error:

Traceback (most recent call last):
  File "/usr/lib/python2.6/dist-packages/numpy/__init__.py", line 130, in 
    import add_newdocs
  File "/usr/lib/python2.6/dist-packages/numpy/add_newdocs.py", line 9, in 
    from lib import add_newdoc
  File "/usr/lib/python2.6/dist-packages/numpy/lib/__init__.py", line 4, in 
    from type_check import *
  File "/usr/lib/python2.6/dist-packages/numpy/lib/type_check.py", line 8, in 
    import numpy.core.numeric as _nx
  File "/usr/lib/python2.6/dist-packages/numpy/core/__init__.py", line 5, in 
    import multiarray
ImportError: /usr/lib/python2.6/dist-packages/numpy/core/multiarray.so: undefined symbol: _Py_ZeroStruct
Traceback (most recent call last):
  File "/home/constantin/workspace/OpenRAVE/src/grasp_behavior_2.py", line 3, in 
    from openravepy import *
  File "/home/constantin/workspace/rospackages/openrave/lib/python2.6/site-packages/openravepy/__init__.py", line 35, in 
    openravepy_currentversion = loadlatest()
  File "/home/constantin/workspace/rospackages/openrave/lib/python2.6/site-packages/openravepy/__init__.py", line 16, in loadlatest
    return _loadversion('_openravepy_')
  File "/home/constantin/workspace/rospackages/openrave/lib/python2.6/site-packages/openravepy/__init__.py", line 19, in _loadversion
    mainpackage = __import__("openravepy", globals(), locals(), [targetname])
  File "/home/constantin/workspace/rospackages/openrave/lib/python2.6/site-packages/openravepy/_openravepy_/__init__.py", line 29, in 
    from openravepy_int import *
ImportError: numpy.core.multiarray failed to import

Answer 1

I experienced the same problem with my application and solved it without linking python to the executable.

The setup is as follows:

Executable --links--> library --dynamically-loads--> plugin --loads--> python interpreter

The solution to avoid the ImportErrors was to change the parameters of dlopen, with which the plugin was loaded to RTLD_GLOBAL.

dlopen("plugin.so", RTLD_NOW | RTLD_GLOBAL)

This makes the symbols available to other things loaded afterwards, i.e. other plugins or the python interpreter.

It can, however, happen that symbol clashes occur, because a plugin later exports the same symbols.

Answer 2

The solution was linking the python2.6 library with my executable as well.

Even though the executable made no python calls, it needed to be linked with the python library. I assume its because my shared library doesn't pass the symbols of python library through to the executable. If anyone could explain why my executable (which loads my dynamic library at runtime, without linking) needs those symbols it would be great.

For clarification, my program model is something like: [My Executable] -(dynamically loads)-> [My Shared Library] -(calls and links with)-> [Python shared Library]

Answer 3

Check your python-headers and python's runtime. It looks like you have mix of 2.5 and 2.6 versions.

Answer 4

there's an example in openrave that shows how to build C++ shared objects that use boost python without having the application know about it:

http://openrave.org/en/coreapihtml/orpythonbinding_8cpp-example.html

search for "python" in the cmake file here:

https://openrave.svn.sourceforge.net/svnroot/openrave/trunk/src/cppexamples/CMakeLists.txt

the relevant info is:

if( Boost_PYTHON_FOUND AND Boost_THREAD_FOUND )
  find_package(PythonLibs)
  if( PYTHONLIBS_FOUND OR PYTHON_LIBRARIES )
    if( PYTHON_EXECUTABLE )
      # get the site-packages directory
      execute_process(
        COMMAND ${PYTHON_EXECUTABLE} -c "from distutils.sysconfig import get_python_lib; print get_python_lib(1)"
        OUTPUT_VARIABLE _python_sitepackage
        RESULT_VARIABLE _python_failed)
      if( ${_python_failed} EQUAL 0 )
        string(REGEX REPLACE "[\r\n]" "" _python_sitepackage "${_python_sitepackage}")
        set(PYTHON_INCLUDE_PATH ${PYTHON_INCLUDE_PATH} ${_python_sitepackage}/numpy/core/include)
      else()
        message(STATUS "failed to get python site-package directory")
      endif()
    endif()

    include_directories(${PYTHON_INCLUDE_PATH} ${OpenRAVE_INCLUDE_DIRS})
    add_library(orpythonbinding SHARED orpythonbinding.cpp)
    target_link_libraries(orpythonbinding ${OpenRAVE_LIBRARIES} ${PYTHON_LIBRARIES} ${Boost_PYTHON_LIBRARY} ${Boost_THREAD_LIBRARY})
    set_target_properties(orpythonbinding PROPERTIES PREFIX "" COMPILE_FLAGS "${OpenRAVE_CXX_FLAGS}")
    if( WIN32 )
      set_target_properties(orpythonbinding PROPERTIES SUFFIX ".pyd")
    endif()
  endif()
endif()