Problems getting fastpython protobuf support

Question

I see that there is supposed to be support for creating python wrappers to the C++ generated protobuf readers.

From http://yz.mit.edu/wp/fast-native-c-protocol-buffers-from-python/ I find a simple way to turn on support by setting the environment variable:

PROTOCOL_BUFFERS_PYTHON_IMPLEMENTATION=cpp

if I do that I get the following error:

    import pyFileLib.db_proc_geo_pb2
  File "C:\sandbox\PrortoBuf\lib\pyFileLib\db_proc_geo_pb2.py", line 6, in <module>
    from google.protobuf import descriptor as _descriptor
  File "c:\python\winpython-64bit-3.6.3.0qt5\python-3.6.3.amd64\lib\site-packages\google\protobuf\descriptor.py", line 46, in <module>
    from google.protobuf.pyext import _message
ImportError: cannot import name '_message'

I see a way to extend the protoc generator to natively generate fastpython ouput: https://github.com/Cue/fast-python-pb

git clone https://github.com/Cue/fast-python-pb.git

cd fast-python-pb

python setup.py install

I get the following error when trying to run protoc with --fastpython_out:

ModuleNotFoundError: No module named 'plugin_pb2'

--fastpython_out: protoc-gen-fastpython: Plugin failed with status code 1.

It looks like this is related to: https://groups.google.com/forum/#!topic/protobuf/5Ywz8gwn9Kk

python installation of protocol buffers does not generate the google.protobuf.compiler.plugin_pb2 python file, while google.protobuf.descriptor_pb2 is explicitly generated by protobuf/python/setup.py

How do I go about making this explicit installation happen...or otherwise making protobuf performance in python acceptable?

Techniquab Techniquab · Accepted Answer · 2018-06-20T14:10:42

I found this little gem at the end of https://developers.google.com/protocol-buffers/docs/reference/python-generated

There is also a C++ implementation for Python messages via a Python extension for better performance. Implementation type is controlled by an environment variable PROTOCOL_BUFFERS_PYTHON_IMPLEMENTATION (valid values: "cpp" and "python"). The default value is currently "python" but will be changed to "cpp" in future release.

Note that the environment variable needs to be set before installing the protobuf library, in order to build and install the python extension. The C++ implementation also requires CPython platforms. See python/INSTALL.txt for detailed install instructions.

WinPython 3.6.3 distributions came with protobuf already installed, but apparently this flag was not set during the installation

use the following commands to fix this:

set PROTOCOL_BUFFERS_PYTHON_IMPLEMENTATION=cpp

python -mpip install protobuf --upgrade

Problems getting fastpython protobuf support

1 Answers