Tensorflow Quantized Graph for Android

Question

I'm trying to load a quantized graph into an Android app. My BUILD file contains

deps = ["//tensorflow/core:android_tensorflow_lib",
      "//tensorflow/contrib/quantization:cc_array_ops",
      "//tensorflow/contrib/quantization:cc_math_ops",
      "//tensorflow/contrib/quantization:cc_nn_ops",
      "//tensorflow/contrib/quantization/kernels:quantized_ops"]

The additional quantization deps work for standalone C++ builds.

I can't compile with Bazel, due to a large number of errors in GEMMLOWP. What's the proper way to include gemmlowp and the quantization ops in Android?

Here is an example error:

external/gemmlowp/eight_bit_int_gemm/eight_bit_int_gemm.cc:125:13: error: 'int32_t' is not a member of 'std'
   MatrixMap<std::int32_t, ResultOrder> result(c, m, n, ldc);

This is on Ubuntu 16.04 with Bazel 0.3.0.

Here's a gist that has the output of two sequential attempts to build the package - it fails on highwayhash the first time and gemmlowp the second. https://gist.github.com/ericdanz/81b799f2e0bbb3cc462aa3c90468c71b

Ultimately got it to compile and run with liberal addition of "-std=c++11" in BUILD files for gemmlowp and highwayhash, and substitution of the android framework for the framework dependencies in the quantized ops. It produces fairly different results though, and runs about 4x slower (26-3200ms vs 6-800 ms). I'll try to do a little deeper investigation.

Sorry you're hitting problems! What host OS (e.g. Ubuntu, OS X) are you compiling on? — Pete Warden
I also did some more digging, and the author of gemmlowp had the following suggestion: "This looks possibly related to the issue that Miao Wang worked around here: github.com/google/gemmlowp/blob/master/profiling/… So my guess would be perhaps this user is targeting an old Android API version, causing STLport to be used, in which case hopefully #defining GEMMLOWP_USE_STLPORT might help?" — Pete Warden
It seems like c++11 isn't being used properly. external/gemmlowp/eight_bit_int_gemm/../public/../internal/../public/bit_depth.h:39:1: warning: scoped enums only available with -std=c++11 or -std=gnu++11 enum class RoundingMode { ^ external/gemmlowp/eight_bit_int_gemm/../public/../internal/../public/bit_depth.h:51:58: error: 'RoundingMode' is not a class or namespace static const RoundingMode kRoundingModeForSmallSizes = RoundingMode::Exact; I am using these values in the cc_binary part of my build file: copts = tf_copts() + ["-std=c++11","-O3"] — Eric D
Editing the BUILD files for gemmlowp in external (probably could have done it an easier/more repeatable way) I got it to compile - once the "-std=c++11" copt was inserted. Now the only problem is highwayhash. — Eric D
@a3nm I have put this aside to work on other parts of the system, so I can't remember exactly what I changed, but I think it was just removing documentation blocks (.Doc ) in the quantization code. — Eric D

user5597458 user5597458 · Accepted Answer · 2016-09-01T17:32:31

Here's what worked for me - it is basically a combination of all the comments from Eric D above, but I wanted to put it all in one place for someone new who comes across this problem:

Add quantized_ops in deps to libtensorflow_demo.so in the BUILD file for the Android app:

deps = ["//tensorflow/core:android_tensorflow_lib",
        "//tensorflow/contrib/quantization/kernels:quantized_ops",]

Modify the deps for quantized_ops in tensorflow/contrib/quantization/kernels/BUILD:

deps = [
        "//tensorflow/contrib/quantization:cc_array_ops",
        "//tensorflow/contrib/quantization:cc_math_ops",
        "//tensorflow/contrib/quantization:cc_nn_ops",
        "//tensorflow/core:android_tensorflow_lib",
        #"//tensorflow/core",
        #"//tensorflow/core:framework",
        #"//tensorflow/core:lib",
        #"//tensorflow/core/kernels:concat_lib_hdrs",
        #"//tensorflow/core/kernels:conv_ops",
        #"//tensorflow/core/kernels:ops_util_hdrs",
        #"//tensorflow/core/kernels:pooling_ops_hdrs",
        #"//tensorflow/core/kernels:eigen_helpers",
        #"//tensorflow/core/kernels:ops_util",
        #"//tensorflow/core/kernels:pooling_ops",
        "//third_party/eigen3",
        "@gemmlowp//:eight_bit_int_gemm",
    ],

Remove/comment out the .Doc() parts in tensorflow/contrib/quantization/ops/array_ops.cc, math_ops.cc and nn_ops.cc

Modify the deps for cc_array_ops, cc_math_ops and cc_nn_ops in tensorflow/contrib/quantization/BUILD:

deps = [
        #"//tensorflow/core:framework",
        "//tensorflow/core:android_tensorflow_lib",
    ],

Run bazel build command for Android app with --cxxopt="-std=c++11" flag.

Tensorflow Quantized Graph for Android

1 Answers