Grayscale image classification with neural network on Android with OpenCV

Question

I'm working on and android app that can digitize numbers from a paper. I use native OpenCV code to find the numbers on the image. After that I want to use OpenCV's dnn module to recognise the number. A nice tutorial on creating the neural net can be found here:
https://www.youtube.com/watch?v=kFWKdLOxykE
The mnist_convnet_graph.pbtxt begins with this:

node {
  name: "conv2d_1_input"
  op: "Placeholder"
  attr {
    key: "dtype"
    value {
    type: DT_FLOAT
  }
  }
  attr {
    key: "shape"
    value {
      shape {
        dim {
          size: -1
        }
        dim {
          size: 28
        }
        dim {
          size: 28
        }
        dim {
          size: 1
        }
      }
    }
  }
}

So the input is a 28x28 grayscale image.
In the tutorial java code is used to use the neural net. However, I would like to use it in C++, because of the speed. I successfully load the model with cv::dnn::Net Dnn.readNetFromTensorflow(String model, String config); and pass the object to the NDK side. I create the input for the neural net with the following:

// The part of the image, we are interested in.
Rect roi(static_cast<int>(w), static_cast<int>(h),
             static_cast<int>(w), static_cast<int>(h));
Mat cropped(image_gray, roi);
// Resize image to 28x28.
Mat resized;
cv::resize(cropped, resized, Size(28,28));

After that, the forwarding should work:

const double IN_SCALE_FACTOR = 0.003921; // 1.0/255.0
Mat blob = dnn::blobFromImage(resized, IN_SCALE_FACTOR, Size(28,28));
net.setInput(blob);
Mat detections = net.forward();

where net is the passed cv::dnn::Net object. But the net.forward() command fails and gives:
OpenCV(3.4.5) Error: Assertion failed (inputs.size() == requiredOutputs) in virtual bool cv::dnn::experimental_dnn_34_v11::DataLayer::getMemoryShapes(const std::vector >&, int, std::vector >&, std::vector >&) const, file /build/3_4_pack-android/opencv/modules/dnn/src/dnn.cpp, line 681

I also tried:

cropping the rgb image
Mat blob = dnn::blobFromImage(resized, 1.0f, Size(28,28));
Not using blobFromImage, but net.setInput(resized); instead

but none of these led to the solutions. Anyone has a solutions for this? Any suggestion or idea will be appreciated.

Could you please provide an output from std::cout << blob.size << std::endl? — Dmitry Kurtaev
Sure. It is a 1x1x28x28 image. I don't know if it's appropriate for the input. Because in the mnist_convnet_graph.pbtxt file the order is not the same. — Marcell Molnár
1x1x28x28 is OK. It's just different data layout which OpenCV can manage. Could you please share an entire text .pbtxt file to see it? And please specify version of OpenCV you use. BTW, have you tried to import model using just a binary .pb file? — Dmitry Kurtaev
Sure. It's here: link. I use OpenCV version 3.4.5 and Android Studio 3.3. The thing is that the readNetFromTensorflow() function's first argument is the .pb file (~17,4 MB) and the second is the .pbtxt (~284 KB). — Marcell Molnár
hmm, it looks like the model is serialized in training mode (it should be in testing mode). There is a procedure called freezing which disabled all the training only nodes (there are in your graph). — Dmitry Kurtaev

Marcell Molnár Marcell Molnár · Accepted Answer · 2019-01-21T17:52:24

Okay, I managed to solve my problem.

Firstly, I realized that the .pb and .pbtxt files are in the wrong directory and getting 2 Failed to upload a file information log.

After putting the files into the right directory, I faced with problem:
error: (-215:Assertion failed) const_layers.insert(std::make_pair(name, li)).second in function 'void cv::dnn::experimental_dnn_34_v11::{anonymous}::addConstNodes(opencv_tensorflow::GraphDef&, std::map<cv::String, int>&, std::set<cv::String>&)'

As Dmitry Kurtaev suggested here, I removed the .pbtxt from the Dnn.readNetFromTensorflow. After that I got error:
OpenCV(3.4.5) Error: Unspecified error (Can't create layer "flatten_1/Shape" of type "Shape") in cv::Ptr<cv::dnn::experimental_dnn_34_v11::Layer> cv::dnn::experimental_dnn_34_v11::LayerData::getLayerInstance(), file /build/3_4_pack-android/opencv/modules/dnn/src/dnn.cpp, line 513

This led me to a link, which I found in Dmitry Kurtaev's comment here. After doing the suggested modifications (removing Const nodes, modifying and removing flatten nodes) on the .pbtxt file, finally I got no errors and got successfully running the neural net.

Note: Adding K.backend.set_learning_phase(0) before creating the model, also can be useful.

Grayscale image classification with neural network on Android with OpenCV

1 Answers