Android OpenCV VideoCapture::retrieve(&mat) causes fatal signal 11

Question

So I'm trying to make an android application, that would open a video file frame by frame with OpenCV througn JNI and NDK, detect faces in each frame and display them (the end goal is to acclerate the algorithm, to better suit crappy mobile processing).

First method I attempted was MediaMetadataRetriever with no results, it just returned null.

The second method was FFMpegMediaMetadataRetriever, which does work as expected, but is very slow (up to 2fps; too slow), partially because you have to convert from FFMpegMMR's bitmap to Mat then detect and draw and then convert back to bitmap, which is very caveman of me to even attempt.

The third method I'm currently working on is the VideoCapture grab() and retrieve(). I made a wrapper for the native code, mostly by copying the facedtector from OpenCV samples. I've also tried read() that supposedly combines the two, but it also causes fatal signal 11 (segmentation fault at some god forsaken level of OpenCV or Android platform).

Here's how the application get the absolute file path (note that it did work for FFMpegMMR):

    //The file name, file path and filerawresource are used in getting the path of the video file on device's disk
String videoFileName="samplemp4";
String videoFileType=".mp4";
int videoFileRawResourceId=R.raw.samplemp4;

public String getVideoFilePath()
{
    InputStream is = getResources().openRawResource(videoFileRawResourceId);
    File sampleDir = getDir(videoFileName, Context.MODE_PRIVATE);
    File sampleFile= new File(sampleDir, videoFileName+videoFileType);
    try {
    FileOutputStream os = new FileOutputStream(sampleFile);

    byte[] buffer = new byte[4096];
    int bytesRead;
    while ((bytesRead = is.read(buffer)) != -1) {
        os.write(buffer, 0, bytesRead);
    }

        is.close();
        os.close();
    } catch (IOException e) {
        // TODO Auto-generated catch block
        e.printStackTrace();
    }
    File videoDir = getDir(videoFileName, Context.MODE_PRIVATE);
    File videoFile = new File(videoDir, videoFileName+videoFileType);
    return videoFile.getAbsolutePath();
}

This is where the java wrappers of the native methods are called and results used:

class AsyncPlay extends AsyncTask<String, Mat, Bitmap>
        {

        @Override
        protected Bitmap doInBackground(String... params) {
            //for(;play;)
            {
                if(play)
                {
                    if(currentTime==0)
                    {
                        test=new Test();
                        test.startTime.setToNow();
                        test.type="Video";
                        frameGrabber.open(videoFilePath);
                    }
            //publishProgress(retriever.getFrameAtTime(currentTime*1000+111,
                    //      FFmpegMediaMetadataRetriever.OPTION_CLOSEST));


                    Mat tmp=new Mat();
                    //frameGrabber.read(tmp);
                    frameGrabber.grab();
                    frameGrabber.retrieve(tmp);
                    publishProgress(tmp);

                    currentTime+=111;
                    if(currentTime*1000>=duration*1000)
                    {
                        currentTime=0;
                        test.endTime.setToNow();
                        tester.publishResult(test);
                        frameGrabber.release();
                    }
                }
                try 
                {
                    Thread.sleep(111);
                } catch (InterruptedException e) 
                {
                    // TODO Auto-generated catch block
                    e.printStackTrace();
                }
            }
            return null;
        }
        @Override
        protected void onProgressUpdate(Mat... values) {

            //face detection logic goes here
            //Bitmap bmp=values[0];

            //Mat rgbMat = new Mat();
            //Utils.bitmapToMat(bmp, rgbMat);

            DetectionResult detectionResult = openCVFaceDetector.detectFromImage(values[0], Imgproc.COLOR_RGB2GRAY);//Detecting with a native detector from OpenCV

            test.addDetection(detectionResult.detection);

            imageViewOriginal.setImageBitmap(detectionResult.detectedImage);

            super.onProgressUpdate(values);
        }

    };`

Where FrameGrabber frameGrabber is the java wrapper of the natives and the signal gets sent when retrieve() or read() are called, note that the constructor, open() and grab() all work (or at least don't crash the application).

This is the wrapper (I have passed the VideoCapture object properly, which I learned was one of the causes of the fatal signal):

import org.opencv.core.Mat;

public class FrameGrabber
{ 
    private long mNativeObj = 0;
    private static native long nativeCreateObject(String fileName);
    private static native void nativeDestroyObject(long thiz);
    private static native boolean nativeOpen(long thiz, String fileName);
    private static native boolean nativeGrab(long thiz);
    private static native boolean nativeRetrieve(long thiz, Mat imageMat);
    private static native boolean nativeRead(long thiz, Mat imageMat);
    private static native void nativeRelease(long thiz);    
    public FrameGrabber(String fileName) {
        mNativeObj = nativeCreateObject(fileName);
    }
    public void destroy() {
        nativeDestroyObject(mNativeObj);
        mNativeObj = 0;
    }
    public boolean open(String fileName)
    {
        return nativeOpen(mNativeObj, fileName);
    }  
    public boolean grab()
    {
        return nativeGrab(mNativeObj);
    }   
    public boolean retrieve(Mat imageMat)
    {
        return nativeRetrieve(mNativeObj, imageMat);
    }    
    public boolean read(Mat imageMat)
    {
        return nativeRead(mNativeObj, imageMat);
    }    
    public void release()
    {
        nativeRelease(mNativeObj);
    }
}

And this is it's native part (the exact line causing the error is result = ((VideoCapture*)thiz)->retrieve((*((Mat*)imageMat)));, which is in the method nativeRetrieve()

#include <string>
#include <vector>
#include <opencv2/core/core.hpp>
#include <opencv2/highgui/highgui.hpp>
#include <FrameGrabber_jni.h>

#include <android/log.h>

#define LOG_TAG "FrameGrabber"
#define LOGD(...) ((void)__android_log_print(ANDROID_LOG_DEBUG, LOG_TAG, __VA_ARGS__))

using namespace std;
using namespace cv;

inline void jStringToString(JNIEnv * jenv, jstring jString, string stdName)
{
    LOGD("Java_boris_springar_diplomska_FrameGrabber_jStringToString enter");
    LOGD("grabber is Making jnamestr");//this is buggy as hell, the line serves as a debug tool
    const char* jnamestr = jenv->GetStringUTFChars(jString, NULL);
    string stdNameTmp(jnamestr);
    stdName=stdNameTmp;
    LOGD("grabber is releasing jnamestr");
    jenv->ReleaseStringUTFChars(jString, jnamestr);
}

/*
 * CONSTRUCTOR AND DESTRUCTOR
 */
JNIEXPORT jlong JNICALL Java_boris_springar_diplomska_FrameGrabber_nativeCreateObject
(JNIEnv * jenv, jclass, jstring jFileName)
{
    LOGD("Java_boris_springar_diplomska_FrameGrabber_nativeCreateObject enter");
    string stdFileName;
    jStringToString(jenv, jFileName,stdFileName);
    jlong result = 0;

    try
    {
        result = (jlong)new VideoCapture(stdFileName);
    }
    catch(cv::Exception& e)
    {
        LOGD("nativeCreateObject caught cv::Exception: %s", e.what());
        jclass je = jenv->FindClass("org/opencv/core/CvException");
        if(!je)
            je = jenv->FindClass("java/lang/Exception");
        jenv->ThrowNew(je, e.what());
    }
    catch (...)
    {
        LOGD("nativeCreateObject caught unknown exception");
        jclass je = jenv->FindClass("java/lang/Exception");
        jenv->ThrowNew(je, "Unknown exception in JNI code of FrameGrabber.nativeCreateObject()");
        return 0;
    }

    LOGD("Java_boris_springar_diplomska_FrameGrabber_nativeCreateObject exit");
    return result;
}

//should work
JNIEXPORT void JNICALL Java_boris_springar_diplomska_FrameGrabber_nativeDestroyObject
(JNIEnv * jenv, jclass, jlong thiz)
{
    LOGD("Java_boris_springar_diplomska_FrameGrabber_nativeDestroyObject enter");
    try
    {
        if(thiz != 0)
        {
            delete (VideoCapture*)thiz;
        }
    }
    catch(cv::Exception& e)
    {
        LOGD("nativeestroyObject caught cv::Exception: %s", e.what());
        jclass je = jenv->FindClass("org/opencv/core/CvException");
        if(!je)
            je = jenv->FindClass("java/lang/Exception");
        jenv->ThrowNew(je, e.what());
    }
    catch (...)
    {
        LOGD("nativeDestroyObject caught unknown exception");
        jclass je = jenv->FindClass("java/lang/Exception");
        jenv->ThrowNew(je, "Unknown exception in JNI code of FrameGrabber.nativeDestroyObject()");
    }
    LOGD("Java_boris_springar_diplomska_FrameGrabber_nativeDestroyObject exit");
}


/*
 * CORE METHODS
 */

//Open function opens the filename for playing
JNIEXPORT jboolean JNICALL Java_boris_springar_diplomska_FrameGrabber_nativeOpen
(JNIEnv * jenv, jclass,jlong thiz,jstring jFileName)
{
    LOGD("Java_boris_springar_diplomska_FrameGrabber_open enter");
    string stdFileName;
    jStringToString(jenv, jFileName,stdFileName);

    jboolean result = false;

    try
    {
        result = ((VideoCapture*)thiz)->open(stdFileName);
    }
    catch(cv::Exception& e)
    {
        LOGD("frame grabber open exception caught cv::Exception: %s", e.what());
        jclass je = jenv->FindClass("org/opencv/core/CvException");
        if(!je)
            je = jenv->FindClass("java/lang/Exception");
        jenv->ThrowNew(je, e.what());
    }
    catch (...)
    {
        LOGD("frame grabber open caught unknown exception");
        jclass je = jenv->FindClass("java/lang/Exception");
        jenv->ThrowNew(je, "Unknown exception in JNI code of FrameGrabber.open()");
        return 0;
    }

    LOGD("Java_boris_springar_diplomska_FrameGrabber_open exit");

    return result;
}

//grab grabs the next frame from file or camera
JNIEXPORT jboolean JNICALL Java_boris_springar_diplomska_FrameGrabber_nativeGrab
(JNIEnv * jenv, jclass,jlong thiz)
{
    LOGD("Java_boris_springar_diplomska_FrameGrabber_grab enter");
    jboolean result = false;

    try
    {
        result = ((VideoCapture*)thiz)->grab();
    }
    catch(cv::Exception& e)
    {
        LOGD("frame grabber grab exception caught cv::Exception: %s", e.what());
        jclass je = jenv->FindClass("org/opencv/core/CvException");
        if(!je)
            je = jenv->FindClass("java/lang/Exception");
        jenv->ThrowNew(je, e.what());
    }
    catch (...)
    {
        LOGD("frame grabber grab caught unknown exception");
        jclass je = jenv->FindClass("java/lang/Exception");
        jenv->ThrowNew(je, "Unknown exception in JNI code of FrameGrabber.grab()");
        return 0;
    }

    LOGD("Java_boris_springar_diplomska_FrameGrabber_grab exit");

    return result;
}

//retrieve retrieves the next frame and writes it to the image matrix
JNIEXPORT jboolean JNICALL Java_boris_springar_diplomska_FrameGrabber_nativeRetrieve
(JNIEnv * jenv, jclass,jlong thiz, jlong imageMat)
{
    LOGD("Java_boris_springar_diplomska_FrameGrabber_retrieve enter");
    jboolean result = false;

    try
    {
        LOGD("grabber trying to retrieve");
        result = ((VideoCapture*)thiz)->retrieve((*((Mat*)imageMat)));//should write the current frame to the image matrix
    }
    catch(cv::Exception& e)
    {
        LOGD("frame grabber retrieve exception caught cv::Exception: %s", e.what());
        jclass je = jenv->FindClass("org/opencv/core/CvException");
        if(!je)
            je = jenv->FindClass("java/lang/Exception");
        jenv->ThrowNew(je, e.what());
    }
    catch (...)
    {
        LOGD("frame grabber retrieve caught unknown exception");
        jclass je = jenv->FindClass("java/lang/Exception");
        jenv->ThrowNew(je, "Unknown exception in JNI code of FrameGrabber.retrieve(fileName)");
        return 0;
    }

    LOGD("Java_boris_springar_diplomska_FrameGrabber_retrieve exit");

    return result;
}

//read combines grab and retrieve and writes the stuff to the image matrix
JNIEXPORT jboolean JNICALL Java_boris_springar_diplomska_FrameGrabber_nativeRead
(JNIEnv * jenv, jclass,jlong thiz, jlong imageMat)
{
    LOGD("Java_boris_springar_diplomska_FrameGrabber_read enter");
    LOGD("grabber setting result to false");
    jboolean result = false;

    try
    {
        LOGD("grabber trying to read capture");
        result = ((VideoCapture*)thiz)->read((*((Mat*)imageMat)));//should write the current frame to the image matrix
    }
    catch(cv::Exception& e)
    {
        LOGD("frame grabber read exception caught cv::Exception: %s", e.what());
        jclass je = jenv->FindClass("org/opencv/core/CvException");
        if(!je)
            je = jenv->FindClass("java/lang/Exception");
        jenv->ThrowNew(je, e.what());
    }
    catch (...)
    {
        LOGD("frame grabber read caught unknown exception");
        jclass je = jenv->FindClass("java/lang/Exception");
        jenv->ThrowNew(je, "Unknown exception in JNI code of FrameGrabber.open(fileName)");
        return 0;
    }

    LOGD("Java_boris_springar_diplomska_FrameGrabber_read exit");

    return result;
}

//Release releases the resource it's using, I hope
JNIEXPORT void JNICALL Java_boris_springar_diplomska_FrameGrabber_nativeRelease
(JNIEnv * jenv, jclass,jlong thiz)
{
    LOGD("Java_boris_springar_diplomska_FrameGrabber_release enter");
    jboolean result = false;

    try
    {
        ((VideoCapture*)thiz)->release();//should release
    }
    catch(cv::Exception& e)
    {
        LOGD("frame grabber read exception caught cv::Exception: %s", e.what());
        jclass je = jenv->FindClass("org/opencv/core/CvException");
        if(!je)
            je = jenv->FindClass("java/lang/Exception");
        jenv->ThrowNew(je, e.what());
    }
    catch (...)
    {
        LOGD("frame grabber release caught unknown exception");
        jclass je = jenv->FindClass("java/lang/Exception");
        jenv->ThrowNew(je, "Unknown exception in JNI code of FrameGrabber.open(fileName)");
    }

    LOGD("Java_boris_springar_diplomska_FrameGrabber_release exit");
}

And also the LogCat output:

07-22 17:36:24.886: D/FrameGrabber(15359): Java_boris_springar_diplomska_FrameGrabber_nativeCreateObject enter
07-22 17:36:24.886: D/FrameGrabber(15359): Java_boris_springar_diplomska_FrameGrabber_jStringToString enter
07-22 17:36:24.886: D/FrameGrabber(15359): grabber is Making jnamestr
07-22 17:36:24.886: D/FrameGrabber(15359): grabber is releasing jnamestr
07-22 17:36:24.886: D/FrameGrabber(15359): Java_boris_springar_diplomska_FrameGrabber_nativeCreateObject exit
07-22 17:36:24.936: I/dalvikvm(15359): threadid=3: reacting to signal 3
07-22 17:36:24.936: I/dalvikvm(15359): Wrote stack traces to '/data/anr/traces.txt'
07-22 17:36:24.946: D/libEGL(15359): loaded /system/lib/egl/libEGL_mali.so
07-22 17:36:24.956: D/libEGL(15359): loaded /system/lib/egl/libGLESv1_CM_mali.so
07-22 17:36:24.976: D/libEGL(15359): loaded /system/lib/egl/libGLESv2_mali.so
07-22 17:36:25.006: D/OpenGLRenderer(15359): Enabling debug mode 0
07-22 17:36:27.336: D/FrameGrabber(15359): Java_boris_springar_diplomska_FrameGrabber_open enter
07-22 17:36:27.336: D/FrameGrabber(15359): Java_boris_springar_diplomska_FrameGrabber_jStringToString enter
07-22 17:36:27.336: D/FrameGrabber(15359): grabber is Making jnamestr
07-22 17:36:27.336: D/FrameGrabber(15359): grabber is releasing jnamestr
07-22 17:36:27.336: D/FrameGrabber(15359): Java_boris_springar_diplomska_FrameGrabber_open exit
07-22 17:36:27.336: D/FrameGrabber(15359): Java_boris_springar_diplomska_FrameGrabber_grab enter
07-22 17:36:27.336: D/FrameGrabber(15359): Java_boris_springar_diplomska_FrameGrabber_grab exit
07-22 17:36:27.336: D/FrameGrabber(15359): Java_boris_springar_diplomska_FrameGrabber_retrieve enter
07-22 17:36:27.336: D/FrameGrabber(15359): grabber trying to retrieve
07-22 17:36:27.336: A/libc(15359): Fatal signal 11 (SIGSEGV) at 0x1d400019 (code=1)

The problem with debugging this code is that it's JNI in Eclipse and I don't know how to even make debugging c++ in Eclipse work, that's why I used the log messages to find the bugs in the code. Those helped immensely, as open() and grab() didn't work and I found and squashed a bug, where I forgot to pass thiz to the methods. The problem with retrieve() is that I can not for the life of me find it's source. There's partial definitions in highgui.hpp, but no implementation, where I could put the log messages to help me debug.

Possible solutions would be:

• a different file format that's supported by retrieve(), except that I don't know which file format that might be
• use a non absolute file path? Although neither open() and grab() did not cause a fatal signal
• scrap the whole thing and use a PNG (I really want to figure this out though)

So if anyone could tell me where I can find the implementations of retrieve() and read(), I'd be most grateful. Or if it's something obvious and stupid that I missed (I wish).

In the meantime, I'll try to find a way to debug c++ in Eclipse and maybe try another format?

Answer 1

Found it, finally, following Aleksander's suggestion, I managed to find the type mismatch on object that were passed from Java to JNI.

Turns out, that the error was in the java part, where the native methods were being declared. Instead of passing Mat, i had to pass Mat.getNativeObjAddr(), which is a long.

So instead of

    private static native boolean nativeRetrieve(long thiz, Mat imageMat);
    private static native boolean nativeRead(long thiz, Mat imageMat);
    public boolean retrieve(Mat imageMat)
    {
        return nativeRetrieve(mNativeObj, imageMat);
    }    
    public boolean read(Mat imageMat)
    {
        return nativeRead(mNativeObj, imageMat);
    }    

I used this:

        private static native boolean nativeRetrieve(long thiz, long imageMat);
        private static native boolean nativeRead(long thiz, long imageMat);
        public boolean retrieve(Mat imageMat)
        {
            return nativeRetrieve(mNativeObj, imageMat.getNativeObjAddr());
        }    
        public boolean read(Mat imageMat)
        {
            return nativeRead(mNativeObj, imageMat.getNativeObjAddr());
        }   

NOTE: passing the wrong type DID NOT trigger an exception in c++, even though the code was in try...catch. The try...catch might be written wrong, I don't know.