How to create cv::Mat from dlib::rectangle on dlib's array2d<rgb_pixel> image?

Question

Im using dlib's hog detector because it finds faces much better than opencv haarcascades. But it can not detect emotions on faces (or can???). I need extract a "sub-image" from dlib::rectangle with face, and create cv::Mat from it to call cv::detectMultiScale() with preloaded "haarcascade_smile.xml".

How to perform this extraction/conversion?

Code sample below...

int DetectionProc(void * param){ // async operation
    auto instance =(Detector *)param;
    const unsigned int delay = 1000 / instance->OPS;
    std::vector<dlib::rectangle> detects;
    array2d<rgb_pixel> sample;

    while(instance->OPS){
        unsigned int elapsed = GetTickCount();

        EnterCriticalSection(&instance->cs_buf);
        assign_image(sample,instance->buffer);
        instance->buffer.clear();
        LeaveCriticalSection(&instance->cs_buf);

        if (sample.size()){
            detects = instance->face_detector(sample);
            if (!detects.empty()){
                detection res(GetTickCount());
                if (!instance->smile_detector.empty()){
                    // TODO
                    // extract subimage from detects.front() on sample to cv::Mat face;
                    // instance->smile_detector.detectMultiScale(); on face
                    // set res.smiled to true on success
                }
                EnterCriticalSection(&instance->cs_result);
                instance->result = res;
                LeaveCriticalSection(&instance->cs_result);
            }
        }
        elapsed = GetTickCount() - elapsed;
        Sleep((elapsed < delay) ? (delay - elapsed) : 0);
    }
    return 0;
}

supervacuus supervacuus · Accepted Answer · 2019-08-24T13:09:20

There are two ways you can achieve this, either through the extract_image_chips functionality in dlib (docs) or by extracting the sub-image from a wrapping cv::Mat using the corresponding OpenCV API. Which one you will use depends on how convenient the choice is for the rest of your processing pipeline.

Concluding from your sample it seems that the OpenCV path is the most convenient (but again, please review the design and API options):

// 1.) Wrap your "sample" dlib-image in cv::Mat
//     dlib::toMat() is available through #include <dlib/opencv.h>
cv::Mat sample_mat = dlib::toMat(sample);

// 2.) Iterate through your detections
for (const auto& rect: detects) 
{
    // 3.) Extract the rectangle sub-image using OpenCV
    cv::Mat rect_sub = sample_mat(
        cv::Rect(rect.left(), rect.top(), rect.width(), rect.height()));

    // 4.) Process the sub-image
}

How to create cv::Mat from dlib::rectangle on dlib's array2d<rgb_pixel> image?

1 Answers