Shouldn't BackgroundSubtractor.apply() return a binary mask?

Question

I'm using OpenCV (wrapped in Java, but that's not important I believe).

I'm trying to detect the foreground using 10 pictures through BackgroundSubtractorMOG2. Here's code, if helps:

    Mat frame = new Mat();
    Mat result = new Mat();
    for (int i = 1; i <= 10; i++) {
        frame = imRead(...+i+...);
        subtractor.apply(frame, result, 0.05);
    }

    frame = imRead(...); //frame whose foreground I'm interested 
    subtractor.apply(frame, result, 0.05);

I expect result to be, as stated [in the docs](http://docs.opencv.org/java/org/opencv/video/BackgroundSubtractor.html#apply(org.opencv.core.Mat, org.opencv.core.Mat, double)), a mask; a binary image. However, as I discovered both visually and through code, it is not B/W and has lots of grays.

Unique values in the resulting matrix

 0    1    2    3    4    5    6    7    8    9  120  121  122  123  124  125   126  127  128  129  130  131  132  133  134  135  245  247   248  249  250  251   252  253  254  255

Casually zoomed-in picture: see the gray

How should I interpret values different than 0 or 255? What do they mean?
What is the best way to get a real mask from this? I could set a threshold and flatten to 0 down, 255 up, but which threshold is the most reasonable? 1, 128, 254?

I might add that, of course, 0s and 255s are the predominant values, but still I'm not ok with a greyscale image - I need it black and white.

@Miki

Going through OpenCV code, you should only get 0, 255, and (if shadow detection is enabled) 127. - thank you, I disabled shadow detection and I'm getting way better results. However, to disable it I need to call [this constructor](http://docs.opencv.org/java/org/opencv/video/BackgroundSubtractorMOG2.html#BackgroundSubtractorMOG2(int, float, boolean)): new BackgroundSubtractorMOG2(int history, float varThreshold, boolean shadowEnabled). Documentation is very poor (to me at least), and I don't know what to put in the first two fields. Could you help me or point me to the default values? Moreover, are these two parameters somehow linked to the 0.05 I'm using in my code? (I don't know what it is, it just happened to work good in respect to other values I have tried).

Do you perform some other operations in your code that can affect this value? - well, writing the image to a file and reading it with MATLAB. Could it be the jpg compression format? If so I'm quite surprised, the loss is huge (look at those unique values!).

Going through OpenCV code, you should only get 0, 255, and (if shadow detection is enabled) 127. I run some tests and I got only these 3 values. Do you perform some other operations in your code that can affect this value? — Miki

Miki Miki · Accepted Answer · 2015-08-20T17:12:21

The output mask will have values:

0   : background 
255 : foreground 
127 : shadow (only if shadow detection is enabled, default = true)

You can enable and disable shadow detection in the constructor:

BackgroundSubtractorMOG2(int history, float varThreshold, boolean bShadowDetection)

where:

history : Length of the history.
varThreshold : Threshold on the squared Mahalanobis distance to decide whether it is well described by the background model (see Cthr??). This parameter does not affect the background update. A typical value could be 4 sigma, that is, varThreshold=4*4=16;
bShadowDetection – Parameter defining whether shadow detection should be enabled (true or false).

The default values (set in the constructor with no arguments) are:

histoty = 500;
varThreshold = 4.0f*4.0f;

The values in the background mask are set in this line:

// bgfg_gaussmix2.cpp

mask[x] = background ? 0 :
                detectShadows && detectShadowGMM(data, nchannels, nmodes, gmm, mean, Tb, TB, tau) ?
                shadowVal : 255;

where the value of shadowVal is set equal to defaultnShadowDetection2

static const unsigned char defaultnShadowDetection2 = (unsigned char)127;

So, the values in the mask are only 0, 255, and 127 (if shadow detection is enabled).

If you saved your image in jpeg format, the compression would create those other values as compression artifacts. Save the image in a lossless format like png.

Shouldn't BackgroundSubtractor.apply() return a binary mask?

1 Answers