Image Processing: Algorithm Improvement for Real-Time FedEx Logo Detector

Question

I've been working on a project involving image processing for logo detection. Specifically, the goal is to develop an automated system for a real-time FedEx truck/logo detector that reads frames from a IP camera stream and sends a notification on detection. Here's a sample of the system in action with the recognized logo surrounded in the green rectangle.

Some constraints on the project:

Uses raw OpenCV (no deep learning, AI, or trained neural networks)
Image background can be noisy
The brightness of the image can vary greatly (morning, afternoon, night)
The FedEx truck/logo can have any scale, rotation, or orientation since it could be parked anywhere on the sidewalk
The logo could potentially be fuzzy or blurry with different shades depending on the time of day
There may be many other vehicles with similar sizes or colors in the same frame
Real-time detection (~25 FPS from IP camera)
The IP camera is in a fixed position and the FedEx truck will always be in the same orientation (never backwards or upside down)
The Fedex Truck will always be the "red" variation instead of the "green" variation

Current Implementation/Algorithm

I have two threads:

Thread #1 - Captures frames from the IP camera using cv2.VideoCapture() and resizes frame for further processing. Decided to handle grabbing frames in a separate thread to improve FPS by reducing I/O latency since cv2.VideoCapture() is blocking. By dedicating an independent thread just for capturing frames, this would allow the main processing thread to always have a frame available to perform detection on.
Thread #2 - Main processing/detection thread to detect FedEx logo using color thresholding and contour detection.

Overall Pseudo-algorithm

For each frame:
    Find bounding box for purple color of logo
    Find bounding box for red/orange color of logo
    If both bounding boxes are valid/adjacent and contours pass checks:
        Combine bounding boxes
        Draw combined bounding boxes on original frame
        Play sound notification for detected logo

Color thresholding for logo detection

For color thresholding, I have defined HSV (low, high) thresholds for purple and red to detect the logo.

colors = {
    'purple': ([120,45,45], [150,255,255]),
    'red': ([0,130,0], [15,255,255]) 
}

To find the bounding box coordinates for each color, I follow this algorithm:

Blur the frame
Erode and dilate the frame with a kernel to remove background noise
Convert frame from BGR to HSV color format
Perform a mask on the frame using the lower and upper HSV color bounds with set color thresholds
Find largest contour in the mask and obtain bounding coordinates

After performing a mask, I obtain these isolated purple (left) and red (right) sections of the logo.

False positive checks

Now that I have the two masks, I perform checks to ensure that the found bounding boxes actually form a logo. To do this, I use cv2.matchShapes() which compares the two contours and returns a metric showing the similarity. The lower the result, the higher the match. In addition, I use cv2.pointPolygonTest() which finds the shortest distance between a point in the image and a contour for additional verification. My false positive process involves:

Checking if the bounding boxes are valid
Ensuring the two bounding boxes are adjacent based on their relative proximity

If the bounding boxes pass the adjacency and similarity metric test, the bounding boxes are combined and a FedEx notification is triggered.

Results

This check algorithm is not really robust as there are many false positives and failed detections. For instance, these false positives were triggered.

While this color thresholding and contour detection approach worked in basic cases where the logo was clear, it was severely lacking in some areas:

There is latency problems from having to compute bounding boxes on each frame
It occasionally false detects when the logo is not present
Brightness and time of day had a great impact on detection accuracy
When the logo was on a skewed angle, color threshold detection worked but was unable to detect the logo due to the check algorithm.

Would anyone be able to help me improve my algorithm or suggest alternative detection strategies? Is there any other way to perform this detection since color thresholding is highly dependent on exact calibration? If possible, I would like to move away from color thresholding and the multiple layers of filters since it's not very robust. Any insight or advice is greatly appreciated!

There is one idea to filter out the false contours by shape matching, it means when you detected the purple and red contours, then you can check if the shapes (purple with purple and red with red) were matched (70%) because the logo has a fixed shape which helps you to easily detect the logo. — Bahramdun Adil
have a look at stackoverflow.com/questions/10168686/… it's basically the same problem. also note stackoverflow.com/questions/24299500/can-sift-run-in-realtime — Piglet
Color segmentation is a good start, but I think you should try using a custom trained Haar cascade. Haar features is the backbone of human face detection. You need some positive and negative samples to train the model. — ZdaR
Is the IP camera in a fixed position and looking at a one way street? The FedEx truck will show up with the logo in the same location and the same orientation - the truck will never be backwards or upside down. Those constraints simplify the problem greatly. — Stephen Meschke
@StephenMeschke the IP camera is in a fixed position and looking at the one way street like the 1st picture. Yes, the FedEx truck will always show up in that orientation, it will never be backwards or upside down. In addition, the FedEx truck will always be the "red" variation like the picture instead of the "green" ground truck — nathancy

Watchdog101 Watchdog101 · Accepted Answer · 2019-06-20T23:04:39

You might want to take a look at feature matching. The goal is to find features in two images, a template image, and a noisy image and match them. This would allow you to find the template (the logo) in the noisy image (the camera image).

A feature is, in essence, things that humans would find interesting in an image, such as corners or open spaces. I would recommend using a scale-invariant feature transform (SIFT) as a feature detection algorithm. The reason I suggest using SIFT is that it is invariant to image translation, scaling, and rotation, partially invariant to illumination changes and robust to local geometric distortion. This matches your specification.

Example of feature detection

I generated the above image using code modified from the OpenCV docs docs on SIFT feature detection:

import numpy as np
import cv2
from matplotlib import pyplot as plt

img = cv2.imread('main.jpg',0)  # target Image

# Create the sift object
sift = cv2.xfeatures2d.SIFT_create(700)

# Find keypoints and descriptors directly
kp, des = sift.detectAndCompute(img, None)

# Add the keypoints to the final image
img2 = cv2.drawKeypoints(img, kp, None, (255, 0, 0), 4)

# Show the image
plt.imshow(img2)
plt.show()

You will notice when doing this that a large number of the features do land on the FedEx logo (Above).

The next thing I did was try matching the features from the video feed to the features in the FedEx logo. I did this using the FLANN feature matcher. You could have gone with many approaches (including brute force) but because you are working on a video feed this is probably your best option. The code below is inspired from the OpenCV docs on feature matching:

import numpy as np
import cv2
from matplotlib import pyplot as plt

logo = cv2.imread('logo.jpg', 0) # query Image
img = cv2.imread('main2.jpg',0)  # target Image


# Create the sift object
sift = cv2.xfeatures2d.SIFT_create(700)

# Find keypoints and descriptors directly
kp1, des1 = sift.detectAndCompute(img, None)
kp2, des2 = sift.detectAndCompute(logo,None)

# FLANN parameters
FLANN_INDEX_KDTREE = 1
index_params = dict(algorithm = FLANN_INDEX_KDTREE, trees = 5)
search_params = dict(checks=50)   # or pass empty dictionary
flann = cv2.FlannBasedMatcher(index_params,search_params)
matches = flann.knnMatch(des1,des2,k=2)

# Need to draw only good matches, so create a mask
matchesMask = [[0,0] for i in range(len(matches))]

# ratio test as per Lowe's paper
for i,(m,n) in enumerate(matches):
    if m.distance < 0.7*n.distance:
        matchesMask[i]=[1,0]

# Draw lines
draw_params = dict(matchColor = (0,255,0),
                   singlePointColor = (255,0,0),
                   matchesMask = matchesMask,
                   flags = 0)


# Display the matches
img3 = cv2.drawMatchesKnn(img,kp1,logo,kp2,matches,None,**draw_params)
plt.imshow(img3, )
plt.show()

Using this I was able to get the following features matched as seen below. You will notice that there are outliers. However the majority of features match:

The final step would then to be to simply draw a bounding box around this image. I will link you to another stack overflow question which does something similar but with the orb detector. Here is another way to get a bounding box using the OpenCV docs.

I hope this helps!

Image Processing: Algorithm Improvement for Real-Time FedEx Logo Detector

2 Answers