Example explanation-09-template_matching template matching

Video tutorial 8 - NCC template matching:https://singtown.com/learn/49598/

This routine is 09-Feathre_detection-template_matching\ The goal of this routine is to implement template matching using NCC (Normalized product correlation algorithm).

# NCC模板匹配示例-Normalized Cross Correlation (NCC)
#
# 这个例子展示了如何使用OpenMV的NCC功能将小部分图像与图像的各个部分
# 进行匹配...期望获得极其可控的环境 NCC并不总是有用的。
#
# 警告：NCC支持需要重做！到目前为止，这个功能需要做大量的工作才能有用。
# 这个脚本将重新表明功能的存在，但在目前的状态是不足的。

import time, sensor, image
from image import SEARCH_EX, SEARCH_DS
#从imgae模块引入SEARCH_EX和SEARCH_DS。使用from import仅仅引入SEARCH_EX, 
#SEARCH_DS两个需要的部分，而不把image模块全部引入。

# 重置传感器
sensor.reset()

# 设置传感器
sensor.set_contrast(1)
sensor.set_gainceiling(16)
# Max resolution for template matching with SEARCH_EX is QQVGA
# 模板与SEARCH_EX匹配的最大分辨率是QQVGA
sensor.set_framesize(sensor.QQVGA)
# 你可以设置windowing窗口来减少搜索图片。
#sensor.set_windowing(((640-80)//2, (480-60)//2, 80, 60))
sensor.set_pixformat(sensor.GRAYSCALE)

# 加载模板。
# 模板应该是一个小的(例如。32x32像素)灰度图像。
template = image.Image("/template.pgm")

clock = time.clock()

#运行模板匹配
while (True):
    clock.tick()
    img = sensor.snapshot()

    # find_template(template, threshold, [roi, step, search])
    # ROI: 感兴趣区域元组 (x, y, w, h).
    # Step:使用的循环步长(y+= Step, x+= Step) 使用更大的步长使其更快。
    # search 为image.SEARCH_EX进行详尽搜索，或者为image.SEARCH_DS进行菱形搜索
    #
    # Note1: ROI必须比图像小，比模板大。
    # Note2:在菱形diamond搜索中，step和ROI都被忽略。
    r = img.find_template(template, 0.70, step=4, search=SEARCH_EX) #, roi=(10, 0, 60, 60))
    # find_template(template, threshold, [roi, step, search]),
    # threshold中的0.7是相似度阈值,roi是进行匹配的区域（左上顶点为（10，0），长80宽60的矩形），
    # 注意roi的大小要比模板图片大，比frambuffer小。
    # 把匹配到的图像标记出来
    if r:
        img.draw_rectangle(r)

    print(clock.fps())

Note that since the size of our template image is larger than openmv's built-in flash, we need to do the following after plugging in the sd card. (Note that insert sd card first and then power on it)

And this template matching can only be used for firmware version 1.6 and later, otherwise, the prompt "can not find SEARCH_EX" will be displayed when running

First, we need to create or import a template. Note that this template must be in pgm format and has a size limit that cannot exceed the pixel size of openmv. We can directly capture a template image from openmv. You can first run the helloworld.py routine to let frambuffer display the image, and then capture it.

Select save image selection to pc. Note that the image directly captured and saved from openmv is in bmp format. We need to convert it to pgm format. You can do online conversion on this website https://convertio.co/zh/bmp-pgm/

Then, we save the converted pgm template to the sd card (there are 8 templates in this sd card, and the template in the above picture is saved as ball0.pgm)

Then we open the template matching routine

Change the template file name template.pgm in line 28 to ball0.pgm just now

And run it directly!

This is the usage of the template matching function find_template:

r = img.find_template(template, 0,7, roi=(10,0,80,60), step=4, search=SEARCH_EX) The 0.7 in threshold is the similarity threshold, and roi is the matching area (the upper left vertex is (10, 0), the length is 80 and the width is 60). Note that the size of roi should be larger than the template image and smaller than frambuffer.

Note that this template matching uses the ncc algorithm, which can only match areas similar to the template size. If you want to match pictures of different sizes, you need to save multiple templates of different sizes.

If the following problems occur after running the program:

1. The template image is too large, it is recommended that the template image is smaller than 80*60

1. OpenMV2 does not have enough memory, change QQVGA to QQCIF

Singtown Technology OpenMV official Chinese document function explanation:

Comparison of template matching and feature point detection:

• Template matching (find_temolate) uses the ncc algorithm, which can only match patterns that are basically the same size and angle as the template image. The limitations are relatively large. If the target pattern in the field of view is slightly larger or smaller than the template image, the match may not be successful.

• Template matching is suitable for situations where the distance between the camera and the target object is determined and dynamic movement is not required. For example, it is suitable for the detection of specific objects on the assembly line, but not for the car to track a moving volleyball (because the distance between the moving volleyball and the camera is dynamic, and the size of the volleyball seen by the camera will change, and it will not be exactly the same as the template image).

• For multi-angle and multi-size matching, you can try to save multiple templates and use Multiple template matching.

• Feature point detection (find_keypoint): If you just started running the program, the routine extracts the first image as the target object feature, and kpts1 saves the target object feature. By default, it will match multiple scales and angles of the target feature, rather than just saving the size and angle of the target feature. It is more flexible than template matching and does not need to save multiple template images like multi-template matching.

• Feature point detection can also save the target features in advance. This was not recommended before because interference from ambient light and other reasons may cause different features in different light each time the program is run, and the matching degree will be reduced. However, in the latest version of the firmware, the adjustment of exposure, white balance, and automatic gain value has been added. The exposure value and white balance value can be manually defined, which will relatively reduce the interference of light. You can also try to save the target features in advance.