Visual recognition is what connects the abstract model to a real, running application. It answers the one question everything else depends on: what is on screen right now? The answer tells the engine which states are active, where an action's target sits, and whether a transition actually reached its destination.
Instead of reading element IDs, CSS selectors, or DOM trees, Qontinui compares screenshot images against the live screen. This makes the model independent of the application's internals — it works the same on a web app, a desktop app, or a game.
Qontinui uses template matching built on OpenCV. A small reference image (the pattern) is slid across the captured screen, and at each position the pixels are compared to produce a similarity score.
The minimum score required to count as a match. A lower threshold tolerates more variation (themes, anti-aliasing) but risks false matches; a higher threshold demands near-exact pixels. It is the central trade-off between flexibility and precision.
Restricting matching to a rectangular area of the screen makes it faster and avoids confusing a target with a look-alike elsewhere. Narrowing where to look is one of the most effective ways to make recognition both quick and reliable.
A single element can be represented by several patterns — one per theme, language, or state. The engine matches if any pattern hits, which is how a model stays robust across visual variations.
Cropping to just the distinctive element and storing it without lossy compression keeps matches fast and accurate. Large or compressed images slow the search and introduce artifacts that degrade the score.
Pure pixel matching has natural limits: heavily dynamic content, large resolution shifts, or elements whose text changes can all lower the score. Qontinui addresses these in two ways.
When exact matching fails, the engine can retry at lower thresholds and multiple scales, fall back to cached locations, or — optionally — ask a vision model to locate an element from a description. Recognition degrades gracefully rather than failing hard.
Note: This page covers the theory. For practical guidance on capturing images, tuning thresholds, and troubleshooting matches in the builder, see the Image Recognition guide.
