Understanding how Qontinui identifies UI elements visually
Qontinui uses template matching to find UI elements on screen. Instead of relying on element IDs, CSS selectors, or hardcoded coordinates, Qontinui visually compares screenshot images against the current screen to locate buttons, dialogs, and other elements.
Controls how closely the screen must match your pattern image
More tolerant of variations. Use for elements that change slightly (e.g., different themes, slight color shifts).
Balanced accuracy. Works well for most UI elements with minor anti-aliasing or compression differences.
Very strict. Only use when elements must match perfectly (e.g., pixel-perfect logos, fixed graphics).
Individual image templates within a StateImage
Each StateImage can contain multiple patterns representing different variations of the same element. This enables matching across:
Optional masks that define which pixels to compare
Masks allow you to ignore certain parts of an image during matching. White pixels (255) are compared, black pixels (0) are ignored. Use masks to:
Rectangular areas that limit where to search for images
Search regions improve performance and accuracy by restricting template matching to specific screen areas. Benefits include:
Capture just the button, icon, or label - not the entire screen or large surrounding areas. Smaller images match faster and more accurately.
PNG preserves exact pixel values without compression artifacts. JPEG compression can cause template matching failures.
If possible, capture screenshots at the same screen resolution and DPI settings where automation will execute. Enable multi-scale search if resolution varies.
Capture elements with distinctive colors, shapes, or text. Avoid generic elements that appear multiple times on screen.
If an element changes appearance (theme, language, state), capture multiple patterns and add them all to the StateImage.
Ensure screenshots are crisp and clear. Motion blur, poor lighting, or low contrast reduces matching accuracy.
Wait for animations to complete before capturing. Animated elements should be captured in their stable, final state.
Text can vary by font rendering, anti-aliasing, and locale. Consider using masks to ignore text or capture icon-only portions.
Possible Solutions:
Possible Solutions:
Possible Solutions:
Possible Solutions:
Search for images at multiple scales/resolutions. Enables resolution-independent matching but significantly slower.
Default: true
Color space for comparison: 'rgb' (most accurate), 'grayscale' (fastest), 'hsv' (lighting independent).
Default: rgb
Use edge detection preprocessing. Helps when lighting conditions vary but may reduce accuracy for color-based matching.
Default: false
Global default threshold for all images. Individual StateImages can override this value.
Default: 0.85
Note: Most users should keep default settings. Only adjust these if you're experiencing specific recognition issues or need to optimize performance.
