A mobile app evolves over time as its look and feel is regularly updated. A button may move to a different location, colors may be updated, and text may be changed. But that button being moved–is it the same button? This question is crucial when testing a mobile app, as the target elements (buttons, fields, headings, images, etc.) that were recorded will change over time. In legacy testing approaches, you may expect tests like this to fail until they’re rewritten or re-recorded, but modern mobile testing calls for tests to be stable, dynamically accounting for changes without the need to re-record steps.
Using machine learning to determine if the target element is the same
Testim’s new Machine Learning Locators use machine learning models, which have been trained with tens of thousands of examples, to determine if the target element is the same. Whether the element changes or ceases to display on the screen, these new Locators ensure your tests remain stable without the need for any user intervention or editing of the target element.
An insider’s look at the machine learning model
As with any machine learning solution, the ML model needs to be trained with a large dataset consisting of thousands of examples. Part of this training calls for the machine learning model to analyze a set of features of the original target element captured in the recording, then comparing it against a set of features of the target element during the execution of the test, known as test playback. These set of features are part of the screen’s DOM, also referred to as locators. Examples of locators include an element’s Id, its text, its class, and more. The system adds a variety of pre-calculated features to give additional context to the target element and its location on screen.
Comparing the original recording to its playback takes the text, images, and other proprietary data of specific applications and yields a new set of aggregate data points that indicate the similarities or differences of each feature. This “anonymized comparative” data, which effectively only includes numbers, is fed to the model during its training. This ensures the privacy of our clients and creates a generalized model that can infer the behavior of target elements across virtually any scenario.
The model uses this anonymized data to analyze the entire screen to find the target element on the screen. This is achieved by analyzing each element to measure if they exceed certain thresholds. This means that the ML model can not only decide if an existing element is different, but also call out if an element does not exist on the screen at all (e.g. if someone has accidentally deleted it).
A robust machine learning model that considers a vast number of data points
The training enables the machine learning model to consider a vast amount of data points across the entire screen, including generated calculated contextual data points, in a weighted manner. The ML model develops a deep understanding of the expected or typical behaviors of different types of elements and assigns different weights to its data points accordingly. For example, “list items” typically grow over time, which mean their relative on-screen location should weigh less. Traditional rule-based algorithms would fail here, since it is nearly impossible to write rules that consider such a vast number of attributes in a weighted manner. A machine learning model, however, continues to improve over time as it undergoes more training.
Adding optional manual fine-tuning of thresholds
Testim Mobile also offers the ability to fine tune the algorithm’s threshold manually to taste through the UI as follows:
- High threshold – target elements found only when the confidence score is high
- Medium threshold – target elements found only when the confidence score is medium
- Low threshold – target elements found, even if the confidence score is low
Each level has its pros and cons. A low threshold, for example, may yield false positives. Conversely, a high threshold high level may discount your target element. We wanted to give you this flexibility, so you can customize your thresholds to fit your needs.
Availability
Testim Mobile’s new Machine Learning Locators will be available to all users on paid plans. These will be enabled by default, but you may switch back to the traditional “Fallback locators mode,” if you prefer. For more information, see https://help.testim.io/docs/editing-target-element-properties-copy.
Experience the ML locator’s power and share your thoughts with us 💬
Enjoy the ride & supercharge your mobile testing!
