About this blog: We asked the community to provide examples of how their team proves the value of QA and their automation efforts in their organizations (publishing test results, sharing ROI). The bullets represent their responses. We’ve done some minimal editing to help provide clarity.
When ramping up a new test automation solution, it’s essential to gain buy-in from your teams quickly. But how do you measure and communicate the value of your impact? Fortunately, our Testim Community leaders are here to help with some examples to help you improve your test automation practices.
Strategies for Measuring ROI:
When measuring and communicating the value of test automation, having KPIs that your team agrees upon is key to your ability to scale.
- As Director of QA, our teams focus on the following metrics: percentage of flaky scripts/total scripts, UI tests vs. API tests ratio, the percentage of stories with automated tests for the acceptance criteria (BDD / ATDD), and the percentage of tests running in CI versus total tests.
- You must focus on the number of defects found by the automation suite, the percentage of coverage for key user workflows, and/or the number of new features developed with automation coverage.
- The best ROI is the decrease in time and team power required for regression testing while increasing the overall confidence in the quality. Even our emergency fixes going to the production environment are verified and deployed faster due to the help of more condensed regression suites.
- I encourage leaders to check for types of test failures (i.e., actual bugs, app changes, poorly designed tests). This measures the tool that you’re using, which can be extrapolated into maintenance time. In addition, the execution time of the suite shows initial success and is a valuable metric to monitor.
- Consider the following when establishing your metrics: regressions caught by the automation suite, number/ratio of flaky tests, how many/ratio of team members contributing to test automation.
- Consider adding another dimension such as all test failures that are handled, investigated within # days, and/or all failures are revisited before the release.
- Embrace the whole team testing approach – everybody is contributing, and test failures are broken down by application change, infrastructure failure, and/or test design.
Expand Your Test Coverage
Communicating the Value of Your Test Automation:
Communicating the value of your test automation requires a combination of tools and live communication across teams.
- Our automation has different triggers upon code push, upon integration, nightly, and self-initiated. We send emails with reports upon failures to the developer who pushed the code and key stakeholders. As a leader, you must remember that it’s culture/discipline that developers will look at your emails, and you should also consider other options for communication. Consider sending daily dashboard statuses with screenshots attached to all key stakeholders.
- Remember that when sending dashboard updates, for each red pipeline, it’s important to represent the overall status of your test failures, which is why we also add the bug numbers that are causing failures.
- Dashboard updates are essential, but we prefer not to send them automatically because sometimes failures are caused by network issues, bugs in tests, and other reasons, which do not add any value to all. Remember first to review the failures, fix everything related to tests, make sure issues reproduce, and then post the final.
- We have a dedicated channel in Slack for communicating and publishing test results, which we’ve integrated with a “testim-bot” (trade secret: it’s an email forwarder). Our automation engineers, manual testers, and developers are part of the same channel and can work together to take care of bugs as they are caught or maintain tests. We also send reports via email each sprint indicating test coverage, issues caught, and other conclusions.
- Our team has various parallel projects within the same organization, and we heavily rely on Slack in specific channels to communicate for each environment. When any job fails, we get a direct notification and can check the cause of the failure.
- To track and communicate defects discovered by automated tests, you should tag and label issues accordingly. This will help better articulate the value of automation and even have some idea about the number of defects reported otherwise.
Scaling the Value of Test Automation:
Our Testim Community provided bonus tips to help you scale your test automation.
- Clients who understand the power of automation are important to our teams. To scale, we focus on microservices functional testing, emailable reports, continuous testing in a cloud pipeline, making an automation suite robust, scalable, and independent in creating its own test data, acting on them, and tearing it down.
- ROI’s highest input is maintainability, so the less effort we spend on maintenance, the better. At the same time, we do not have flaky tests in my test suite. But we keep track of the retries and why it’s happening in the report.
- My teams have found it more useful to set our goals and then find good ways to measure our progress. Our rule for failing or flaky tests was to fix them right away, zero tolerance, though we were not always disciplined enough! I’ve also found measurements such as cycle time and story rejection rate more useful. For example, we decided to experiment with example mapping when we had many stories rejected by the PO. We hypothesized that it would cut our cycle time and rejection rate by # percentage. We could then see in a week or two if the trend was going the right way. If it wasn’t, we could try something else.
- I prefer keeping automated test-related work in the same project/board where developers prioritize all work items together. I like this approach because it visualizes all the team’s work and not just product code development. I hope I am making sense.
- Are you running manual tests that are now automated? Be sure to compare the time and people it takes to manually run your specific test suite versus the suite via automation. Let automation do the busy work for your teams and free up your time for exploratory testing.
Conclusion:
While measuring and communicating the value of your test automation can feel like a daunting task, remember that your Testim Community leaders will always be here to support you. Pro-tip? The best way to maximize your ROI is always going to be minimizing the time spent writing and designing tests. Grab a free calculator from Testim to help you measure the ROI of your automation here, and be sure to watch our webinar about justifying and realizing a strong ROI on test automation here.
Join the Testim Community
If you are interested in connecting with industry leaders, attending live community events, or networking with other developers and QA automation professionals, join us in the Testim Community.
What to read next
Test Automation Plans for 2021: Insights from the Testim Community
Connecting Tests to Your CI: The Good, the Bad, and the Ugly
