It sounds like a simple question and answer. This blog should be complete in 30 words or less, right? Not so fast, my friend.
There are plenty of outdated and unhelpful definitions of TestOps in the blogosphere. We’ll define TestOps, including its underlying elements and some capabilities designed to help.
Let’s start by describing some testing trends and implications to illustrate the need for TestOps.
Trends
Two trends drive an increased need for end-to-end testing: growth in web applications and modernization of development practices. Let’s explain.
First, web applications continue to grow in popularity for B2B and B2C purposes. As browsers have evolved, more of the application code has shifted to the front end of applications.
Second, modern development methodologies are driving faster iteration and releases. Developers can’t throw code over the wall to be evaluated by testers without slowing delivery. With shorter, more frequent cycles, the pace of innovation is too fast to permit manual regression testing.
As a result, functional UI and E2E automated testing are gaining traction. Simultaneously, shift-left, continuous testing, and agile testing approaches bring more people into the testing discipline. Developers and product owners are now joining automation engineers and quality analysts in the testing process.
What are the implications?
More of anything typically drives up complexity. Juggling three balls is significantly more complicated than juggling one or two. In software quality, complexity comes from more tests, additional people and roles, and extra (faster) releases.
Smaller teams can typically survive through hard work and communication. Larger organizations need repeatable processes and systems to help them scale. These systems can start manually but will eventually need automation to increase productivity.
Modern software organizations seek to deliver innovation faster. Each step, from requirements gathering to delivery, needs to be smooth and efficient. You can’t have a truly agile process if you have to stop and test for a week.
TestOps is the discipline of managing and scaling test automation people, processes, and tests to maximize efficiency, delivery speed, and application quality.
What are its components?
TestOps spans different testing types (unit, integration, end-to-end) and can involve multiple tools. There are four primary areas where TestOps can drive efficiencies: planning, control, management, and insights, as shown below.
Planning should occur early in the development cycle and be an ongoing process for rapid iteration. Planning helps the team prioritize the tests, often by perceived risk, code complexity, existing coverage, or other variables. It helps them determine what to test, the test’s goals, and who will perform the test. The product and business owners, developers, and quality team members should jointly plan and document their decisions to help drive team alignment.
Control refers to how you create tests of consistently high standards. As mentioned earlier, a small team can rely on memory and oral agreements, whereas a larger team needs processes. Testing subject matter experts can guide team members to reuse existing testing components. Change control processes, including required reviews, can help standardize best practices.
Management represents the organization and administration of tests, teams, and tools. It can include coordinating who works on tests or which tests to prioritize. Naming conventions, labeling, and how you organize tests in suites and folders can help. Flaky tests not only slow you down, but they also damage the team’s confidence in them. You’ll want to manage the tests to high standards, building trust that only stable tests serve as blockers in your CI.
Insights refer to the information you derive from your testing and teams to help you better understand release readiness, project quality, and your team’s effectiveness. In addition to the operational side, leaders of the quality process also need to be mindful of potential process improvements. After all, learning and continuously improving can help streamline processes, eliminate recurring problems and save time.
What are some capabilities to help with TestOps?
Your tools, team size, skills diversity, and the size and complexity of your application will determine what capabilities you need. Here is a sample of some capabilities you should consider as part of your journey to TestOps.
Planning: You need to discuss, decide, and document your plans. Smaller teams can get by with spreadsheets or simple forms. More complex projects require project management productivity tools to help align testing with development activities. Testing teams should consider using tools familiar to the engineering team, such as JIRA or GitHub, to simplify learning and encourage engagement.
Control: Processes become more critical as you scale. Think of tests similar to your application code and treat them as IP assets. You should have the ability to prevent unauthorized changes. Typically this means requiring reviews by subject matter experts and resolving conflicts before merging changes. Pull requests, read-only master, and required reviews are well-known Git processes for change control that should be part of your testing tools.
Management: You’ll want flexibility in how you organize your tests and team. You may use labels, folders, suites, or test plans in different situations. Assigning owners to tests helps clarify responsibilities and distribute the workload. You will also want to manage the test’s status to ensure CI readiness. That is, flaky tests can stop builds, cause troubleshooting, and destroy confidence. Test Status helps you get the test stable and ready to serve as a definitive gate in your CI.
Insights: More than just reporting on whether a test passed or failed, you need to understand and validate your application’s quality—on a feature or the full application. Insights can help you evaluate how your team and software quality are progressing over time and inform process improvements that help you gain efficiencies.
What Are Some Focus Areas For TestOps?
A TestOps approach can be diverse and as such focus on different areas. We’ll now briefly walk you through some examples of focus areas for TestOps. Keep in mind that your specific approach can decide to focus more on one area than another, and this decision will weigh on the actual capabilities (see the previous heading) that you’ll need.
Version Control
The “version control” area is related to the “control” capability. Should you decide to focus on it, your TestOps approach will need strong merging and branching capabilities for your test cases. A solid version control focus is of great help in scaling your testing approach because it allows professionals and even teams to isolate their testing work when needed and then integrate it together when it’s time to do so. That way, they can play around and experiment safely, and integrate the successful experiments so the whole team can benefit.
AI
AI (artificial intelligence) is a force to reckon with when it comes to advances in software testing, and it’s strongly linked to the “insight” capability. AI-powered test automation tools can greatly help teams in producing test suites that are robust and efficient.
Visualization
Another important area of focus for TestOps—and strongly related to the previous point—is data visualization. Great insights aren’t of much use if you can’t easily access and interpret them. That’s why having comprehensive and usable dashboards is key for
Let’s also state what it’s not.
It’s not the “intersection of testing and operations.” Several blogs and articles (intentionally unreferenced) point to the need for testing and operations to collaborate better. Better collaboration is always good, but is this definition helpful? DevOps, which includes testing, already aims to improve collaboration across the devops infinity loop.
Modern, agile teams already collaborate on testing. They work together to define tests when designing code. They integrate testing into build operations, use tests to monitor critical scenarios, and leverage monitoring data to inform new tests. In summary, a modern approach to testing involves communication and collaboration across the loop, and simply saying it should happen between testing and operations is redundant.
Another definition described TestOps as running critical test scenarios in production. Indeed, there is value in identifying your most important use cases and running them frequently, as discussed above. Though, I point to the devops infinity loop and ask how is this different from monitoring?
In conclusion
Many software organizations fall short of their test automation coverage goals because they can’t keep up with frequent application changes. They add more people to help create tests. Eventually, they have a behemoth project that requires a team to manage.
TestOps is the discipline of managing testing operations to scale test coverage, people, and activities while optimizing quality and delivery speed. It extends across all testing activities to help teams better plan their testing, organize and control tests, and gain insights that drive continuous improvement.
At Testim, we are focused on helping organizations simplify UI and functional testing. Whether the user is a developer testing the new feature they wrote or a QA person building test coverage, practitioners benefit from AI-powered authoring and stability. And if you are part of the team managing testing, you can benefit from our approach to TestOps, providing features to provide control, management, and insights as you scale quality.
