At Testim, a lot of what we do is build AI-based features on top of automation frameworks to add stability, accelerate test creation, and improve root-cause analysis.
We evaluate a number of test automation frameworks to understand their strengths and weaknesses, feature differences, and non-functional attributes.
As a new framework, Playwright reaches stable versions, it’s time to compare the popular alternatives.
I believe that we are uniquely qualified to make an evaluation because:
- We get to see hundreds of different test automation projects at Testim.
- Since we build on top of these tools (and love them) rather than compete with them – we are relatively unbiased.
- We have a lot of experience with large-scale projects as well as small scale automation projects. We’ve had companies grow with us from 10 employees to 1000 so we have a pretty good understanding of their struggles.
Disclaimer: We are two people writing this blog post, we are biased because we work a lot on infrastructure. Your experience will probably be different from ours and your mileage might vary. You should choose the best alternative given the criteria of your specific automation project.
Weight the categories to see your scores.
Here is the summary of our ratings. If you want to learn more, then read on. However, since we are all busy and impatient :-), we are giving you the table upfront.
- Customize the scores by adjusting the weight to reflect your preference (from very important to not important)
- When you are done adjusting the weights, press “calculate score” to see the weighted average scores.
[formidable id=7]
Now back to the blog, to see why we rated things as we did.
Expand Your Test Coverage
Sad realities about E2E tests
Based on survey results and similar to last year – most companies do not have automated end-to-end tests running as part of their CI process. We are at a point where most companies now have unit tests (yay!) but few have automated end-to-end tests.
Most companies we polled (85% out of 284) perform manual end-to-end tests as part of their release process. This makes releasing software significantly slower and more error-prone. Out of the remaining 15%, the vast majority of users running E2E tests use Selenium.
I believe that a big part of it is because in theory software is this nice clean thing and in practice software is a mess.
Most test automation projects like most software projects fail. This is frustrating and we have to change this. Luckily – that’s the trend and the explosion of innovation in the testing space is impacting the way we’re all writing code.
Automation Frameworks
Let’s start with the basics. An automation framework in this post is something that automates your browser by simulating user actions like clicks.
There are two primary ways automation frameworks perform clicks and user actions:
- They can execute JavaScript on the page (dispatching events on the page).
- They can use the devtools protocol (or a similar protocol in non-chromium browsers) to execute browser commands “natively” with a privileged capability.
To make a long story short, most frameworks, including Selenium, used to take the first approach – but since it was inherently flakey and problematic – they moved to the second approach.
There are quite a lot of things that happen when you perform a click. The original debugger click eventually ends up as a native operating system call:
Popular Automation Frameworks
There are four popular automation frameworks we get asked about that we’ve evaluated as infrastructure for our AI-based features like smart locators.
We will first discuss them individually and then perform a detailed comparison. We’re going to start with Selenium – it is by far the most popular framework and it’s also the most mature.
Selenium
Uses an HTTP REST JSON protocol for sending commands called the “WebDriver Protocol”. WebDriver is an open standard:
This means that with Selenium it is very easy to use any source language and any target platform. Selenium can automate a vast number of browsers including Internet Explorer, mobile browsers, and even mobile apps (by using Appium).
Since Selenium is a REST JSON API it is pretty easy to understand. Creating a session is just sending a POST request to `/session`. Performing a click is just sending a POST request to /session/:session-id/:element-id/click.
Underneath the hood – the actual automation is performed by ChromeDriver (in Chrome) which is just an http server. When ChromeDriver starts, it connects via the debugger to Chrome. Then, when the user performs a click it controls the debugger and performs a sequence of “mouse moved, mouse down, mouse up” (using the debugger command Input.dispatchMouseEvent).
WebDriver is also an open standard, so there are a lot of grid options and different ways to scale Selenium to run hundreds or thousands of tests concurrently.
This means that selenium avoids the pitfall of JavaScript event-based automation. Selenium thus has a pretty simple architecture:
Using Selenium + syntax
I would start by installing the official driver (there are some decent alternatives ):
npm install selenium-webdriver
Creating a driver and using it is easy—the syntax is verbose compared to the alternatives but still pretty straightforward:
Executive summary – Selenium
Pros:
- Runs on all browsers
- Many drivers and clients
- Dispatches clicks with a debugger
- Lots of grid options
- Can use any language like Java or Python and not just JavaScript
Cons:
- Not Bi-Directional yet because it’s an Http server. This means things like collecting network events or console logs is very hard
- Harder to set up yourself than alternatives
- Verbose API compared to some alternatives
- Oldest alternative. Although code doesn’t rust.
Cypress
Cypress is an E2E testing framework. It focuses on trying to provide a good developer experience and an integrated environment. Cypress is open-source but it is not based on open standards like WebDriver.
When we evaluated Cypress for internal use – there were a few show-stoppers for us. Clicking in Cypress works like Selenium 1 (the predecessor to Selenium WebDriver) and dispatches DOM Events Directly.
In addition, the lack of support of multiple-tabs and frames and the absence of wait-fors in frames were also problems for us. In practice, our Cypress suite was a lot less stable than the other three alternatives.
Moreover, as a maintainer of an open-source library used by Cypress, I was inclined to like them. But like code in 2013, Cypress doesn’t let you write regular JavaScript. That felt incredibly outdated IMO, even compared to the official Selenium driver.
That said, when evaluating Cypress we enjoyed the great documentation and the streamlined process.
Executive summary – Cypress
Pros:
- Good documentation
- Friendly community
- Uses libraries Benji was involved with under the hood ❤️
Cons:
- Uses the same techniques Selenium 1 switched away from for automation
- No multi-tab support
- Tests with multiple frames were very flakey when we evaluated them
- Very limited browser support
- The jQuery-based API felt very outdated
- In order to do parallelism well, you need to use vendor-locked software.
Puppeteer
Puppeteer is a popular test automation tool maintained by Google. It automates Chrome and Firefox. It is relatively simple and stable.
Fundamentally Puppeteer is an automation tool and not a test tool. This means it is incredibly popular for use cases such as scraping, generating PDFs, etc.
Puppeteer uses the same debugger protocol Selenium (well, ChromeDriver) uses to perform clicks and in practice Puppeteer (Playwright which we’ll discuss later) and Selenium, all use the same code for performing clicks. In practice, Puppeteer’s architecture looks something like this:
So, it’s just a really simple and cool wrapper over the devtools protocol (and the equivalent for Firefox).
Puppeteer also takes care of downloading Chrome for you and is generally easier to set up than Selenium for the development flow.
Using Puppeteer + syntax
npm install puppeteer
Then the syntax is pretty straightforward, modern and nice:
Puppeteer also gives you direct access to the CDP if you need it. Which can be very useful at times and in general it feels like there are fewer moving parts.
Executive summary – Puppeteer
Pros:
- Simple to set up
- Good documentation
- Installs Chrome in a working version automatically
- Thin wrapper
- Bi-Directional (events) – automating things like console logs is easy
- Maintained by Google.
- JavaScript first, so the code feels very natural
Cons:
- Limited cross-browser support—only Chrome and Firefox
- Feels like an automation framework and not a test framework—you often have to re-implement testing-related tools
- Grids (running concurrently) in production are often a challenge
- The automatic browser set up downloads Chromium and not Chrome and there are subtle differences between the two.
Playwright
Playwright is the new kid on the block. It is written by some of the same people who authored Puppeteer and it is maintained by Microsoft.
Playwright utilizes the same architecture as Puppeteer and is a thin WebSocket client. It uses a very similar syntax and language but there are a few differences—namely that Playwright supports more browsers (Safari) and that Playwright feels like a test automation tool rather than just an automation tool.
This means there are things that are easy to do with Playwright that are harder with Puppeteer:
- Selecting an element by text instead of by a CSS selector
- Working with Shadow DOM
- Waiting for elements to be available automatically
- Solid network validations and network mocking.
Those things are all possible with Puppeteer but feel natural with Playwright. Playwright still feels like infrastructure to build on, but it feels like test infrastructure and not automation infrastructure. They are also working on isolated sessions in browsers for grids which I’m not entirely a fan of but it’s definitely interesting. The syntax and installation are so similar to Puppeteer there is no need to copy/paste it again just to show it.
Executive summary – Playwright
Pros:
- Simple to set up
- Test framework stability features. When we evaluated Playwright compared to Cypress internally, Playwright consistently outperformed Cypress in terms of stability
- Installs Chrome, Firefox or WebKit (Safari) in a working version automatically
- Thin wrapper
- Bidirectional (events) – automating things like console logs is easy
- Maintained by Microsoft people with experience maintaining Puppeteer
- JavaScript first, so the code feels very natural
Cons:
- Very new so the APIs are evolving
- No support for IE11 or non-browser platforms
- Still very few integrations and tutorials
- Grids are still a challenge
- Docs and community are not as good as the others.
Key Differences
Disclaimer: This is just our PoV. It’s built based on evaluation criteria we believe are important, but biased towards tools that Testim can use as infrastructure. Your mileage, criterion, and cats might vary.
Cross-Browser
- Selenium: ✅✅✅✅ runs everywhere.
- Cypress: ❌ (Only Chrome/Firefox)
- Puppeteer: ❌ (Only Chrome/Firefox)
- Playwright: ✅✅ (Chrome/Safari/Firefox)
Multiple Tabs + Frames
- Selenium: ✅✅ (Supported with bad switch API)
- Cypress: ❌ (No real support)
- Puppeteer: ✅✅✅✅ (Intuitive API)
- Playwright: ✅✅✅✅ (Intuitive API)
Test creation speed
- Selenium: ✅ Yes (with Testim Playground / Selenium IDE)
- Cypress: ❌ (If you want us to add support for Cypress in the playground please let us know)
- Puppeteer: ✅ Yes (with Testim Playground)
- Playwright: ✅ Yes (with Testim Playground)
Note: when we polled companies – test creation speed and in particular, the creation speed of stable tests was a severely limiting factor in the success of automation projects. When evaluating tests authored with Testim that was not the case.
Trusted Actions
This criterion means dispatching events by the user agent which allows for user agent behaviors like hovers.
- Selenium: ✅ Yes
- Cypress: ❌ No support (can use Puppeteer plugin)
- Puppeteer: ✅ Yes
- Playwright: ✅ Yes
Parallelism Grids and Infrastructure
- Selenium: ✅ Yes (managed, costly) or build your own solution
- Cypress: 🤷 Only in their closed source paid cloud or build your own
- Puppeteer: ❌ Usually people build their own (will change soon)
- Playwright: ❌ Usually people build their own (will change soon)
Performance
End-to-end tests are very fast in practice but people suffer from misconceptions regarding the execution speed of Selenium tests. Typically, it’s the website or web-app that are slow and the tests end up waiting for the web app to be ready most of the time.
- Selenium: ✅ It’s fast enough, really
- Cypress: ✅ It’s fast enough, really
- Puppeteer: ✅ It’s fast enough, really
- Playwright: ✅ It’s fast enough, really
Stability
This means how often tests fail after being authored, other than when detecting a real application bug
- Selenium: ❌✅ Complex Automatic Wait For mechanism
- Cypress: ❌✅ Complex mechanism that doesn’t work with frames
- Puppeteer:❌✅ Wait fors for certain things, but have to waitFor manually for others
- Playwright: ❌✅✅ Better wait fors for certain things, but have to waitFor manually for others
In practice, waiting for elements in test infrastructure really depends on how your website or app works.
Smart(er) Locators
- Selenium: ❌ No support for selecting elements in multiple ways
- Cypress: ❌ No support for selecting elements in multiple ways
- Puppeteer❌ No support for selecting elements in multiple ways
- Playwright: ❌✅✅ Very promising start of supporting custom selector engines.
Self-Healing tests and automatically improving tests
- Selenium: ❌ No
- Cypress: ❌ No
- Puppeteer❌ No
- Playwright: ❌ No
If you are not sure what self-healing tests are check out this webinar we did.
Debugging
- Selenium: ❌✅ A bit hard to figure out all the terminology. Debugging remote grids relies on the grid provider
- Cypress: ❌✅ You’re not even writing regular JavaScript, you’re chaining promises. Makes up with DOMs
- Puppeteer: ✅ Writing and debugging JavaScript from your IDE
- Playwright: ✅ Writing and debugging JavaScript from your IDE
When we built TDK we took the same approach as Playwright and Puppeteer, and we think it’s the preferable one.
Documentation and Resources
- Selenium: ✅✅ Very large community. Many testers.
- Cypress: ✅✅ Small community but buzz – and very nice documentation.
- Puppeteer: ✅ Small community but lots of tutorials at this point
- Playwright: ✅❌Docs and tutorials out of date due to changing API. Still feels a bit experimental.
Autonomous Testing
- Selenium: ❌ No.
- Cypress: ❌ No.
- Puppeteer: ❌ No.
- Playwright: ❌ No.
How to choose?
Note that this is a guide for evaluating test infrastructure. No matter what you choose, unless you go with a managed platform you will need to spend a considerable amount of time on your test infrastructure (unsurprisingly and like any other software development project).
The biggest mistake in test automation projects we see is people don’t plan. They start writing tests and then abandon the project when it becomes unmaintainable. Like we stated earlier, most test automation projects fail and most companies perform manual QA.
If you don’t want to use JavaScript you are probably better off using Selenium anyway. The community and ecosystem size for Java and Python support are considerably smaller for all frameworks outside of Selenium. There are projects like jpuppeteer and puppeteer-sharp but they are third-party and much much smaller than the official Selenium alternatives.
If you want to use JavaScript then you can either mix Selenium and Puppeteer or use Playwright. You can’t mix Playwright and Selenium together at the moment.
Testim also created an open-source project called Root Cause to help Puppeteer and Playwright users troubleshoot their tests. We also created a couple of free tools that allow you to record a test and export code for Puppeteer or Playwright.
All three are good choices.
Just remember that writing a successful automation project is more than just infrastructure. Treat automation like any other software project. You wouldn’t write unmaintainable frontend code (intentionally :])—don’t write tests or code you can’t maintain.
Automation is exploding with new things. You can and probably should probably get Involved!
These tools are all open source. Get involved and you can help make next year’s comparison ✅ all around. Automation has a lot of tradeoffs. Test automation tools are different from each other, because of when they were developed, who developed them, and what goals they set out. There is no, one-size-fits-all automation framework, and many companies mix and match depending on their applications and needs.
There are also great commercial options, like Testim. Testim simplifies test automation by turning recorded user flows into test steps that can be configured, customized, or exported as code. We integrate with your development pipelines for event-triggered test runs, and we manage the infrastructure so you don’t have to. Start by getting a free account and try it yourself.
