In our recent Ask Me Anything event, our community joined guest speaker, Benjamin Gruenbaum, Testim Dev Leader, to get all of their questions around Testim automation and software development answered.
The Testim Community is a free platform where developers and QA automation professionals can learn from peers, attend online events with industry leaders, and uplift their skills with the latest in testing and quality resources. Read on for more with Benji and Tristan Lombard, Community and Social Media Manager at Testim. Watch our full interview here.
Tristan Lombard
We’re having a lot of fun in the Testim community, creating casual, fun, small, intimate learning environments for you all. And we’ve got plenty ahead scheduled for 2021. I am particularly excited about today’s speaker. For those of you that don’t know Benji, he is a developer at Testim, an international speaker, an open-source community lover. He’s a node.js collaborator, core team Bluebird, Sinan, modex, and other open-source libraries.
We appreciate you being here, Benji, we’ve got a few people joining in the chat, but we’re just going to go ahead and dive in. We’ve got a lot of great pre-submitted questions today. So I want to dive into the first question. In your opinion. What do frontend developers think about end-to-end testing?
Benji Greunbaum
Most developers just don’t write end-to-end tests. What they typically do is they either don’t have end-to-end tests, or they have very few end-to-end tests. They don’t run on the PRs. [Whereas] many of the developers that I know from open source and meetups and such, do like to have end-to-end tests. And they take it very seriously. They acknowledge that unit tests are great, but they don’t catch all the bugs, and they have E2E tests.
Tristan
That’s super helpful. Next question from the community: As a developer, how do you evaluate platforms and software when making considerations for new solutions?
Benji
It’s very hard. Because the market is filled with people trying to sell you stuff. Everyone is giving you all the reasons why you should use them. And usually none of the reasons why you shouldn’t. I tend to ask friends and look at case studies, more than like GitHub stars and or download numbers…there is no replacement to the human aspect. If you are not part of the community, join one and ask.
Tristan
This is super helpful. And it’s a good follow up to our next question from the community. You published this amazing study, “Puppeteer, Selenium, Playwright, Cypress: How to Choose,” and this article has kind of been trending ever since. Can you tell the audience a little bit about the research process that you put into that and what inspired you to publish this work?
Benji
So what inspired me to publish this work? This article was a conference talk that I gave around the world, which was fun. Basically, it’s the work of a few years of reading the source code for Puppeteer, Playwright, and Selenium and working on it in testing. At Testim, we have the privilege of checking thousands of different automation projects and seeing what works and what doesn’t work. To be honest, when I joined Testim, I said, “Hey, there is this new cool Cypress thing I want to try and use it!”
However, I kept trying to use it and I kept running into walls. And eventually, we added building on top of Selenium, because it was the most stable and most portable so it’s basically just taking my experience building the TDK (Testim Development Kit) and working on a testing platform and just trying to extend it. Another part is I talked a lot to engineers working on those projects. And I am always trying to understand why they build in a certain way. The last part is talking to companies because we were building Testim Root Cause, which touches on Puppeteer and Playwright and is a free open source tool. So we talked to pretty much everyone we could find using Puppeteer, Playwright, and Cypress.
Tristan
Nice. We have another question from Kevin in the community who is wondering, as a frontend developer, would you rather use a codeless or low code tool? Or do you want to code your own tests?
Benji
My instinct as a developer is I want to use code, but your instincts can be wrong. Developers tend to really value simplicity over complexity. And learning like a low code or codeless tool can feel like more work. However, once you get familiar with these tools, they are much faster. For example, Oren, our CEO loves to challenge developers, and tells them, “Okay, pick any random website. I’ll write 10 tests with codeless solutions. You code 10 tests. Let’s see who finishes first.” And he always wins with the codeless tools, because they’re just that much faster. But it’s a nice way to put people in their place when they say, hey, our coding is much faster.
However, there are advantages of coding, like the tooling is much easier to integrate. But when we built a coded version of Testim, we found the codeless version was generally just better. There are, of course, advantages and reasons to consider the coded version. But, I think codeless shines in that it’s a smaller problem domain, you can do fewer things in a codeless version, which means it’s much easier to do things like auto grouping, or the AI features in Testim codeless versions. And just like limiting your problem domain gives you so much power and keeps the developers in bounds and doesn’t let them build the levels of abstraction that they otherwise typically build.
Tristan
Nice, nice. So one last community question before we flip over to another subject. What principles did you learn from coding that you would be able to apply to designing tests?
Benji
Everything that applies to coding applies to testing. We have this weird divide in the industry between automation engineers and developers. But it’s a very artificial divide, as they all deal with code, just different types of code. It’s a different type of automation, just like frontend programming or backend programming. It’s just the amount of code and the number of tests you have to write in order to make the project as hard as a frontend project. So everything that you do in regular code you don’t repeat yourself to try to keep as a practice, don’t write very long functions, give variables meaningful names, think about the architecture of your tests, and so on, are equally important for tests or just like any other type of code.
Tristan
That’s super insightful. We appreciate that. Pivoting over to another community question – testing in production. Your thoughts?
Benji
Testing in production is the thing that people say you should do, but it’s tricky. There are a couple of reasons. First your tests need to be meaningful and test real flows in your app. The problem is that your app has states like the status in the database and when you run your tests a lot it creates fake transactions that need to be corrected. You can’t just heal and restart your database like you do in E2E in a staging environment. You can get a corrupted state. Second, testing in production requires lots of data points so that you can get an accurate representation of quality using only a small percentage of your users. Most applications don’t have the tens of thousands of users that allow you to do this effectively. B2C applications are more likely to meet this criteria than B2B.
You can do cool things like house engineering and human services in production, which isn’t E2E testing in production. It’s testing stability in production. Cool, but I don’t consider this E2E testing.
We do run E2E testing in production at Testim. We are testing in production to test regression. We have to monitor, but it’s still not for everyone. The amount of tests that I would write for production is much smaller than the amount of testing I write for staging. However, the caveat is if you do microservices, you probably want to test one service in staging. The database that you are using against a bunch of services in production so that when you deploy that service you get the same service that you would in production. Basically, ideal testing in production is mostly monitoring the simple things. You have to be careful to not corrupt your database because you could lose all of the customers’ data.
Tristan
I like that. We have a community member that has a follow-up question.
Community Member
Yes, that’s exactly where I’m stuck right now. We want to get at least a little bit of testing done in production. But on the other hand, if we only do the very high-level stuff, we don’t really get much value out of it. And if we actually want to do something that is more like an end-to-end test case, like let’s say checkout process or something like that, then you have all the state that you have to manage. And that’s where I’m stuck right now. But I think you explained the problem very well. But I don’t think I’ve yet found a solution for this. I would love to discuss further another time.
Tristan
Absolutely. I want to make sure we leave time for our other community questions. What are three quick things that someone can do to quickly prove automation ROI wins to their boss?
Benji
Catch a bug before it reaches production and use it to explain how you can prevent bugs in production in the future. For some companies, automation is a hard sell, because it’s an investment for the future. Automation doesn’t fix bugs now, but it will prevent bugs in the future. It’s a big time-saver. Many companies won’t give you time to implement automation, which takes time. Building expertise takes time. Actually authoring the tests takes time.
So either find your five last bugs and automate them. When there is a regression, do it quietly – you don’t have to tell anyone. And when regression automation catches it, that’s where you’re suddenly the all-star. The other thing is if there is a flow that breaks a lot already, and you can automate it, I guess it’s kind of the same thing from the other angle, like finding a bug ahead of time.
Tristan
Nice! Benji, thank you for letting us have this time to network and connect with you. You’re just the best.
Benji
Thank you very much.
Tristan
Bye, y’all. We appreciate you being here and we’ll see you again soon. Take care.
About Testim
Testim is the leading AI-based end-to-end testing automation solution that gives you the speed and stability of codeless tests with the flexibility of code.
Join the Testim Community
If you are interested in connecting with industry leaders, attending live community events or networking with other developers and QA automation professionals, join us in the Testim Community.
