Engagio is a two-year-old marketing software startup that helps marketers land and expand their target accounts. The company is rapidly growing with more than 150 customers and 45 employees. Based in San Mateo, Engagio was founded by Jon Miller who was previously the Co-Founder of Marketo.
Today, I had the pleasure of speaking with Helge Scheil, VP of Engineering for Engagio who shared why he selected Testim, his overall development philosophy and the results his team was able to achieve. Check out the series of videos or read the Q&A from our conversation below
- Engagio’s Evaluation of Test Automation
- Why Engagio Selected Testim
- Engagio’s Development Philosophy
- Engagio’s Toolset and Evolving Processes
- The Results Testim Delivered
- 40 Tests in a 1 Day Blitz
Q: Can you tell us a little about what your software does?
A: We help B2B marketers drive new business and expand relationships with high-value accounts. Our marketing orchestration software helps marketers create and measure engagement in one tool, drive ongoing success, and measure impact easily. Engagio orchestrates integrated account-based programs, providing the scale benefits of automation with the personalization benefits of the human touch. Our software complements Salesforce and existing marketing automation solutions.
Q: What does your development process look like?
A: Our developers work in 2-week sprints, developing features rapidly and deploying to production daily without any production downtime. We’re running on AWS and have the entire develop-test-build-deploy process fully automated. Each developer can deploy at any time, assuming all quality criteria have been met.
Q: What tools do you use to support your development efforts?
A: We are using Atlassian JIRA and Confluence for product strategy, roadmaps, requirements management, work breakdown, sprint planning, and release management. We’re using a combination of Codeship, Python (buildbot), Docker, Slack, JUnit and Testim for our continuous build/test/deploy. We have integrated Testim into our deployment bot (which is integrated into Slack).
Q: Prior to Testim, how were you testing your software? What were some of the challenges?
A: Our backend had great coverage with unit and integration tests, including the API level. On the front-end, we had very little coverage and the small amount we had was written in Selenium, which was very time-consuming with little fault-tolerance and many “flaky” failures.
Q: What were some things you tried to do to solve these challenges?
A: We were trying to simply allocate more time for Selenium tests to engineers. We considered hiring automation engineers but weren’t too fond of the idea because we believe in engineers being responsible for quality out of the gate, including the regression tests creation and maintenance.
Q: Were there other solutions you evaluated? Why did you select Testim?
A: We didn’t really consider any other solutions after we started evaluating Testim, which was our first vendor to evaluate. Despite some skepticism around the “record and play” concept, we found that Testim’s tolerance (meaning “tests don’t fail”) to UI and feature changes is much greater than we had expected. Other solutions that we considered are relying on pixels/images/coordinates which are inherently sensitive and non-tolerant to changes. We also found that non-engineers (i.e. product manager, functional QA engineers) can write tests, which is unlike Selenium.
Q: After selecting Testim, can you walk me through your getting started experience?
A: During the first week of implementation the team got together to learn the tool and started creating guinea pig tests which evolved into much deeper tests with more intricate feature testing. Those test ended up in our regression test suite which are ran nightly. Rather than allocating two days per sprint, we decided to crank up the coverage with a one day blitz.
Q: After using Testim, what were some of the benefits you experienced?
A: We were able to increase our coverage by 4-5x within 6 weeks of using Testim. We can write more tests in less time and maintenance is not as time-consuming. We integrated Testim via its CLI and made “run test <label>” commands available in our “deployment” Slack channel as well as a newly created “regression_test” channel. Any deployment that involves our web app now automatically runs our smoke tests. In addition to that, we run nightly full regression tests. Running four cloud/grid Testim VMs in parallel we’re able to run our full regression test suite in roughly 10 minutes.
Q: How was your experience working with Testim’s support team?
A: Testim’s responsiveness was extraordinary. We found ourselves asking questions late in the evenings and over the weekends and the team was there to help us familiarize ourselves with the product. If it wasn’t immediately in the middle of the night, we would have the answer by the time we got started the next day.
