Transforming software testing with cutting-edge technology

The emergence of AI creates dramatic new enterprise opportunities, but it also spawns harrowing new challenges. Software will be relied upon to perform more essential — and more complex — tasks. Software failure never really was an option, but in today’s AI-enabled systems, software bugs turn into enterprise contagions. More than ever, organizations need to make sure their software systems work correctly, every time, under all conditions. That brings software testing to the forefront of business value propositions, operations, and innovations.

QA Wolf was founded in 2019 by Jon Perl and Co-Founders Laura Cressman and Scott Wilson with the goal of automating software test coverage in a world that still relies on manual tasks and arcane testing tools, which results in grossly inadequate software quality.

QA Wolf combines a software platform with advanced AI technologies and world-class expertise to provide test-coverage-as-a-service to the world’s most demanding digital businesses. Here’s Jon’s story.

Q: How did you get into the nitty-gritty of software development?

My mom is an architect, and my dad is an electrical engineer. Both used powerful computers professionally. Because my mom worked at home, we had high-speed internet pretty early on. So, I grew up on a computer. My parents tell me stories of how I would bang my fingers on the keyboard and move the mouse with intention, rapidly clicking through different actions on the computer, before I could talk. I don’t know what it is about computers. I remember growing up and just loving them.

I’ve been coding hardcore since high school. One of my first summer jobs in high school was working as a dispatcher at a medical company near Chantilly, Virginia. I actually got the job from my sister, who was dating the owner of the company. And, of course, they broke up right before I started. Thus, I had a bit of a dilemma: Do I still work here or not? I guess you could call it my first professional crisis! After discussing the situation with each of them, they both decided they were okay with me taking the job. So, I ended up working there.

When I started, I saw that their drivers were spending a ton of time manually filling out these manifests, writing in details like addresses or the name of what they were bringing in terms of medications. This tedious task ended up costing each of their ten drivers about 30 minutes a day. So, my sister’s ex-boyfriend’s business was losing around five hours of driver time a day. It got me wondering, could I figure out how to write a program that could automatically fill out these manifests and reclaim that productivity? I did a lot of research online and figured out how to write a program that could automatically print these manifests. And it worked! I was so amazed by the fact that I could write this software — and write it pretty quickly within the constraints of a high school summer job. And I was also amazed that it would save people so much time. No one wants to be doing this sort of task. From that point on, that sort of became my mission. How can I free people from doing repetitive work?

Fast forward a few years, and right out of college I started a company called Found Ops, which created logistics software for small businesses. We made a ton of mistakes getting that off the ground, but I also learned a lot. We ended up getting acquired by Angie’s List, which was based out of Indianapolis. As part of the deal, I worked at Angie’s List for a few years on their platform team. After that gig, I went on to lead engineering at a Boston-based logistics startup, called Dispatch. I’d written some great software systems, but Dispatch had figured out how to sell small businesses software. That’s kind of what we missed at Found Ops.

Dispatch worked with home service providers, people that worked with networks of subcontractors, and we helped manage subcontract networks. So, if you have a home and need to fix something, they send a job through our platform and then a plumber would get a text message from us. You’ve got a plumber on their way. And then you could give them a review later, and from the plumber’s side they could use our software to dispatch their team to track all their work and make their lives easier.

Q: Sounds like you developed “founder initiative” early on, but was there an event that started you on the path to QA Wolf?

Oh, yeah. I had my “formative moment,” you might say.

I think it was sometime in 2016. Dispatch was just starting to scale up with our first enterprise customer. We had about 30,000 plumbers and contractors and electricians across the nation that were relying on our software to figure out where to go and what jobs to do. One day I ended up shipping a bug that broke our platform. All these small businesses that had embedded our platform into their business, trusted Dispatch for their core service operations, couldn’t receive or conduct work. They had to fall back to manual processes, like sending and receiving faxes. We were able to fix things pretty quickly, but shipping that one bug messed up not just our business, but all our customers’ businesses; they lost a lot of money that day.

I felt terrible. I felt like I’d let down all these small businesses that I’d been spending so many years of my life trying to help. Yeah, that was the day I discovered the impact that one bug can have on even a small company like Dispatch with just, like, five engineers. Our software deeply touched tens of thousands of companies who served hundreds of thousands of customers and that entire ecosystem of busy, hard-working people were utterly disrupted and nearly shut down.

But, like I said, Dispatch at the time was relatively small. Take that same type of event — shipping a bug — to the scale of millions of businesses and potentially billions of customers and you have a catastrophe like the one CrowdStrike recently faced. One bug at a global scale, but one bug that touched literally everything: hospitals being unable to take care of patients; people unable to withdraw money out of ATMs; 911 call centers going dark and taking down emergency services. I think the day of the CrowdStrike bug will go down like other seminal events in history. People will remember where they were and what they were doing when they couldn’t use their computer because of the CrowdStrike bug.

Q: But software has been around forever. How can software failures still occur?

Safety and testing are baked into building, maintenance, and management practices for critical physical infrastructure. For example, when you’re driving over a bridge, you know the bridge has been inspected. Serious experts — generally the best civil engineers around — are making sure that the bridge’s structure still has integrity; the foundation is intact, the steel is viable, loads are still within operational tolerances.

But we don’t have that sort of regulation, or even the appropriate disciplines, generally in place for software. Most software today just goes untested. And when software is being tested, testing processes often are manual and prone to error. Think about this: two-thirds of companies have less than 50% of their software automatically tested.

And, for two reasons, the problem is getting worse. First, software is increasingly interconnected, often in unanticipated and undocumented ways. If you make a change in a software system over here — where “here” is defined in terms of different features, integrations, or configurations — you can end up impacting something over “there.” Crossing all those permutations with manual software testing activities is nearly impossible. And the truth is that unless you’re really testing the complete array of execution flows, it’s very easy to ship a critical bug and have it impact huge numbers of people.

The second reason why the problem is getting worse is that AI is going to embed software in an expanded array of human activities. Now we’re talking about increasingly complex and essential software systems; systems that aren’t just process-driven, but increasingly data-driven. What happens when your AI system “hallucinates” and spits out a value or call or event that blows up everything? How do you manually test for that?

Q: OK, now I’m experiencing a cold sweat. What’s the way forward?

At Dispatch, I promised myself I would not repeat the mistake of shipping another devastating bug.

My first response was to find someone or some outfit who’s great at software quality assurance, or QA, and get them involved in my team’s engineering efforts. Makes sense, right? Let’s solve the problem with someone who knows what they’re doing. I hired a coterie of expert consultants to set up automated tests, and that went horribly. We spent a ton of money on setting up tests. Like $60,000 for 25 tests. And these tests were super slow to run. They were unreliable and didn’t scale. After a while of working with this group and seeing their results, I realized we probably needed like ten times that number of tests. To fully build out our testing requirements with that outfit would have cost our business over half a million dollars just to set up automating the tests we needed. We didn’t have that kind of cash.

Next, we tried to rely on available software testing tool sets, but that didn’t really help much either. At the time, software testing tools were very complex to install, use, and maintain; we found that we were constantly maintaining our software testing suite, instead of shipping features. We ended up hiring two software QA professionals to manually address our testing needs using incomplete software testing packages. Our QA people would go through our app every time we had to make a change. It took a minimum of two weeks to complete manual testing, but I was resolute: I wasn’t going to ship another damn bug, so we manually tested everything. And I still sweated every single release! And this story isn’t unique. That’s exactly how most companies handle software testing today: they hire a manual QA team and sweat out QA testing windows that are getting longer, more complex, and pose increasing failure risks.

Q: Anytime I hear “that’s exactly how most companies” do something, I think opportunity. Is that what led you to start QA Wolf?

Indeed, it was! As my career advanced, my network of superstar software professionals expanded. Eventually, I found myself in the company of other excellent software engineers who had become keenly aware of the software testing dilemma, including Laura, one of my co-founders. She’d been working at a startup that built a platform for delivering personalized healthcare. They had a great track record in their sector, but knew that one bug could kill their whole business. Consequently, Laura was struggling with the exact challenges I was facing, but perhaps doing so under even greater pressure.

We were thinking deeply about the problems of software testing: limited tests, complex tool kits, and outrageously expensive services that made no quality promises. This was around 2019. At the same time, the tech world was buzzing about the possibilities of machine learning algorithms and how they could be applied to new classes of complex problems. That got us thinking: Could ML, machine learning, algorithms and tools be applied to revolutionize software QA? We took about nine months to immerse ourselves in ML technology and started building the system that became the core of QA Wolf. At some point, I had to go all-in on the effort, so we moved out of expensive New York City and set up shop in cheaper Montana and pinned a “publish or shut up” date on our calendars.

In 2020, we published the first version of QA Wolf as an open-source tool. We announced it on Twitter and, somewhat surprisingly, got a flurry of “likes.” That was very refreshing. After building in the dark and not really shipping anything, discovering that people were interested in what we were creating was extremely motivating. It catalyzed the momentum we really needed. We quickly worked to transform the discussion thread into a community and started to talk with and work with and learn from our community members. It helped us avoid many of the traps that typically snag engineering-focused startups. QA Wolf was a free tool at the time, but we had this very active support chat.

In the chat we repeatedly heard two common problems: First, I don’t know how to install this thing; and second, once I install it, I’m running into problems operating it. Remember, software QA testing is super complex to begin with. Of course, if we wanted to evolve the open-source version of QA Wolf into a commercial endeavor, we had to find a way to solve these problems. Software testing tools are complex enough that you have to share screens to conduct debugging efforts, which nobody likes — especially in a domain as sensitive as software testing. We quickly realized that the solution was an online version of QA Wolf. That solves the software distribution and remote support problem, and introduces a natural opportunity for a scalable product and business. That was 2021.

Once we had QA Wolf online, we discovered that our customers found it much easier, like ten times easier, to create testing assets and run tasks. We had a ton of interest. One of the things we found really surprising was that customers loved using QA Wolf, but they ended up not having that many tests. You would think a company would have thousands of tests for their software base, but what we saw was that most — including some of our bigger customers — only had a dozen or two dozen tests. We were successfully automating only a small part of our customers’ software testing requirements. A crucial part, we found, but still sub-optimal. Going back to my original fixation: I asked myself if I was really helping companies prevent the shipping of bugs.

We realized that to really help companies make a quantum leap in software testing, an easier-to-use tool wasn’t going to be enough. We needed that tool to have high usage and great coverage across all possible software systems. Being able to support one or two dozen automated tests? That was way too few. We needed to find a way to help a company automate hundreds of thousands of tests. That’s when we decided that instead of just selling a tool, we needed to sell test coverage itself. So, in 2021, we launched test-coverage-as-a-service, and it’s taken off.

Q: Test-coverage-as-a-service is a huge value proposition. How do you stay focused, and where are you today?

We’re making huge progress on both the technology and business side of things, but we’re doing everything according to four basic values. The first is “make magic.” While we certainly want to solve this complex business problem, we realize that we’ll do so by making people successful. That’s our moment-to-moment motivation: Make this support call a success; make this architecture meeting a success; make this sales call a success. If we can string together enough of these magic moments, the big picture becomes real.

The second value is “be open.” I’ll say it again: software testing is hard! There are thousands of bad ways to do it. We’re trying to forge the best way to do it. That means there’s a lot of discovery going on at QA Wolf, a lot of “try it for the first time and let me know how it works.” We’re trying to build a culture that applauds that kind of inventive spirit, supports intelligent risk taking, and rewards decent and civil interactions, both inside and outside QA Wolf.

The third is “freedom and ownership.” We want to minimize encumbrances to getting work done. Our organization aspires to be plastic in nature. Sure, we want to be elastic and scale, but we also want to be able to quickly and cost effectively reconfigure as we learn new things and solve problems. If we can better align responsibility and accountability among our people and work requirements, we’ll be a better partner and a more profitable business.

The fourth is “deliver impact fast.” This is the cumulative value that the other three values feed into. I mean, look at where QA Wolf is today, 30 months or so after we first introduced test-coverage-as-a-service. We’ve got a marquee list of 150 customers. We’re catching over 1,000 critical bugs per month. We’re saving our customers $130 million a year compared to in-house or outsourcing alternatives. And we’re preventing critical incidents, the kind that might take down a business if left to fester, on a daily basis. That kind of success at speed depends on every function delivering impact fast. We’re creating tools and services, but we’re also creating markets, which requires great engineering, great operations, and great marketing. The truth is, we can only realize our opportunities as fast as our slowest function can operate. I can honestly say that by applying our core values to a really important, modern problem, QA Wolf is enjoying tremendous success.