Manual vs Automated Testing: Our Strategy for Stable Bi-Weekly Releases

Before we changed our testing strategy, we released a new feature to the app store once in three months. Now, we release every two weeks. No, we didn’t completely eliminate automated tests. We simply assessed our unique startup conditions and adopted what worked for us.

The software industry is full of best practices. However, best practices don’t always account for the unique circumstances of teams or products.

Automated tests could slow down product development. And if you’re taking too long to provide value to your customer, it defeats the whole purpose, doesn’t it? Manual tests could enable you to release products quicker, but how about in six months' time? Would you still be building at your current pace? Or would adding a new feature break another?

So, to help you devise a strategy that suits your team and helps you build reliable products quickly, we’ll consider four factors on which you’ll assess your product. I’ll also briefly discuss what worked for us at Edubaloo, an Edtech startup. In the end, you should be able to devise a testing strategy that fits your circumstances.

So let’s start with the first.

Team Expertise & Practices

You might not need extensive automated tests if you have a team of experienced engineers who build quality into your product.

At Edubaloo, we focused on building quality from the start by building a modular architecture with the SOLID principle. Adding features became a breeze since we didn’t have to worry about breaking other parts of the product. Yes, we were slow in the beginning, but it paid off later. Our development velocity increased, and we released more frequently without extensive automated testing.

When a team diligently follows good coding and contribution practices, they will need fewer automated tests to release quality products. For instance, creating small pull requests makes peer reviews easier, allowing you to spot bugs easily. Setting up lint in your projects and including it as part of your checks in your CI/CD pipeline (which could also include automated tests) would sniff out common bugs that could occur due to human error or negligence.

Yes, automated tests don’t necessarily slow you down. The mistake most teams make is not knowing when to use it.

So when deciding what ratio of the manual to automated tests you want to employ or whether you even need automated tests at all, consider your team’s experience. If you’re a team of experienced engineers, maybe you don’t need extensive automated tests.

But if you’re a team of junior engineers, you had better invest in automated tests if you plan on working on the project long term. It would save you a lot of headaches later.

Which brings us to the next factor to consider.

Product Phase & Objectives

A startup’s phase and goals largely determine a suitable testing strategy. Have you achieved product fit, or are you still searching for your target audience? Are you in for the long haul, or are you looking to build quickly and get acquired?

Our goal was to build an EdTech system that could be easily refactored or repurposed to fit a wide range of learning levels, so we were in for the long road. Which is why we integrated automated tests from day one.

However, if you’re building to get acquired, you might adopt a different testing strategy. If you have you haven’t found your market, your goal would be to release as quickly as possible, which might mean cutting corners like automated tests. Your potential investors likely care more about your product being sellable than its reliability. If users are willing to buy a product that isn’t very stable, that could signal a promising product to investors. Also, if you’re working with a small team, there’s a good chance that whatever you’re building could be rebuilt faster by a company with more resources.

So, it’s simple: When you’re building for the long term, invest in automated tests, and if you’re trying to get acquired, do manual testing. Right?

Well, not exactly.

For some investors, a well-tested product means they can build without starting from scratch, and that might be a prerequisite for acquisition.

So, make sure to understand your potential investors’ expectations before deciding whether to ignore automated tests completely.

Product Complexity & Industry

If you’re working in high-stakes industries like the finance or health industry, relying solely on manual testing isn’t just impractical, it’s reckless.

At Edubaloo, we work in the edtech industry, so we can heavily rely on manual testing and still build a reliable product. The most critical area where we need to focus is protecting our users’ data, and that’s pretty much the only aspect where we ensured we had automated tests. Other than that, we rely on manual testing for most learning features.

However, in industries where you deal with people’s lives or hard-earned income, automated testing isn’t negotiable, it’s a requirement. In fact, to gain the trust of your users in industries like the blockchain industry, you need to conduct security audits and invest heavily in automated testing.

Automated tests speed up development in industries like these. You can build and integrate features quicker with greater confidence, knowing that new features won’t break existing ones.

Resource Allocation

Startups mostly run on limited resources. And you might ask, why test something that's going to get scrapped anyway?

That’s a reasonable question, and you might be right. There’s no need to set up automated tests for every feature. If a feature ends up getting scrapped, it probably wasn’t important. But that doesn’t mean you should throw automated tests out of the window. The key is risk assessment and prioritization.

Rather than viewing automated tests as an upfront investment that would slow down the team, you could see them as an ongoing process for critical features. Users don’t care about 100% test coverage, they just want their problems solved. So, assess your product and identify those features that are critical to solving your users’ problems. Write automated tests for those, and the rest can be manual tests.

By focusing your limited resources on critical features, you can release products quickly and with confidence.

Nobody Knows Your Product Better

Testing is a choice.

You know your product best, so whether you choose to do automated or manual testing is up to you. Best practices were all written based on projects people worked on. While some of these projects might be similar to yours, they weren’t built under the exact conditions as yours.

You are in the best position to determine what sort of testing strategy suits you.

So, consider the factors we’ve discussed and adopt what suits your startup and helps you achieve your goals. Whether it’s all-out manual testing, extensive automated tests, or a mix of both.

Just make sure that your decisions ensure the survival of your startup and are in the interests of your users.

References

Manual vs Automated Tests: Which is Better?

Manual Testing vs. Test Automation: A Practical Guide on Choosing the Right Approach