Ambitious Software Teams Test in Production to Stay Ahead

Testing in production is how you launch products quicker, save costs, and make customers happier.

By focusing more on monitoring in production than on pre-launch tests, you can allocate more resources to building more features and improving customer experiences, contributing to your startup growth, and getting you closer to that exit stage if that’s your goal. Of course, it's easier in theory than in practice. Focusing on pre-release testing increases the risk of introducing bugs into your product, which could potentially worsen your user experience.

So how do you keep a balance? How do you test in production without sacrificing quality?

Implement a Risk-Based Approach to Quality Assurance

The goal of QA isn't to create a flawless system but to ensure that a product meets the business objective.

So, It doesn't make sense to aimlessly test everything in an attempt to create a perfect product, especially as a startup limited on resources. Instead, you need to identify your business objectives and prioritize your testing efforts based on them.

You can usually identify your business objectives from the user's story, which would require communicating with the product team as they define the product requirements. Identifying your business objective enables you to deduce user-centric features, then you can adopt a user-centric approach to risk assessment and test prioritization. Satisfied customers drive business growth, so you want to test for metrics that satisfy user needs and achieve your business objectives, metrics like performance, reliability, and user experiences. However, to do that effectively, you must first conduct a risk assessment. By conducting a risk assessment, you will identify risks such as financial risks, reputational risks, or operational risks that could impact your business. For instance, if a feature breaking could prevent users from making payments, you’d want to prioritize it in your testing strategy.

After conducting a risk assessment, create a risk management plan to mitigate incidents that might occur in production. One part of your risk management plan could be integrating feature flags into your systems so that you can simply turn off a new feature in production if it introduces bugs in your product.

You don't have to worry about breaking production or creating a perfect system. You simply need to know the potential risks you're taking and have plans to mitigate them if they occur.

Automate Automate Automate

“What can be automated should be automated” — can’t remember where I heard this.

Automation is one risk mitigation strategy that every startup should invest heavily in. In fact, everything that can be automated should be automated.

Automation essentially does three things. It:

• mitigates risk by reducing human error
• saves time and resources
• creates confidence in releases.

It mitigates risks by offloading critical features to automated processes and constantly checking for issues as changes are made in production. It saves time by allowing developers and testers to focus on new features rather than manual testing. And it creates confidence in releases, cutting down on development time. When new features are tested as they are added, the team can worry less about breaking the system and focus more on adding new features.

When implementing automation into your testing strategy, integrate automated tests into your CI/CD pipelines. This ensures that every time changes are made or code is pushed to your repositories, they are quickly tested and validated before release. You should also make sure your automation tests produce good documentation. Great documentation provides an ongoing understanding of the system’s state and facilitates prompt resolutions to issues by concerned stakeholders.

In essence, automating tests reduces the risks of critical features breaking when changes are made. It speeds development up since more resources can be assigned to new features being built. It also helps with confidence in releases, helping developers or the entire team be more motivated about building quicker since the consequences of things going wrong are much reduced.

To Monitor in Production is to Test in Production

Monitoring and feedback are the backbone of production testing. The goal is to automate as much of the technical testing as possible while closely monitoring user behaviour and product performance.

But you can't monitor every aspect of an application, or you’ll be overloaded with information. Instead, focus on key metrics that align with your business goal ( we already established how to identify them earlier)—metrics that will help you make actionable decisions.

For instance, if improving user experience is a key objective, you might track metrics that affect customer satisfaction, like response time, error rates, load times, and bounce rates. The challenge usually comes when it's time to choose monitoring tools. The challenge that comes with choosing a monitoring stack, you should choose a solution that's easy to implement, doesn't slow down your team, fits your budget (potentially open source, we’re on a budget here), supports scalability, and offers real-time user monitoring. Data privacy is another concern, as real-time data collection may be subject to regulations like GDPR and CCPA, depending on your users' locations.

To address integration challenges, start with a simple tool that's quick to implement, such as Open Observe. Then as your needs grow, you can transition to more comprehensive solutions like Prometheus or NetData. For privacy concerns, ensure transparency about data collection practices and maintain compliance with relevant laws.

Now you’ve started collecting data, what next? The real value of monitoring is utilizing the data to improve your product and achieve your business objectives. For example, high error rates might indicate bugs that need fixing or inconsistencies in how your code behaves for different users. It could also suggest that your server is overloaded, requiring scaling solutions. If response times are an issue, you might need to split certain functionalities into multiple services for quicker user interactions.

In addition to monitoring, you should also implement mechanisms to collect and analyze user feedback and regularly review them. This way you can gather new insights and incorporate them into your testing process.

Measure for Success

If you are not actively tracking your metric, addressing issues promptly will prove difficult.

Set up alerts for significant deviations from normal ranges, but also regularly review your overall performance data to spot gradual shifts that might not trigger immediate alerts. Testing and production or monitoring is a continuous process so continuously align your metrics with key business KPIs. If you’re focusing on user engagement, you can track metrics metrics like daily active users and feature adoption rates. Or if it’s revenue, conversion rates and customer lifetime value are great metrics to measure.

By employing a risk-based approach to testing, automating as much as you can and continuously monitoring production, you will be able to respond swiftly to user needs and remain competitive in the tech market.

References

Testing in production Cost monitoring technique: Cost effective solutions for scaling startups

What is end user experience monitoring (EUEM)?

Aligning your QA testing goals with business growth