Test Automation has become an indispensable player in software testing over the last decade that promises efficiency, accuracy, and speed. However, like any other tool, its effectiveness solely depends on the context it is used. Understanding when test automation offers a return on investment and when it does not is essential for making decisions about its implementation. Test automation can help to reduce testing time, increase test coverage, and reduce development costs. However, it requires significant effort to set up the tests and to maintain them and may not be suitable for certain projects.
This has two parts - for the first article, let’s focus on where automation is well suited with some practical examples.
Automation is designed and well-suited to execute repetitive, routine tests. Test cases or scenarios that need to perform the same set of actions repetitively; for instance regression testing, can be efficiently automated, saving time and effort. Regressions executed via automation will offer faster feedback and leave room for testers to focus on other tasks that require human intuitiveness.
Assume you have an e-commerce application with the below tests to be verified in your regression.
Aren’t they repetitive tasks? Do you need to verify them in every iteration?
Yes, these functionalities are the core of your application and need to be tested in every iteration expecting the same result. Hence these types of repetitive tests are well-suited for automation.
When working with many test cases or a wide variety of devices, automation can ensure thorough coverage and consistency.
This avoids two aspects; unnecessary pressure in the mind of the testers due to the number of tests that need to be repeated within the given time-frame. Secondly, with repetition, actions become habits and offer a high risk of slipping defects due to unconsciousness.
Looking at our e-commerce example above, the application is expected to work across the most widely used 03 browsers and the latest and latest-1 versions of each of the browsers. Therefore, you need to repeat the same set of tests 6 times.
Let’s assume you have 400 regression tests and out of that you have selected 200 to be used for cross-browser testing. So, you will have to repeat these 200 tests 06 times in the two versions of each browser. That is going to be 1200 tests to be executed.
Remember, this is only across different browsers. In practice, tests are executed across different devices too ranging from PCs to Mobile phones to tablets. Speechless!
Yes, that’s right it is a tedious, stressful process for the tester. Don’t you think, automation is the penicillin here?
Automated tests are excellent for assessing the stability and basic functionality of an application. They can quickly verify if the application is functioning as expected after code changes. Hence, smoke and sanity checks are ideal examples of leveraging the power of automation.
Smoke or sanity checks could be treated as a subset of the regression tests, which is not a thorough test but verifies the stability of the build.
What is the value addition of automating the smoke test? The developers will have awareness of their build right away after the deployment and no waiting time for them until a test engineer attends to the task of verifying the smoke.
You can add basic CRUD operations like, login to the application, customer creation, invoice creation, etc. Set a standard criterion in selecting tests for your smoke test. What practically should happen is to cover the most important functionalities of the application in an extremely short time, touching all the components.
Imagine you want to simulate thousands of virtual users to perform your performance test. is it possible via manual testing? NO. Let’s go one step forward. Now, imagine if you had to repeat this performance test repeatedly. It’s a nightmare, isn’t it? This is where automation comes in handy. Tests that cannot be practically carried out by a human tester can be easily performed and repeated effortlessly.
You are getting ready for Black Friday and expected to reach 1 million users during the day. Practically, it is impossible to simulate 1 million users accessing the application manually even if the whole team got together including HR and Marketing. Let’s make it worse. Everything may not work as expected and the application may fail to meet the performance levels expected, where you will have to repeat the tests while fixing the issues and optimising the application performance. Can this be practically done by gathering all the members of the team? No!
This is where automation becomes our saviour. You can use performance testing tools like JMeter and automate required scenarios where performance should be measured and run on demand. The hidden jewel is you need to automate once, and you can repeat the test indefinitely unless there is a change in the flow.
Data-driven testing is where the tests are performed and validated based on the data provided and, automation is well-suited for data-driven testing scenarios, where the same set of actions are performed with multiple sets of test data. This can help to test and evaluate the application behaviour with a wide array of data combinations.
In the e-commerce application, you have the customer registration form with different types of form validation tests.
Repetitively carrying out these different combinations would be far easier to automate than testing them manually. On the other hand, when it comes to automation, even though it looks like you have different combinations of tests, it is different datasets. So, you can create a single test and run it with different datasets to validate the functionality.
Automated tests can efficiently validate an application’s compatibility not only across different browsers and devices but also in different environments such as development, stage, and production and operating systems ranging from Windows to Linux and so on. With modern-day cloud-based applications, we need to make sure the infrastructure is compatible to ensure a consistent user experience.
First, let’s talk about the deployment model of your e-commerce application. Once the developers have completed the development, they first deploy their changes to the development environment and tests are triggered, once the test execution has been completed and passed, the new build is deployed to the stage or the QA environment, where the tests are executed again to verify the environment independence. If you don’t have automated tests, each time a build is deployed to an environment, testers may have to perform the tests manually. Beyond the main environments, it is quite straightforward to manage feature environments too as you have automated tests to validate the build.
Now your e-commerce application has been gaining lots of traction and has gained third-party vendors' attention, where their requirement is to deploy in different cloud service providers like AWS or GCP. While the test combinations are skyrocketing, since we are armed with good automation coverage, we can run these tests without any issue in any cloud service provider to verify the smooth functioning of the application.
Test automation has been a game-changing asset that remarkably enhances the quality of deliverables and speed of delivery in the right contexts. It performs perfectly as explained above in repetitive tasks, large-scale testing, stability checks, performance assessments, and data-driven scenarios. Nevertheless, automation tests running in parallel frequently, help the teams to identify issues faster, saving time.
Caution! There are contexts where test automation doesn’t fit in. Let’s have a look at them in the next article.