In our previous article, we discussed where automation is well-suited with some practical examples. Here in this article, let’s focus on discussing where automation will not be suitable and where human creativity, critical thinking, and analytical skills are best suited.

If you haven’t read the previous article where we discussed well-suited scenarios of test automation with some practical examples, please refer to the article from this link.

In this article also, we will use the same e-commerce application to discuss the practical scenarios.

Early Development Stages

When the projects are in the very early stages with changing requirements, the application flows tend to change often making the circumstances hard to gain a proper return from the automation. This will lead to lots of maintenance overhead and waste of time.

Let’s take an example from our e-commerce application. In the early days of the development of the e-commerce application, the customer registration form undergoes lots of changes from adding new fields, removing certain fields, and changing the flow of navigation several times due to changes in the requirements. It was not an ideal time to think about test automation as the application is undergoing drastic changes regularly, which means you will have to change your automated tests along with the application changes. Especially when the requirements are frequently refined, automating tests adds additional maintenance overhead due to the frequent changes it undergoes in the flow.

User Experience and UI Changes

Automated tests struggle with capturing subtle changes in the user interface and may fail to detect issues related to user experience, aesthetics, or layout changes. On the other hand, the products that are designed to be used by human beings should always be assessed by human eyes to evaluate the user experience before it is delivered to the end-user.

Below are some examples of usability validations in the e-commerce application.

• Verify the error messages are clear, understandable, and actionable. 

Bad message – Invalid Password.

Good message - Your password is too short. Please enter a password that is at least 8 characters long.

• Verify the application is mobile-friendly.

How easy and comfortable for the user to use the application on the mobile? Are the images clear enough, alignments are correct, and no overlaps? These are some of the facts you may need to evaluate in your usability test.

Considering the above examples, do you think automation can handle this? No. Test Automation is not capable of evaluating the aesthetics and easiness of the applications, or in other words, the emotions, and feelings of users.

Exploratory Testing

Automation is designed to follow predefined scripts, making it less suitable for exploratory testing where testers need to think creatively and uncover unforeseen issues. The whole concept of exploratory testing is thinking of new ways to test the software and find defects, which today's test automation is not capable of.

Look at the below examples.

• Verify the behavior of the application when checkout with an address outside of the delivery area.
• Verify the behavior of the application when the database service is unavailable.
• Verify the checkout process while the customer session has expired.

The above scenarios are not generic or common scenarios that we carry out during our regressions and will be done during an exploratory test session. On the other hand, automating these tests may offer lots of challenges.

Complex Scenarios

Highly complex and interdependent scenarios that involve multiple systems, external dependencies, or unique user paths can be challenging to automate effectively. However, in some cases, you may be able to leverage partial automation to speed up the testing and avoid human errors. Having said that, dependencies may offer unexpected challenges in gaining an advantage from the automated tests. Look at the below example.

With the use of analytics to predict customer preferences and sales growth, our e-commerce application is using data cubes to process data. It takes varying amounts of time to process data from the warehouse to formulate data cubes depending on the dataset. As testers, we need to verify the functionality of data processing and the formation of cubes. Here, we can use partial automation to validate the formation of cubes. In such a complex scenario, it is hard to depend on test automation solely.

Initial Setup Time

When it comes to short-term or one-time testing requirements, for example, data migration from one system to another, would benefit performing manually compared to automation due to the initial setup time required and further considering the time spent in automating the scenarios.

Our e-commerce application was backed by a SQL database and due to the exponential growth of the usage, the scalability became a problem for certain operations. Hence, some of the tables were migrated to a No-SQL database. As testers, now, we need to validate, if the data migration was successful, number of entries migrated was correct and randomly check a few entries in the No-SQL database. This is a one-time job and hence, these tests are not ideal for automation. 

Subjective Testing

Tests that involve subjective assessments or human judgement, such as usability testing or user feedback analysis, are not well-suited for automation. This type of testing requires an experienced tester to provide feedback that can be accurately interpreted. Automated testing cannot accurately replicate the decisions made by a human tester. Subjective tests are also highly dependent on the tester’s knowledge and experience.

The user interfaces of the e-commerce application have been updated with a new look and feel by the new UI/UX engineer who joined recently. The new theme has been constructed with lots of bright colours. As testers, we understand the problem with bright colours is users will not spend a longer time surfing the application due to the brightness of the colours. Such things can only be suggested by experienced testers. No argument, that the automation tools today cannot determine such issues.

Test automation hinders its power when it comes to handling exploratory testing, coping with rapid changes, dealing with extreme complexity, and addressing subjective assessments. 

However, using the correct test execution strategy in the right context helps organisations and teams to deliver high-quality software while optimising resources and time.