With the increasing adoption of Agile, Scrum, and DevOps methodologies, the demand for automated testing and CI/CD has grown significantly. As the field of automated testing has matured, the test automation pyramid has gained popularity, replacing outdated models like the ice cream cone. Additionally, E2E testing has faced criticism due to its tendency to be flaky.

Amidst these ongoing debates, it is essential to recognize that the key to successful test automation lies in consistent and sustainable test execution. Ideally, tests should produce accurate results without false negatives or false positives. In this article, we will delve into the significance of maintaining unchanged test cases, a practice that is not only relevant to automated testing but also extends to manual tests.

In this article, we'll cover:

• Why it's crucial to keep test cases unchanged
• The benefits of maintaining unchanged test cases

However, we won't be discussing:

• Solutions for flaky test issues
• Non-functional tests

Changing a Test Case: Four Categories

So, how should we approach changes to test cases? In this context, we're referring to test cases written in natural programs or languages with specific expected values and proper confirmation procedures. Changing test cases can be challenging and frustrating, especially when addressing flaky tests. The ideal solution is to refrain from changing any test cases unless there are changes to the test target itself.

To understand this better, let's explore the four types of fixes carried out by developers and their relationship to test cases: 

1. Refactoring: This involves maintenance without changing any functions to improve code-based test targets' performance and readability. The goal is to ensure that existing functions (mainly automated ones) in the test case still work properly. Therefore, the test case itself shouldn't be changed.
2. Addition of New Functions: Adding new functions shouldn't affect existing functions. All existing test cases (mainly automated ones) should yield satisfactory results, meaning the test case doesn't need to be changed. 
3. Fixing of Defects: Test case insufficiency often leads to issues. Similar to adding new functions, existing functions shouldn't be affected, and existing test cases (mainly manual ones) don't need to be changed either. Only additional test cases need to be added to address the insufficiencies.
4. Specification Changes: Specification changes involve minor modifications to a single function in an operational system. These changes are not backward compatible, meaning they alter the functions that users rely on. The impact can be significant, and in such cases, the expected values in existing test cases should be updated.

In summary, we ideally expect that there is no need to change test cases following refactoring, adding new functions, and fixing defects. Test case changes should only be made for specification changes.

But how can we strive for this ideal? How can we create test cases that require minimal maintenance? The answer lies in avoiding unnecessary details whenever possible.

Unit Tests

Let me introduce one of the techniques to avoid delving into the 'details (methods)' in unit tests. You may already be familiar with it, and that is testing using public APIs.

Why is this the case? The key to understanding lies in the definition of public APIs. Public APIs are APIs that are freely accessible to users and third parties. As mentioned earlier, unless there are changes in their behavior, public APIs remain unchanged, making the test cases written for them theoretically stable.

On the other hand, private APIs are relatively easy to change due to their internal usage within the team and their role in supporting system functions. As such, APIs are prone to become targets for refactoring, for example, and test cases relying on private APIs tend to be fragile.

Web/Mobile E2E Tests

We've established that using public APIs, which are visible to end users and third parties, is preferable for testing. This suggests that E2E tests, in a simplified sense, are desirable. Undoubtedly, a sustainable web/mobile E2E test that requires no maintenance is commendable. It builds confidence within your team when you can rely on a test to determine if everything is "okay" without encountering false positives or false negatives under the same conditions as real users.

As a developer at a company focusing on the development and operation of E2E testing automation services for web/mobile, we celebrate the rise of web/mobile E2E tests. However, caution is necessary, because running web/mobile E2E tests is a constant battle against flakiness. When it comes to E2E tests, scenarios that are not written in test cases also tend to occur, and one of the reasons for this is "the Web is a detail" (this phrase was popularized by Robert C. Martin A.K.A Uncle Bob, who wrote it in his book Clean Architecture: A Craftman's Guide to Software Structure and Design).

What do I mean by "the Web is a detail"? I'm referring to the fact that the Web is a means to an end. The reason why so many services on the Web are created is mostly usability. Buying specialized hardware or going to a service vendor's data center to use a service is unthinkable in this day in age. And since software engineers account for a very small percentage of the world's population, it's not practical to expect users to use command prompts, shells, and terminals. Therefore, the user experience is greatly improved by the ability to access services through a Web browser from personal computers. The Web is thus a means to achieve usability. 

However, since the Web is simply a means to an end, there are many ways to leverage it to serve one purpose. For example, even if the layout of an application changes, the application will still serve the same purpose as before. The problem, when it comes to testing, is that even minor changes like shifting button positions or altering IDs for improved layout can impact E2E testing. The same applies to mobile testing as well. To avoid getting caught up in the detail of the Web, testing with web APIs has gained popularity. Web APIs help bypass specific web details like screen layout.

Regarding Manual Tests

Does the practice of avoiding unnecessary details only apply to automated tests? Not at all. The technique is also recognized in manual tests. Two notable examples are exploratory tests and concrete logical tests. Concrete logical tests strike a balance between abstractness and concreteness, depending on the test case's desired level of detail. While there are trade-offs, maintaining test cases becomes crucial for regression tests. In such cases, writing techniques that are independent of test case changes are key.

Regarding Web/Mobile E2E Test Automation Services

We all understand that web/mobile E2E tests are prone to flakiness. Although E2E tests provide the best fidelity (i.e. how tests reproduce real user's interaction), they inevitably face challenges due to their reliance on various aspects related to the test subject. A popular function provided by test automation services is self-healing, which ensures test execution continues even when test target elements cannot be found. For example, changes in button positions or IDs no longer pose significant issues. This approach aims to avoid human interference since completely avoiding details is extremely difficult.

Summary: Striving for Unchanged Test Cases

In conclusion, the key to achieving successful test automation lies in ensuring consistent and sustainable test execution. A fundamental aspect of this is maintaining unchanged test cases, which greatly contributes to the reliability and efficiency of the testing process. This principle holds true for both automated and manual testing, emphasizing the importance of avoiding unnecessary details. While the level of abstraction in test cases may vary depending on the operational context and the testing perspective of your team, test cases written with appropriate details offer significant advantages for continuous execution. By embracing this principle, we can ensure that our test cases withstand the test of time and consistently provide accurate results.

Author's Note: The sections "Changing a Test Case: Four Categories" and "Unit Tests" were partly inspired by the book Software Engineering at Google: Lessons Learned from Programming Over Time which offers valuable insights based on Google's unit tests.

The featured image for this article was created by Manuel Will on Unsplash.