Context-Aware Test Automation with LLMs: Keeping Regression Tests Aligned with Requirement Changes

One of the biggest challenges in QA is maintaining the regression test suite.

Every time a requirement changes or a new one is added, you must modify the regression suite to ensure it remains aligned with the constantly changing requirements.

This is time-consuming for QA teams, who already have to design test cases for new functionality within tight deadlines.

It is also error-prone, as manual updates can easily miss edge cases or negative paths.

A common shortcut is to use large language models (LLMs) like ChatGPT to generate Gherkin scenarios by pasting requirements into a prompt.

While this approach can quickly produce tests for a single feature, it falls short in real-world projects.

A one-off LLM prompt has no memory of your requirement changes and doesn't link new functionality to regression risks. It also often ignores boundaries or negative flows since it doesn't understand the application context.

This is where context-awareness becomes important.

By storing past and present requirements in a way that preserves their meaning, you can guide an LLM to generate test cases. These test cases not only cover new functionality but also confirm that existing features still work as expected.

In this guide, you'll see how a context-aware framework works, why it goes beyond basic LLM approaches, and how you can apply it to keep your regression tests current with much less effort.

Why Context-Awareness Matters

Traditional regression testing struggles because requirements change rapidly. Tests written yesterday can already be outdated by today's changes.

This forces testers to spend time rewriting cases instead of improving overall coverage.

That tedious manual work not only slows down delivery but also increases the chance of missing negative scenarios and edge cases.

Basic LLM-based test generation doesn't really solve this problem. When you paste requirements into a single prompt, the model generates scenarios in isolation. It overlooks older requirements, fails to connect new features with regression risks, and generates one-off test cases instead of an evolving suite.

For example, an LLM might generate a checkout scenario for a new "guest user" feature, but it won't automatically remind you to retest existing "registered user" flows.

Context-awareness changes this dynamic.

By semantically storing requirements, the system understands that concepts like "login" and "authentication" are related, even if they are worded differently.

When new requirements are introduced, it can automatically detect changes, highlight the affected areas, and generate new test cases, ensuring that existing workflows remain covered.

This allows you to keep your regression packs in sync with product changes rather than constantly trying to catch up later.

Context-aware test generation doesn't replace testers. It functions as an AI assistant that automates the repetitive tasks associated with regression maintenance.

You still decide testing priorities and strategy, but you don't waste hours rewriting scenarios by hand.

How the Approach Works

A context-aware test generation framework typically follows four steps: storing requirements with meaning, spotting changes, generating test cases, and validating format.

Each step combines conceptual understanding with practical tools, allowing you to transition from theory to implementation without feeling overwhelmed.

1. Storing Requirements with Meaning

Instead of saving requirements as plain text, you store them in a vector database such as ChromaDB.

Vectorization captures the semantic meaning of sentences, not just their keywords. This means that if one document mentions "login" and another mentions "authentication," the system recognizes they describe related concepts.

To make this effective, requirements are broken into smaller, meaningful chunks by feature or module. These are ingested from common formats, such as .txt, .md, or .pdf.

You can improve the framework to add more file type support.

2. Spotting Changes

When a new requirement is added, the system compares it against existing requirements using embeddings such as BAAI/bge-small-en-v1.5. This functions like an automated impact analysis.

The embedding search identifies related features and highlights which parts of your system may be affected.

Now, instead of manually scanning requirement docs to figure out what's changed, you instantly see where regression tests are needed.

3. Generating Test Cases

Once changes are identified, an LLM such as LLaMA drafts new Gherkin test cases.

These cover positive paths, negative scenarios, boundary conditions, and regression risks. Using Cloudflare's inference layer also means that requirement documents are not retained. This addresses privacy concerns that come with sending sensitive project details to public GPTs.

Tests will generate automatically, but you still need to review them for accuracy and business alignment.

4. Validating Format

Finally, a lightweight syntax check ensures that generated test cases follow proper Gherkin formatting. This keeps scenarios clear for both technical and non-technical stakeholders, preventing malformed steps from creating bottlenecks during execution.

The combination of semantic storage, automated impact detection, AI-driven test generation, and format validation yields regression packs that remain up to date without requiring heavy manual rework.

Example: Guest Checkout Requirement

Imagine your e-commerce platform originally required users to register before checkout. A new requirement now allows guest checkout with only an email and shipping address.

Let's check what one-off LLM prompts like ChatGPT generate with the following prompt:

Prompt:

Write test cases for an e-commerce platform. Guests are allowed to check out with only an email and a shipping address.

1. Positive Test Cases

TC-001: Guest checkout with a valid email and shipping address

• Precondition: User has items in the cart.
• Steps:
  1. Go to cart and click "Checkout as Guest."
  2. Enter a valid email (e.g., test@example.com).
  3. Enter a valid shipping address.
  4. Proceed to payment and complete order.
• Expected Result: Order is successfully placed, order confirmation email is sent.

TC-002: Guest checkout with apartment/unit details in address

• Ensure the system correctly captures and stores extended address details.

TC-003: Guest checkout with international address

• Test with different country formats (e.g., US ZIP, UK postcode, etc.).

TC-004: Guest checkout with different shipping and billing addresses

• Confirm system accepts different addresses and applies them correctly.

2. Negative Test Cases

TC-005: Guest checkout with invalid email format

• Enter abc@com or user@ and try to proceed.
• Expected Result: Validation error shown, user cannot proceed.

TC-006: Guest checkout with missing required shipping address fields

• Leave city, postal code, or street blank.
• Expected Result: Error messages for missing mandatory fields.

TC-007: Guest checkout with unsupported characters in address

• Enter special characters (e.g., ####, <script>).
• Expected Result: Validation error or sanitisation applied.

TC-008: Guest checkout with excessively long email or address

• Enter >255 characters.
• Expected Result: System rejects and shows validation message.

3. Functional / Integration Cases

TC-009: Guest checkout triggers confirmation email

• Verify that confirmation email is sent to the guest's provided email.

TC-010: Guest checkout with existing registered user's email

• Enter an email already associated with a registered account.
• Expected Result: System should still allow checkout as guest, but may offer a prompt to log in.

TC-011: Order history for guest

• Confirm that guests cannot view order history after session ends, but can track order via email link.

TC-012: Payment integration check

• Ensure guest checkout flows correctly to payment gateway and back.

TC-013: Guest clicks back button before confirmation page

• Ensure no duplicate order is created.

Now let's try with a context-aware approach.

In addition to the regular test cases above, the context-aware test case generator will generate the following additional test cases to cover regression scenarios.

Regression Test Cases

Feature: Existing Account Creation and Login Process

Scenario: Existing User Can Still Create Account

• Given the guest checkout option is enabled
• When the user tries to create an account
• Then the account creation process should still work as expected
• And the user should be able to log in with their new account

Scenario: Existing User Can Still Log In

• Given the guest checkout option is enabled
• When the user tries to log in with their existing account
• Then the login process should still work as expected
• And the user should be logged in successfully

What makes these scenarios stronger than a basic LLM output is the regression awareness.

A simple one-off LLM prompt might generate only the new guest checkout flow, but the context-aware approach ensures you also test invalid inputs and confirm that the existing registered checkout still works.

This balance of new coverage, negative paths, and regression protection gives teams far more confidence in their updates.

Best Practices and Things to Avoid

To get the most out of context-aware test generation, you need to follow some best practices.

• Treat generated tests as drafts. AI accelerates test creation, but human review is essential to ensure accuracy and alignment with business rules.
• Include regression checks for every requirement change. It's easy to focus only on the new feature, but regression risks are where production bugs often appear.
• Keep scenarios in Gherkin format. Gherkin makes tests readable to developers, testers, and business stakeholders, bridging communication gaps.
• Use privacy-first tools. Hosting models on platforms like Cloudflare Workers AI or using in-house LLMs ensures requirement data isn't retained by public services, which is critical when working with sensitive or client-owned information.

Things to Avoid

• Don't assume the AI "knows" your business logic. LLMs only work with the requirements you provide. If your docs are incomplete, your tests will be too.
• Don't overcomplicate the setup initially. Start with one module or feature before scaling up, so you understand the workflow and build trust in the process.
• Never enter sensitive data, such as real usernames or credentials. Even privacy-friendly setups should treat test data responsibly.
• Don't feed client requirements into an AI model without approval. Always explain to stakeholders how the process works and secure their consent to protect confidentiality and trust.

By following these practices, you ensure that AI-assisted test generation strengthens your process rather than introducing new risks.

Implementing Context-Awareness in Your Workflow

Keeping regression tests aligned with changing requirements remains one of the most challenging tasks for QA teams.

Manual updates consume valuable time, while basic LLM prompts often generate isolated scenarios with no awareness of regression risks.

The result is either outdated test packs or incomplete coverage, which compromises product quality.

Context-awareness breaks this cycle.

Try this concept on your own project. Take a recent requirement change and compare your manual updates with AI-generated Gherkin scenarios.

You'll quickly see the time savings and coverage improvements that context-awareness provides.

Over time, you can integrate this approach into your workflows, starting with lightweight scripts, then expanding into deeper integration with your test management tools.

To see the framework in action, you can explore the GitHub repository. All the required technical information to set up the framework is mentioned in the ReadMe file.

References

• Embedding-based retrieval techniques 
  
• Embedding + LLM evaluation
  
• Creating a Cloudflare API token
• https://github.com/chamiz/test-case-generator - Framework repository link
• Meta LLaMA 3 8B Model
• Mistral-7B on HuggingFace
• ChromaDB
• FastAPI Docs
• Cucumber Gherkin Syntax
• BAAI/bge-small-en-v1.5
• Cloudflare Workers AI