CT-GenAI ISTQB Certified Tester Testing with Generative AI (CT-GenAI) v1.0 Questions and Answers

The primary differentiator between an LLM-powered test infrastructure and a traditional chatbot is the move from "deterministic" to "probabilistic" logic. Traditional chatbots (Option D) rely on "if-then" logic, decision trees, and predefined scripts. They can only respond to queries that match specific keywords or patterns mapped in their database. In contrast, an LLM-powered infrastructure utilizes the generative capabilities of Large Language Models to synthesize and create new content based on context. This allows it todynamically generate test insights(Option A)—such as predicting potential regression risks based on unstructured code diffs or drafting test cases for a brand-new feature described in natural language. While traditional bots provide fixed, scripted responses (Option B), LLMs can "reason" through multi-step testing problems and provide nuanced explanations. This contextual awareness is powered by the model's training on vast amounts of technical documentation, enabling it to assist in exploratory testing and complex analysis that traditional, rule-based systems simply cannot handle.

Generative AI is exceptionally capable of creating structured and unstructured data, but its role is limited to "generation" and "transformation," not infrastructure management or direct database administration. Creating production database backups (Option A) is a physical data management task involving the copying of actual stateful data from a server to storage; this is handled by database management systems (DBMS) and DevOps pipelines, not LLMs. Conversely, LLMs excel at the logic-based tasks listed in the other options. They can analyze requirements to identify and set boundary values (Option B) for input validation. They are also highly effective at creating combinatorial data (Option C), such as pairwise or all-combinations tables, by understanding the relationships between variables. Finally, one of the most powerful uses of GenAI in testing is generating synthetic datasets (Option D)—creating "fake" but realistically structured data that mimics production patterns without exposing Sensitive Personally Identifiable Information (SPII), thereby supporting privacy-compliant testing.

Differentiating between prompting techniques is essential for a tester to select the right tool for the task.Few-shot promptingis characterized by providing the model with a few examples of inputs and desired outputs, allowing it to learn the pattern and format.Prompt Chaininginvolves breaking a complex task into a sequence of smaller, interconnected prompts, where the output of one step becomes the input for the next (e.g., first extract requirements, then generate test cases from those requirements).Meta-promptingis a more advanced technique where the user asks the LLM to help design, write, or refine the prompt itself—essentially using the AI as a "prompt engineer" to optimize the instructions. Option D correctly identifies these core characteristics. Options A, B, and C contain fundamental mischaracterizations: for instance, Few-shotrequiresexamples (contradicting A), and Chaining is theoppositeof a single prompt (contradicting A). Mastering these distinctions allows testers to move from simple "chatting" to sophisticated AI orchestration that can handle complex, multi-stage testing workflows with high reliability.

Tokenizationis the foundational process by which an LLM breaks down raw text into smaller, manageable units called "tokens." These tokens can represent individual words, parts of words (sub-words), or even punctuation marks. This is a critical step because LLMs do not "read" words like humans do; they process numerical representations of these tokens. The way text is tokenized directly impacts the model's efficiency and its ability to understand complex technical terminology used in software testing. For example, a rare technical term might be broken into several sub-word tokens. This process is closely linked to theContext Window(Option C), which is the maximum number of tokens a model can "remember" or process at one time. WhileEmbeddings(Option B) are the numerical vectors that represent the meaning of these tokens, and theTransformer(Option A) is the underlying architecture that processes them, tokenization is the specific mechanism for initial text decomposition. Understanding tokenization is vital for testers when managing long requirement documents to ensure they do not exceed the model's limits.

This scenario describes a form ofRequest Manipulation, specifically a type of "Prompt Injection" or "Adversarial Prompting." In this attack vector, the user (or an external attacker) provides malicious or deceptive input—in this case, via an image in a multimodal LLM—to bypass the model's intended constraints or to steer its logic toward an unintended outcome. By crafting an image that tricks the LLM into seeing "acceptance criteria" that aren't actually there, the attacker manipulates the model’s request processing to generate false validation results. This is different fromData Poisoning(Option A), which involves corrupting the training data before the model is even built. It is also distinct fromData Exfiltration(Option B), which aims to steal data from the model. In a testing environment, request manipulation is a significant risk because it can lead to "Silent Failures," where the AI reports that tests have passed or requirements are met based on deceptive input, thereby compromising the integrity of the entire Quality Assurance process.

Hallucinations—where an LLM generates factually incorrect or nonsensical information—occur primarily when the model lacks sufficient specific information and "fills in the gaps" using probabilistic patterns from its training data. The most effective mitigation strategy is "grounding," which involves providing the model with detailed, project-specific context. By including technical specifications, existing API schemas, business rules, and identified constraints within the prompt, the tester restricts the model’s operational space to the "project realities." This ensures the model does not have to guess or improvise details about the System Under Test (SUT). In contrast, randomizing prompts (Option B) or relying on generic examples (Option C) increases the likelihood of inconsistent and inaccurate outputs. Furthermore, using "longer" or higher temperature settings (Option D) actually encourages creativity and randomness, which is the opposite of the precision required for testing and significantly increases the risk of hallucinations. Therefore, rich contextual grounding is the technical foundation for reliable AI-assisted test analysis.

Prompt Chainingis a technique where a complex task is decomposed into several smaller, sequential steps, where the output of one step serves as the context or input for the next. This is far more reliable than a "one-shot" approach (Option A) because it reduces the cognitive load on the LLM and allows for intermediate verification. In the scenario of test analysis, the most logical and effective chain begins by extracting discretetest conditionsfrom the raw requirements. Once these conditions are established, the next "link" in the chain is toprioritize them based on risk(impact and likelihood), which requires the model to reason specifically about the importance of each condition. The final step is tomap these prioritized conditions back to the original requirementsto identify any "coverage gaps." This systematic flow (Option B) mirrors the professional test analysis process defined in the ISTQB/CT-GenAI standards. By following this sequence, the tester ensures that the AI-generated output is logically derived and thorough, providing a clear "audit trail" from the initial requirement to the final prioritized test suite.

A hallucination refers to a phenomenon where a Large Language Model generates text that is grammatically correct and seemingly plausible but is factually incorrect or unsupported by the provided context or real-world data. In the context of software testing, this is a critical limitation. For example, an LLM might generate a test case for a software feature that does not exist or cite a non-existent API parameter. These errors occur because LLMs are probabilistic engines designed to predict the "most likely" next token rather than "reasoning" from a set of verified facts. They do not have a built-in "truth" mechanism. While a logical mistake (Option B) is a failure in reasoning and a systematic preference (Option D) describes bias, a hallucination is specifically about the fabrication of information. Testers must be particularly vigilant regarding hallucinations, as they can lead to "false confidence" in test coverage or the creation of invalid bug reports. Mitigations include grounding the model with Retrieval-Augmented Generation (RAG) and implementing rigorous "human-in-the-loop" verification of all AI-generated test artifacts.

The statement that fine-tuning "replaces the model's general knowledge entirely" isincorrect. Fine-tuning is a process of "incremental learning" where a pre-trained model (which already possesses vast general knowledge) is further trained on a smaller, domain-specific dataset—such as an organization's internal API documentation or historical test scripts. The goal is to adjust the model's internal weights so that it becomes more proficient in a specific area (Option A) and adheres better to local terminology and formatting standards (Option C). It doesnoterase the foundational language capabilities of the model. Furthermore, fine-tuning is a common strategy for Small Language Models (SLMs) to allow them to punch above their weight class in specific tasks while remaining computationally efficient (Option D). However, if done poorly, fine-tuning can actuallycauseoverfitting (where the model becomes too rigid and loses its ability to generalize), rather than preventing it. Therefore, fine-tuning should be viewed as a "specialization" layer rather than a total replacement of the model's base intelligence.