{"@ID": "1426", "@Name": "Improper Validation of Generative AI Output", "@Abstraction": "Base", "@Structure": "Simple", "@Status": "Incomplete", "Description": "The product invokes a generative AI/ML\n\t\t\tcomponent whose behaviors and outputs cannot be directly\n\t\t\tcontrolled, but the product does not validate or\n\t\t\tinsufficiently validates the outputs to ensure that they\n\t\t\talign with the intended security, content, or privacy\n\t\t\tpolicy.", "Related_Weaknesses": {"Related_Weakness": {"@Nature": "ChildOf", "@CWE_ID": "707", "@View_ID": "1000", "@Ordinal": "Primary"}}, "Weakness_Ordinalities": {"Weakness_Ordinality": [{"Ordinality": "Primary"}, {"Ordinality": "Resultant"}]}, "Applicable_Platforms": {"Language": {"@Class": "Not Language-Specific", "@Prevalence": "Undetermined"}, "Architecture": {"@Class": "Not Architecture-Specific", "@Prevalence": "Undetermined"}, "Technology": [{"@Name": "AI/ML", "@Prevalence": "Undetermined"}, {"@Class": "Not Technology-Specific", "@Prevalence": "Undetermined"}]}, "Modes_Of_Introduction": {"Introduction": [{"Phase": "Architecture and Design", "Note": {"xhtml:p": "Developers may rely heavily on protection mechanisms such as\ninput filtering and model alignment, assuming they are more effective\nthan they actually are."}}, {"Phase": "Implementation", "Note": {"xhtml:p": "Developers may rely heavily on protection mechanisms such as\ninput filtering and model alignment, assuming they are more effective\nthan they actually are."}}]}, "Common_Consequences": {"Consequence": {"Scope": "Integrity", "Impact": ["Execute Unauthorized Code or Commands", "Varies by Context"], "Note": {"xhtml:p": "In an agent-oriented setting,\n\t\t\t\t\toutput could be used to cause unpredictable agent\n\t\t\t\t\tinvocation, i.e., to control or influence agents\n\t\t\t\t\tthat might be invoked from the output.  The impact\n\t\t\t\t\tvaries depending on the access that is granted to\n\t\t\t\t\tthe tools, such as creating a database or writing\n\t\t\t\t\tfiles."}}}, "Detection_Methods": {"Detection_Method": [{"Method": "Dynamic Analysis with Manual Results Interpretation", "Description": "Use known techniques for prompt injection\n\t\t\t and other attacks, and adjust the attacks to be more\n\t\t\t specific to the model or system."}, {"Method": "Dynamic Analysis with Automated Results Interpretation", "Description": "Use known techniques for prompt injection\n\t\t\t and other attacks, and adjust the attacks to be more\n\t\t\t specific to the model or system."}, {"Method": "Architecture or Design Review", "Description": "Review of the product design can be\n\t\t\t effective, but it works best in conjunction with dynamic\n\t\t\t analysis."}]}, "Potential_Mitigations": {"Mitigation": [{"Phase": "Architecture and Design", "Description": "Since the output from a generative AI component (such as an LLM) cannot be trusted, ensure that it operates in an untrusted or non-privileged space."}, {"Phase": "Operation", "Description": "Use \"semantic comparators,\" which are mechanisms that\n\t\t\t\t\tprovide semantic comparison to identify objects that might appear\n\t\t\t\t\tdifferent but are semantically similar."}, {"Phase": "Operation", "Description": {"xhtml:p": "Use components that operate\n\t\t\t\t\texternally to the system to monitor the output and\n\t\t\t\t\tact as a moderator. These components are called\n\t\t\t\t\tdifferent terms, such as supervisors or\n\t\t\t\t\tguardrails."}}, {"Phase": "Build and Compilation", "Description": {"xhtml:p": "During model training, use an appropriate variety of good\n\t\t\t\t  and bad examples to guide preferred outputs."}}]}, "Observed_Examples": {"Observed_Example": {"Reference": "CVE-2024-3402", "Description": "chain: GUI for ChatGPT API performs\n\t\t\t\t\tinput validation but does not properly \"sanitize\"\n\t\t\t\t\tor validate model output data (CWE-1426), leading\n\t\t\t\t\tto XSS (CWE-79).", "Link": "https://www.cve.org/CVERecord?id=CVE-2024-3402"}}, "References": {"Reference": [{"@External_Reference_ID": "REF-1441"}, {"@External_Reference_ID": "REF-1442"}, {"@External_Reference_ID": "REF-1443"}, {"@External_Reference_ID": "REF-1444"}, {"@External_Reference_ID": "REF-1445"}]}, "Mapping_Notes": {"Usage": "Discouraged", "Rationale": "There is potential for this CWE entry to be modified in the future for further clarification as the research community continues to better understand weaknesses in this domain.", "Comments": {"xhtml:p": "This CWE entry is only related to \"validation\" of output and might be used mistakenly for other kinds of output-related weaknesses. Careful attention should be paid to whether this CWE should be used for vulnerabilities related to \"prompt injection,\" which is an attack that works against many different weaknesses. See Maintenance Notes and Research Gaps. Analysts should closely investigate the root cause to ensure it is not ultimately due to other well-known weaknesses. The following suggestions are not comprehensive."}, "Reasons": {"Reason": [{"@Type": "Potential Major Changes"}, {"@Type": "Frequent Misinterpretation"}]}, "Suggestions": {"Suggestion": [{"@CWE_ID": "77", "@Comment": "Command Injection. Use this CWE for most cases of 'prompt injection' attacks in which additional prompts are added to input to, or output from, the model. If OS command injection, consider CWE-78."}, {"@CWE_ID": "94", "@Comment": "Code Injection. Use this CWE for cases in which output from genAI components is directly fed into components that parse and execute code."}, {"@CWE_ID": "116", "@Comment": "Improper Encoding or Escaping of Output. Use this CWE when the product is expected to encode or escape genAI outputs."}]}}, "Notes": {"Note": [{"@Type": "Research Gap", "#text": "This entry is related to AI/ML, which is not well\n\t\t\t understood from a weakness perspective. Typically, for\n\t\t\t new/emerging technologies including AI/ML, early\n\t\t\t vulnerability discovery and research does not focus on\n\t\t\t root cause analysis (i.e., weakness identification). For\n\t\t\t AI/ML, the recent focus has been on attacks and\n\t\t\t exploitation methods, technical impacts, and mitigations.\n\t\t\t As a result, closer research or focused efforts by SMEs\n\t\t\t is necessary to understand the underlying weaknesses.\n\t\t\t Diverse and dynamic terminology and rapidly-evolving\n\t\t\t technology further complicate understanding. Finally,\n\t\t\t there might not be enough real-world examples with\n\t\t\t sufficient details from which weakness patterns may be\n\t\t\t discovered. For example, many real-world vulnerabilities\n\t\t\t related to \"prompt injection\" appear to be related to\n\t\t\t typical injection-style attacks in which the only\n\t\t\t difference is that the \"input\" to the vulnerable\n\t\t\t component comes from model output instead of direct\n\t\t\t adversary input, similar to \"second-order SQL injection\"\n\t\t\t attacks."}, {"@Type": "Maintenance", "#text": "This entry was created by members\n           of the CWE AI Working Group during June and July 2024. The\n           CWE Project Lead, CWE Technical Lead, AI WG co-chairs, and\n           many WG members decided that for purposes of timeliness, it\n           would be more helpful to the CWE community to publish the\n           new entry in CWE 4.15 quickly and add to it in subsequent\n           versions."}]}, "Content_History": {"Submission": {"Submission_Name": "Members of the CWE AI WG", "Submission_Organization": "CWE Artificial Intelligence (AI) Working Group (WG)", "Submission_Date": "2024-07-02", "Submission_Version": "4.15", "Submission_ReleaseDate": "2024-07-16"}, "Modification": [{"Modification_Name": "CWE Content Team", "Modification_Organization": "MITRE", "Modification_Date": "2025-12-11", "Modification_Version": "4.19", "Modification_ReleaseDate": "2025-12-11", "Modification_Comment": "updated Weakness_Ordinalities"}, {"Modification_Name": "CWE Content Team", "Modification_Organization": "MITRE", "Modification_Date": "2026-04-30", "Modification_Version": "4.20", "Modification_ReleaseDate": "2026-04-30", "Modification_Comment": "updated Relationships"}]}}