Search

Find a vulnerability

Search criteria ⓘ Use this form to refine search results.
Full-text search supports keyword queries with ranking and filtering.
You can combine vendor, product, and sources to narrow results.
Enable “Apply ordering” to sort by date instead of relevance.

    1 vulnerability found for MathGPT (https://mathgpt.streamlit.app/)

    AVID-2023-V016

    Vulnerability from avid – Published: 2023-03-31 – Updated: 2023-03-31 ATLAS Case Study
    Summary
    The publicly available Streamlit application [MathGPT](https://mathgpt.streamlit.app/) uses GPT-3, a large language model (LLM), to answer user-generated math questions. Recent studies and experiments have shown that LLMs such as GPT-3 show poor performance when it comes to performing exact math directly[<sup>\[1\]</sup>][1][<sup>\[2\]</sup>][2]. However, they can produce more accurate answers when asked to generate executable code that solves the question at hand. In the MathGPT application, GPT-3 is used to convert the user's natural language question into Python code that is then executed. After computation, the executed code and the answer are displayed to the user. Some LLMs can be vulnerable to prompt injection attacks, where malicious user inputs cause the models to perform unexpected behavior[<sup>\[3\]</sup>][3][<sup>\[4\]</sup>][4]. In this incident, the actor explored several prompt-override avenues, producing code that eventually led to the actor gaining access to the application host system's environment variables and the application's GPT-3 API key, as well as executing a denial of service attack. As a result, the actor could have exhausted the application's API query budget or brought down the application. After disclosing the attack vectors and their results to the MathGPT and Streamlit teams, the teams took steps to mitigate the vulnerabilities, filtering on select prompts and rotating the API key. [1]: https://arxiv.org/abs/2103.03874 "Measuring Mathematical Problem Solving With the MATH Dataset" [2]: https://arxiv.org/abs/2110.14168 "Training Verifiers to Solve Math Word Problems" [3]: https://lspace.swyx.io/p/reverse-prompt-eng "Reverse Prompt Engineering for Fun and (no) Profit" [4]: https://research.nccgroup.com/2022/12/05/exploring-prompt-injection-attacks/ "Exploring prompt-based attacks"
    Risk domain
    Security
    SEP view
    S0403: Adversarial Example
    Lifecycle
    L06: Deployment
    Organisations
    MathGPT (https://mathgpt.streamlit.app/) (deployer)
    Affected artifacts
    References
    URL Label
    https://atlas.mitre.org/studies/AML.CS0016 Achieving Code Execution in MathGPT via Prompt Injection
    https://arxiv.org/abs/2103.03874 Measuring Mathematical Problem Solving With the MATH Dataset
    https://arxiv.org/abs/2110.14168 Training Verifiers to Solve Math Word Problems
    https://lspace.swyx.io/p/reverse-prompt-eng Reverse Prompt Engineering for Fun and (no) Profit
    https://research.nccgroup.com/2022/12/05/explorin… Exploring prompt-based attacks
    To clipboard 

    {
  "affects": {
    "artifacts": [
      {
        "name": "MathGPT (https://mathgpt.streamlit.app/)",
        "type": "System"
      }
    ],
    "deployer": [
      "MathGPT (https://mathgpt.streamlit.app/)"
    ],
    "developer": []
  },
  "credit": null,
  "data_type": "AVID",
  "data_version": "0.2",
  "description": {
    "lang": "eng",
    "value": "The publicly available Streamlit application [MathGPT](https://mathgpt.streamlit.app/) uses GPT-3, a large language model (LLM), to answer user-generated math questions.\n\nRecent studies and experiments have shown that LLMs such as GPT-3 show poor performance when it comes to performing exact math directly[\u003csup\u003e\\[1\\]\u003c/sup\u003e][1][\u003csup\u003e\\[2\\]\u003c/sup\u003e][2]. However, they can produce more accurate answers when asked to generate executable code that solves the question at hand. In the MathGPT application, GPT-3 is used to convert the user\u0027s natural language question into Python code that is then executed. After computation, the executed code and the answer are displayed to the user.\n\nSome LLMs can be vulnerable to prompt injection attacks, where malicious user inputs cause the models to perform unexpected behavior[\u003csup\u003e\\[3\\]\u003c/sup\u003e][3][\u003csup\u003e\\[4\\]\u003c/sup\u003e][4].   In this incident, the actor explored several prompt-override avenues, producing code that eventually led to the actor gaining access to the application host system\u0027s environment variables and the application\u0027s GPT-3 API key, as well as executing a denial of service attack.  As a result, the actor could have exhausted the application\u0027s API query budget or brought down the application.\n\nAfter disclosing the attack vectors and their results to the MathGPT and Streamlit teams, the teams took steps to mitigate the vulnerabilities, filtering on select prompts and rotating the API key.\n\n[1]: https://arxiv.org/abs/2103.03874 \"Measuring Mathematical Problem Solving With the MATH Dataset\"\n[2]: https://arxiv.org/abs/2110.14168 \"Training Verifiers to Solve Math Word Problems\"\n[3]: https://lspace.swyx.io/p/reverse-prompt-eng \"Reverse Prompt Engineering for Fun and (no) Profit\"\n[4]: https://research.nccgroup.com/2022/12/05/exploring-prompt-injection-attacks/ \"Exploring prompt-based attacks\""
  },
  "impact": {
    "avid": {
      "lifecycle_view": [
        "L06: Deployment"
      ],
      "risk_domain": [
        "Security"
      ],
      "sep_view": [
        "S0403: Adversarial Example"
      ],
      "taxonomy_version": "0.2"
    }
  },
  "last_modified_date": "2023-03-31",
  "metadata": {
    "vuln_id": "AVID-2023-V016"
  },
  "problemtype": {
    "classof": "ATLAS Case Study",
    "description": {
      "lang": "eng",
      "value": "Achieving Code Execution in MathGPT via Prompt Injection"
    },
    "type": "Advisory"
  },
  "published_date": "2023-03-31",
  "references": [
    {
      "label": "Achieving Code Execution in MathGPT via Prompt Injection",
      "type": "source",
      "url": "https://atlas.mitre.org/studies/AML.CS0016"
    },
    {
      "label": "Measuring Mathematical Problem Solving With the MATH Dataset",
      "type": "source",
      "url": "https://arxiv.org/abs/2103.03874"
    },
    {
      "label": "Training Verifiers to Solve Math Word Problems",
      "type": "source",
      "url": "https://arxiv.org/abs/2110.14168"
    },
    {
      "label": "Reverse Prompt Engineering for Fun and (no) Profit",
      "type": "source",
      "url": "https://lspace.swyx.io/p/reverse-prompt-eng"
    },
    {
      "label": "Exploring prompt-based attacks",
      "type": "source",
      "url": "https://research.nccgroup.com/2022/12/05/exploring-prompt-injection-attacks"
    }
  ],
  "reports": null
}