A standard use case with generative AI that we normally see prospects consider for a manufacturing use case is a generative AI-powered assistant. Nonetheless, earlier than it may be deployed, there may be the standard manufacturing readiness evaluation that features issues resembling understanding the safety posture, monitoring and logging, value monitoring, resilience, and extra. The very best precedence of those manufacturing readiness assessments is normally safety. If there are safety dangers that may’t be clearly recognized, then they’ll’t be addressed, and that may halt the manufacturing deployment of the generative AI utility.
On this put up, we present you an instance of a generative AI assistant utility and display find out how to assess its safety posture utilizing the OWASP Prime 10 for Giant Language Mannequin Functions, in addition to find out how to apply mitigations for frequent threats.
Generative AI scoping framework
Begin by understanding the place your generative AI utility matches throughout the spectrum of managed vs. customized. Use the AWS generative AI scoping framework to know the precise mixture of the shared accountability for the safety controls relevant to your utility. For instance, Scope 1 “Shopper Apps” like PartyRock or ChatGPT are normally publicly going through purposes, the place many of the utility inside safety is owned and managed by the supplier, and your accountability for safety is on the consumption facet. Distinction that with Scope 4/5 purposes, the place not solely do you construct and safe the generative AI utility your self, however you might be additionally accountable for fine-tuning and coaching the underlying massive language mannequin (LLM). The safety controls in scope for Scope 4/5 purposes will vary extra broadly from the frontend to LLM mannequin safety. This put up will give attention to the Scope 3 generative AI assistant utility, which is likely one of the extra frequent use circumstances seen within the discipline.
The next determine of the AWS Generative AI Safety Scoping Matrix summarizes the sorts of fashions for every scope.
OWASP Prime 10 for LLMs
Utilizing the OWASP Prime 10 for understanding threats and mitigations to an utility is likely one of the most typical methods utility safety is assessed. The OWASP Prime 10 for LLMs takes a tried and examined framework and applies it to generative AI purposes to assist us uncover, perceive, and mitigate the novel threats for generative AI.
Answer overview
Let’s begin with a logical structure of a typical generative AI assistant utility overlying the OWASP Prime 10 for LLM threats, as illustrated within the following diagram.
On this structure, the end-user request normally goes by the next parts:
- Authentication layer – This layer validates that the consumer connecting to the appliance is who they are saying they’re. That is sometimes finished by some kind of an id supplier (IdP) functionality like Okta, AWS IAM Id Middle, or Amazon Cognito.
- Utility controller – This layer comprises many of the utility enterprise logic and determines find out how to course of the incoming consumer request by producing the LLM prompts and processing LLM responses earlier than they’re despatched again to the consumer.
- LLM and LLM agent – The LLM offers the core generative AI functionality to the assistant. The LLM agent is an orchestrator of a set of steps that may be crucial to finish the specified request. These steps may contain each the usage of an LLM and exterior information sources and APIs.
- Agent plugin controller – This part is accountable for the API integration to exterior information sources and APIs. This part additionally holds the mapping between the logical identify of an exterior part, which the LLM agent may check with, and the bodily identify.
- RAG information retailer – The Retrieval Augmented Era (RAG) information retailer delivers up-to-date, exact, and access-controlled data from numerous information sources resembling information warehouses, databases, and different software program as a service (SaaS) purposes by information connectors.
The OWASP Prime 10 for LLM dangers map to numerous layers of the appliance stack, highlighting vulnerabilities from UIs to backend programs. Within the following sections, we talk about dangers at every layer and supply an utility design sample for a generative AI assistant utility in AWS that mitigates these dangers.
The next diagram illustrates the assistant structure on AWS.
Authentication layer (Amazon Cognito)
Widespread safety threats resembling brute pressure assaults, session hijacking, and denial of service (DoS) assaults can happen. To mitigate these dangers, implement finest practices like multi-factor authentication (MFA), fee limiting, safe session administration, automated session timeouts, and common token rotation. Moreover, deploying edge safety measures resembling AWS WAF and distributed denial of service (DDoS) mitigation helps block frequent internet exploits and keep service availability throughout assaults.
Within the previous structure diagram, AWS WAF is built-in with Amazon API Gateway to filter incoming visitors, blocking unintended requests and defending purposes from threats like SQL injection, cross-site scripting (XSS), and DoS assaults. AWS WAF Bot Management additional enhances safety by offering visibility and management over bot visitors, permitting directors to dam or rate-limit undesirable bots. This function will be centrally managed throughout a number of accounts utilizing AWS Firewall Supervisor, offering a constant and sturdy strategy to utility safety.
Amazon Cognito enhances these defenses by enabling consumer authentication and information synchronization. It helps each consumer swimming pools and id swimming pools, enabling seamless administration of consumer identities throughout gadgets and integration with third-party id suppliers. Amazon Cognito provides safety features, together with MFA, OAuth 2.0, OpenID Join, safe session administration, and risk-based adaptive authentication, to assist defend towards unauthorized entry by evaluating sign-in requests for suspicious exercise and responding with extra safety measures like MFA or blocking sign-ins. Amazon Cognito additionally enforces password reuse prevention, additional defending towards compromised credentials.
AWS Defend Superior provides an additional layer of protection by offering enhanced safety towards refined DDoS assaults. Built-in with AWS WAF, Defend Superior delivers complete perimeter safety, utilizing tailor-made detection and health-based assessments to boost response to assaults. It additionally provides round the clock help from the AWS Defend Response Staff and contains DDoS value safety, making purposes stay safe and cost-effective. Collectively, Defend Superior and AWS WAF create a safety framework that protects purposes towards a variety of threats whereas sustaining availability.
This complete safety setup addresses LLM10:2025 Unbound Consumption and LLM02:2025 Delicate Data Disclosure, ensuring that purposes stay each resilient and safe.
Utility controller layer (LLM orchestrator Lambda operate)
The applying controller layer is normally susceptible to dangers resembling LLM01:2025 Immediate Injection, LLM05:2025 Improper Output Dealing with, and LLM 02:2025 Delicate Data Disclosure. Outdoors events may often try to take advantage of this layer by crafting unintended inputs to govern the LLM, probably inflicting it to disclose delicate data or compromise downstream programs.
Within the bodily structure diagram, the appliance controller is the LLM orchestrator AWS Lambda operate. It performs strict enter validation by extracting the occasion payload from API Gateway and conducting each syntactic and semantic validation. By sanitizing inputs, making use of allowlisting and deny itemizing of key phrases, and validating inputs towards predefined codecs or patterns, the Lambda operate helps stop LLM01:2025 Immediate Injection assaults. Moreover, by passing the user_id downstream, it allows the downstream utility parts to mitigate the chance of delicate data disclosure, addressing issues associated to LLM02:2025 Delicate Data Disclosure.
Amazon Bedrock Guardrails offers a further layer of safety by filtering and blocking delicate content material, resembling personally identifiable data (PII) and different customized delicate information outlined by regex patterns. Guardrails can be configured to detect and block offensive language, competitor names, or different undesirable phrases, ensuring that each inputs and outputs are protected. You can too use guardrails to stop LLM01:2025 Immediate Injection assaults by detecting and filtering out dangerous or manipulative prompts earlier than they attain the LLM, thereby sustaining the integrity of the immediate.
One other vital facet of safety is managing LLM outputs. As a result of the LLM may generate content material that features executable code, resembling JavaScript or Markdown, there’s a threat of XSS assaults if this content material isn’t correctly dealt with. To mitigate this threat, apply output encoding methods, resembling HTML entity encoding or JavaScript escaping, to neutralize any probably dangerous content material earlier than it’s introduced to customers. This strategy addresses the chance of LLM05:2025 Improper Output Dealing with.
Implementing Amazon Bedrock immediate administration and versioning permits for steady enchancment of the consumer expertise whereas sustaining the general safety of the appliance. By rigorously managing modifications to prompts and their dealing with, you’ll be able to improve performance with out introducing new vulnerabilities and mitigating LLM01:2025 Immediate Injection assaults.
Treating the LLM as an untrusted consumer and making use of human-in-the-loop processes over sure actions are methods to decrease the probability of unauthorized or unintended operations.
LLM and LLM agent layer (Amazon Bedrock LLMs)
The LLM and LLM agent layer often handles interactions with the LLM and faces dangers resembling LLM10: Unbounded Consumption, LLM05:2025 Improper Output Dealing with, and LLM02:2025 Delicate Data Disclosure.
DoS assaults can overwhelm the LLM with a number of resource-intensive requests, degrading general service high quality whereas growing prices. When interacting with Amazon Bedrock hosted LLMs, setting request parameters resembling the utmost size of the enter request will reduce the chance of LLM useful resource exhaustion. Moreover, there’s a arduous restrict on the utmost variety of queued actions and whole actions an Amazon Bedrock agent can take to satisfy a buyer’s intent, which limits the variety of actions in a system reacting to LLM responses, avoiding pointless loops or intensive duties that might exhaust the LLM’s assets.
Improper output dealing with results in vulnerabilities resembling distant code execution, cross-site scripting, server-side request forgery (SSRF), and privilege escalation. The insufficient validation and administration of the LLM-generated outputs earlier than they’re despatched downstream can grant oblique entry to extra performance, successfully enabling these vulnerabilities. To mitigate this threat, deal with the mannequin as every other consumer and apply validation of the LLM-generated responses. The method is facilitated with Amazon Bedrock Guardrails utilizing filters resembling content material filters with configurable thresholds to filter dangerous content material and safeguard towards immediate assaults earlier than they’re processed additional downstream by different backend programs. Guardrails routinely consider each consumer enter and mannequin responses to detect and assist stop content material that falls into restricted classes.
Amazon Bedrock Brokers execute multi-step duties and securely combine with AWS native and third-party providers to cut back the chance of insecure output dealing with, extreme company, and delicate data disclosure. Within the structure diagram, the motion group Lambda operate underneath the brokers is used to encode all of the output textual content, making it routinely non-executable by JavaScript or Markdown. Moreover, the motion group Lambda operate parses every output from the LLM at each step executed by the brokers and controls the processing of the outputs accordingly, ensuring they’re protected earlier than additional processing.
Delicate data disclosure is a threat with LLMs as a result of malicious immediate engineering could cause LLMs to unintentionally reveal unintended particulars of their responses. This may result in privateness and confidentiality violations. To mitigate the difficulty, implement information sanitization practices by content material filters in Amazon Bedrock Guardrails.
Moreover, implement customized information filtering insurance policies primarily based on user_id and strict consumer entry insurance policies. Amazon Bedrock Guardrails helps filter content material deemed delicate, and Amazon Bedrock Brokers additional reduces the chance of delicate data disclosure by permitting you to implement customized logic within the preprocessing and postprocessing templates to strip any sudden data. If in case you have enabled mannequin invocation logging for the LLM or applied customized logging logic in your utility to document the enter and output of the LLM in Amazon CloudWatch, measures resembling CloudWatch Log information safety are essential in masking delicate data recognized within the CloudWatch logs, additional mitigating the chance of delicate data disclosure.
Agent plugin controller layer (motion group Lambda operate)
The agent plugin controller often integrates with inside and exterior providers and applies customized authorization to inside and exterior information sources and third-party APIs. At this layer, the chance of LLM08:2025 Vector & Embedding Weaknesses and LLM06:2025 Extreme Company are in impact. Untrusted or unverified third-party plugins might introduce backdoors or vulnerabilities within the type of sudden code.
Apply least privilege entry to the AWS Id and Entry Administration (IAM) roles of the motion group Lambda operate, which interacts with plugin integrations to exterior programs to assist mitigate the chance of LLM06:2025 Extreme Company and LLM08:2025 Vector & Embedding Weaknesses. That is demonstrated within the bodily structure diagram; the agent plugin layer Lambda operate is related to a least privilege IAM function for safe entry and interface with different inside AWS providers.
Moreover, after the consumer id is set, limit the information aircraft by making use of user-level entry management by passing the user_id to downstream layers just like the agent plugin layer. Though this user_id parameter can be utilized within the agent plugin controller Lambda operate for customized authorization logic, its main goal is to allow fine-grained entry management for third-party plugins. The accountability lies with the appliance proprietor to implement customized authorization logic throughout the motion group Lambda operate, the place the user_id parameter can be utilized together with predefined guidelines to use the suitable degree of entry to third-party APIs and plugins. This strategy wraps deterministic entry controls round a non-deterministic LLM and allows granular entry management over which customers can entry and execute particular third-party plugins.
Combining user_id-based authorization on information and IAM roles with least privilege on the motion group Lambda operate will usually reduce the chance of LLM08:2025 Vector & Embedding Weaknesses and LLM06:2025 Extreme Company.
RAG information retailer layer
The RAG information retailer is accountable for securely retrieving up-to-date, exact, and consumer access-controlled data from numerous first-party and third-party information sources. By default, Amazon Bedrock encrypts all data base-related information utilizing an AWS managed key. Alternatively, you’ll be able to select to make use of a buyer managed key. When organising an information ingestion job in your data base, you can too encrypt the job utilizing a customized AWS Key Administration Service (AWS KMS) key.
When you determine to make use of the vector retailer in Amazon OpenSearch Service in your data base, Amazon Bedrock can cross a KMS key of your option to it for encryption. Moreover, you’ll be able to encrypt the classes during which you generate responses from querying a data base with a KMS key. To facilitate safe communication, Amazon Bedrock Information Bases makes use of TLS encryption when interacting with third-party vector shops, supplied that the service helps and permits TLS encryption in transit.
Relating to consumer entry management, Amazon Bedrock Information Bases makes use of filters to handle permissions. You possibly can construct a segmented entry answer on high of a data base utilizing metadata and filtering function. Throughout runtime, your utility should authenticate and authorize the consumer, and embody this consumer data within the question to take care of correct entry controls. To maintain the entry controls up to date, you must periodically resync the information to replicate any modifications in permissions. Moreover, teams will be saved as a filterable attribute, additional refining entry management.
This strategy helps mitigate the chance of LLM02:2025 Delicate Data Disclosure and LLM08:2025 Vector & Embedding Weaknesses, to help in that solely licensed customers can entry the related information.
Abstract
On this put up, we mentioned find out how to classify your generative AI utility from a safety shared accountability perspective utilizing the AWS Generative AI Safety Scoping Matrix. We reviewed a typical generative AI assistant utility structure and assessed its safety posture utilizing the OWASP Prime 10 for LLMs framework, and confirmed find out how to apply the OWASP Prime 10 for LLMs menace mitigations utilizing AWS service controls and providers to strengthen the structure of your generative AI assistant utility. Study extra about constructing generative AI purposes with AWS Workshops for Bedrock.
In regards to the Authors
Syed Jaffry is a Principal Options Architect with AWS. He advises software program firms on AI and helps them construct fashionable, sturdy and safe utility architectures on AWS.
Amit Kumar Agrawal is a Senior Options Architect at AWS the place he has spent over 5 years working with massive ISV prospects. He helps organizations construct and function cost-efficient and scalable options within the cloud, driving their enterprise and technical outcomes.
Tej Nagabhatla is a Senior Options Architect at AWS, the place he works with a various portfolio of purchasers starting from ISVs to massive enterprises. He makes a speciality of offering architectural steerage throughout a variety of matters round AI/ML, safety, storage, containers, and serverless applied sciences. He helps organizations construct and function cost-efficient, scalable cloud purposes. In his free time, Tej enjoys music, taking part in basketball, and touring.
