Consider conversational AI brokers with Amazon Bedrock

As conversational synthetic intelligence (AI) brokers acquire traction throughout industries, offering reliability and consistency is essential for delivering seamless and reliable consumer experiences. Nevertheless, the dynamic and conversational nature of those interactions makes conventional testing and analysis strategies difficult. Conversational AI brokers additionally embody a number of layers, from Retrieval Augmented Technology (RAG) to function-calling mechanisms that work together with exterior information sources and instruments. Though present massive language mannequin (LLM) benchmarks like MT-bench consider mannequin capabilities, they lack the flexibility to validate the applying layers. The next are some frequent ache factors in growing conversational AI brokers:

Testing an agent is usually tedious and repetitive, requiring a human within the loop to validate the semantics that means of the responses from the agent, as proven within the following determine.
Establishing correct take a look at instances and automating the analysis course of will be tough as a result of conversational and dynamic nature of agent interactions.
Debugging and tracing how conversational AI brokers path to the suitable motion or retrieve the specified outcomes will be complicated, particularly when integrating with exterior information sources and instruments.

Agent Analysis, an open supply answer utilizing LLMs on Amazon Bedrock, addresses this hole by enabling complete analysis and validation of conversational AI brokers at scale.

Amazon Bedrock is a totally managed service that gives a alternative of high-performing basis fashions (FMs) from main AI corporations like AI21 Labs, Anthropic, Cohere, Meta, Stability AI, and Amazon by means of a single API, together with a broad set of capabilities to construct generative AI functions with safety, privateness, and accountable AI.

Agent Analysis offers the next:

Constructed-in assist for fashionable companies, together with Brokers for Amazon Bedrock, Data Bases for Amazon Bedrock, Amazon Q Enterprise, and Amazon SageMaker endpoints
Orchestration of concurrent, multi-turn conversations together with your agent whereas evaluating its responses
Configurable hooks to validate actions triggered by your agent
Integration into steady integration and supply (CI/CD) pipelines to automate agent testing
A generated take a look at abstract for efficiency insights together with dialog historical past, take a look at cross price, and reasoning for cross/fail outcomes
Detailed traces to allow step-by-step debugging of the agent interactions

On this put up, we display find out how to streamline digital agent testing at scale utilizing Amazon Bedrock and Agent Analysis.

Resolution overview

To make use of Agent Analysis, that you must create a take a look at plan, which consists of three configurable elements:

Goal – A goal represents the agent you wish to take a look at
Evaluator – An evaluator represents the workflow and logic to guage the goal on a take a look at
Check – A take a look at defines the goal’s performance and the way you need your end-user to work together with the goal, which incorporates:
- A collection of steps representing the interactions between the agent and the end-user
- Your anticipated outcomes of the dialog

The next determine illustrates how Agent Analysis works on a excessive stage. The framework implements an LLM agent (evaluator) that can orchestrate conversations with your individual agent (goal) and consider the responses in the course of the dialog.

The next determine illustrates the analysis workflow. It reveals how the evaluator causes and assesses responses based mostly on the take a look at plan. You may both present an preliminary immediate or instruct the evaluator to generate one to provoke the dialog. At every flip, the evaluator engages the goal agent and evaluates its response. This course of continues till the anticipated outcomes are noticed or the utmost variety of dialog turns is reached.

By understanding this workflow logic, you possibly can create a take a look at plan to totally assess your agent’s capabilities.

Use case overview

For example how Agent Analysis can speed up the event and deployment of conversational AI brokers at scale, let’s discover an instance state of affairs: growing an insurance coverage declare processing agent utilizing Brokers for Amazon Bedrock. This insurance coverage declare processing agent is anticipated to deal with numerous duties, corresponding to creating new claims, sending reminders for pending paperwork associated to open claims, gathering proof for claims, and trying to find related info throughout present claims and buyer information repositories.

For this use case, the purpose is to check the agent’s functionality to precisely search and retrieve related info from present claims. You wish to ensure the agent offers right and dependable details about present claims to end-users. Completely evaluating this performance is essential earlier than deployment.

Start by creating and testing the agent in your improvement account. Throughout this section, you work together manually with the conversational AI agent utilizing pattern prompts to do the next:

Have interaction the agent in multi-turn conversations on the Amazon Bedrock console
Validate the responses from the agent
Validate all of the actions invoked by the agent
Debug and test traces for any routing failures

With Agent Analysis, the developer can streamline this course of by means of the next steps:

Configure a take a look at plan:
1. Select an evaluator from the fashions supplied by Amazon Bedrock.
2. Configure the goal, which needs to be a kind that Agent Analysis helps. For this put up, we use an Amazon Bedrock agent.
3. Outline the take a look at steps and anticipated outcomes. Within the following instance take a look at plan, you could have a declare with the ID claim-006 in your take a look at system. You wish to affirm that your agent can precisely reply questions on this particular declare.
```
evaluator:
  kind: bedrock-claude
  mannequin: claude-haiku
goal:
  kind: bedrock-agent
  bedrock_agent_alias_id:xxxxxxx
  bedrock_agent_id:xxxxxxx
assessments:
  - title: GetOpenClaimsWithDetails
    steps:
      - Ask the agent which claims are open.
      - Ask the agent for particulars on claim-006.
    expected_results:
      - The agent returns an inventory of open claims.
      - The agent returns the small print on claim-006.
```
Run the take a look at plan from the command line:
The Agent Analysis take a look at runner will robotically orchestrate the take a look at based mostly on the take a look at plan, and use the evaluator to find out if the responses from the goal match the anticipated outcomes.
View the end result abstract.
A end result abstract will probably be supplied in markdown format. Within the following instance, the abstract signifies that the take a look at failed as a result of the agent was unable to offer correct details about the present declare claim-006.

Debug with the hint recordsdata of the failed assessments.
Agent Analysis offers detailed hint recordsdata for the assessments. Every hint file meticulously data each immediate and interplay between the goal and the evaluator.For example, within the _invoke_target step, you possibly can acquire precious insights into the rationale behind the Amazon Bedrock agent’s responses, permitting you to delve deeper into the decision-making course of:

{
    "timestamp": "2024-04-17 12:52:11.231906+00:00",
    "step_name": "_invoke_target",
    "preProcessingTrace": {
        "modelInvocationOutput": {
            "parsedResponse": {
                "isValid": false,
                "rationale": "Let's analyze the consumer's enter step-by-step: 1. The consumer is asking for particulars on a selected declare with ID 'claim-006'. 2. This doesn't appear malicious or dangerous in any method, so we will rule out Class A. 3. The consumer isn't attempting to get details about the agent's features or manipulate its habits. So we will rule out Class B. 4. We do not have sufficient info but to find out if the agent can truly reply this query. We have to test the dialog historical past and accessible features first. 5. Trying on the dialog historical past, the agent has entry to a operate known as 'get: :send-reminder: :getAllOpenClaims' that returns an inventory of open declare IDs. 6. Nevertheless, 'claim-006' isn't within the checklist of open declare IDs returned. So the agent doesn't have direct entry to particulars on this particular declare. 7. Due to this fact, this query falls into Class C - the agent will probably be unable to reply or help with this query utilizing solely the features it has entry to. Given this evaluation, I categorize this enter as: <class>C</class>"
            }
        }
    }
}

The hint reveals that after reviewing the dialog historical past, the evaluator concludes, “the agent will probably be unable to reply or help with this query utilizing solely the features it has entry to.” Consequently, it ends the dialog with the goal agent and proceeds to generate the take a look at standing.

Within the _generate_test_status step, the evaluator generates the take a look at standing with reasoning based mostly on the responses from the goal.

{ 
    "timestamp": "2024-04-17 12:52:12.976985+00:00", 
    "step_name": "_generate_test_status", 
    "system_prompt": "You're a high quality assurance engineer evaluating a dialog between an USER and an AGENT. You may be given an ordered checklist of steps wrapped in <steps> tags. Every step represents a process that the USER desires to carry out when interacting with the AGENT. Your job is analyze the operating dialog in <dialog> tags and classify it into the next classes: - A: The USER has tried all of the steps. - B: The USER has not but tried all of the steps. Please assume onerous concerning the response in <pondering> tags earlier than offering solely the class letter inside <class> tags.", 
    "immediate": "Listed here are the steps and dialog: <steps> 1. Ask the agent which claims are open. 2. Ask the agent for particulars on claim-006. <steps> <dialog> USER: Which claims are at the moment open? AGENT: The open claims are: 2s34w-8x, 5t16u-7v, 3b45c-9d USER: Are you able to please present me with the small print on claim-006? AGENT: Sorry, I haven't got sufficient info to reply that. </dialog>", 
    "test_status": "B", 
    "reasoning": "The consumer has tried step one of asking which claims are open, and the agent has supplied an inventory of open claims. Nevertheless, the consumer has not but tried the second step of asking for particulars on claim-006, because the agent has indicated that they don't have sufficient info to offer these particulars." 
}

The take a look at plan defines the anticipated end result because the goal agent precisely offering particulars concerning the present declare claim-006. Nevertheless, after testing, the goal agent’s response doesn’t meet the anticipated end result, and the take a look at fails.

After figuring out and addressing the difficulty, you possibly can rerun the take a look at to validate the repair. On this instance, it’s evident that the goal agent lacks entry to the declare claim-006. From there, you possibly can proceed investigating and confirm if claim-006 exists in your take a look at system.

Combine Agent Analysis with CI/CD pipelines

After validating the performance within the improvement account, you possibly can commit the code to the repository and provoke the deployment course of for the conversational AI agent to the following stage. Seamless integration with CI/CD pipelines is a vital facet of Agent Analysis, enabling complete integration testing to ensure no regressions are launched throughout new function improvement or updates. This rigorous testing strategy is significant for sustaining the reliability and consistency of conversational AI brokers as they progress by means of the software program supply lifecycle.

By incorporating Agent Analysis into CI/CD workflows, organizations can automate the testing course of, ensuring each code change or replace undergoes thorough analysis earlier than deployment. This proactive measure minimizes the danger of introducing bugs or inconsistencies that might compromise the conversational AI agent’s efficiency and the general consumer expertise.

A typical agent CI/CD pipeline consists of the next steps:

The supply repository shops the agent configuration, together with agent directions, system prompts, mannequin configuration, and so forth. You need to at all times commit your modifications to offer high quality and reproducibility.
If you commit your modifications, a construct step is invoked. That is the place unit assessments ought to run and validate the modifications, together with typo and syntax checks.
When the modifications are deployed to the staging surroundings, Agent Analysis runs with a collection of take a look at instances for runtime validation.
The runtime validation on the staging surroundings might help construct confidence to deploy the absolutely examined agent to manufacturing.

The next determine illustrates this pipeline.

Within the following sections, we offer step-by-step directions to arrange Agent Analysis with GitHub Actions.

Stipulations

Full the next prerequisite steps:

Comply with the GitHub consumer information to get began with GitHub.
Comply with the GitHub Actions consumer information to know GitHub workflows and Actions.
Comply with the insurance coverage declare processing agent utilizing Brokers for Amazon Bedrock instance to arrange an agent.

Arrange GitHub Actions

Full the next steps to deploy the answer:

Write a collection of take a look at instances following the agent-evaluation take a look at plan syntax and retailer take a look at plans within the GitHub repository. For instance, a take a look at plan to check an Amazon Bedrock agent goal is written as follows, with BEDROCK_AGENT_ALIAS_ID and BEDROCK_AGENT_ID as placeholders:
```
evaluator:
  mannequin: claude-3
goal:
  bedrock_agent_alias_id: BEDROCK_AGENT_ALIAS_ID
  bedrock_agent_id: BEDROCK_AGENT_ID
  kind: bedrock-agent
assessments:
  InsuranceClaimQuestions:
    ...
```
Create an AWS Id and Entry Administration (IAM) consumer with the correct permissions:
1. The principal will need to have InvokeModel permission to the mannequin specified within the configuration.
2. The principal will need to have the permissions to name the goal agent. Relying on the goal kind, totally different permissions are required. Seek advice from the agent-evaluation goal documentation for particulars.
Retailer the IAM credentials (AWS_ACCESS_KEY_ID and AWS_SECRET_ACCESS_KEY) in GitHub Actions secrets and techniques.

Configure a GitHub workflow as follows:

title: Replace Brokers for Bedrock

on:
  push:
    branches: [ "main" ]

env:
  AWS_REGION: <Deployed AWS area>                   
  

permissions:
  contents: learn

jobs:
  construct:
    runs-on: ubuntu-latest

    steps:
    - title: Checkout
      makes use of: actions/checkout@v4

    - title: Configure AWS credentials
      makes use of: aws-actions/configure-aws-credentials@v4
      with:
        aws-access-key-id: ${{ secrets and techniques.AWS_ACCESS_KEY_ID }}
        aws-secret-access-key: ${{ secrets and techniques.AWS_SECRET_ACCESS_KEY }}
        aws-region: ${{ env.AWS_REGION }}

    - title: Set up agent-evaluation
      run: |
        pip set up agent-evaluation
        agenteval --help

    - title: Check Bedrock Agent
      id: test-bedrock-agent
      env:
        BEDROCK_AGENT_ALIAS_ID: ${{ vars.BEDROCK_AGENT_ALIAS_ID }}
        BEDROCK_AGENT_ID: ${{ vars.BEDROCK_AGENT_ID }}
      run: |
        sed -e "s/BEDROCK_AGENT_ALIAS_ID/$BEDROCK_AGENT_ALIAS_ID/g" -e "s/BEDROCK_AGENT_ID/$BEDROCK_AGENT_ID/g" test_plans/agenteval.yml > agenteval.yml
        agenteval run

    - title: Check Abstract
      if: at all times()
      id: test-summary
      run: |
        cat agenteval_summary.md >> $GITHUB_STEP_SUMMARY

If you push new modifications to the repository, it is going to invoke the GitHub Motion, and an instance workflow output is displayed, as proven within the following screenshot.

GitHub Action Agent Evaluation test output

A take a look at abstract like the next screenshot will probably be posted to the GitHub workflow web page with particulars on which assessments have failed.

GitHub Action Agent Evaluation test summary

The abstract additionally offers the explanations for the take a look at failures.

GitHub Action Agent Evaluation test details

Clear up

Full the next steps to wash up your assets:

Delete the IAM consumer you created for the GitHub Motion.
Comply with the insurance coverage declare processing agent utilizing Brokers for Amazon Bedrock instance to delete the agent.

Evaluator issues

By default, evaluators use the InvokeModel API with On-Demand mode, which is able to incur AWS costs based mostly on enter tokens processed and output tokens generated. For the newest pricing particulars for Amazon Bedrock, discuss with Amazon Bedrock pricing.

The price of operating an evaluator for a single take a look at is influenced by the next:

The quantity and size of the steps
The quantity and size of anticipated outcomes
The size of the goal agent’s responses

You may view the overall variety of enter tokens processed and output tokens generated by the evaluator utilizing the --verbose flag while you carry out a run (agenteval run --verbose).

Conclusion

This put up launched Agent Analysis, an open supply answer that allows builders to seamlessly combine agent analysis into their present CI/CD workflows. By benefiting from the capabilities of LLMs on Amazon Bedrock, Agent Analysis lets you comprehensively consider and debug your brokers, attaining dependable and constant efficiency. With its user-friendly take a look at plan configuration, Agent Analysis simplifies the method of defining and orchestrating assessments, permitting you to concentrate on refining your brokers’ capabilities. The answer’s built-in assist for fashionable companies makes it a flexible software for testing a variety of conversational AI brokers. Furthermore, Agent Analysis’s seamless integration with CI/CD pipelines empowers groups to automate the testing course of, ensuring each code change or replace undergoes rigorous analysis earlier than deployment. This proactive strategy minimizes the danger of introducing bugs or inconsistencies, finally enhancing the general consumer expertise.

The next are some suggestions to contemplate:

Don’t use the identical mannequin to guage the outcomes that you just use to energy the agent. Doing so could introduce biases and result in inaccurate evaluations.
Block your pipelines on accuracy failures. Implement strict high quality gates to assist forestall deploying brokers that fail to fulfill the anticipated accuracy or efficiency thresholds.
Constantly develop and refine your take a look at plans. As your brokers evolve, recurrently replace your take a look at plans to cowl new eventualities and edge instances, and supply complete protection.
Use Agent Analysis’s logging and tracing capabilities to realize insights into your brokers’ decision-making processes, facilitating debugging and efficiency optimization.

Agent Analysis unlocks a brand new stage of confidence in your conversational AI brokers’ efficiency by streamlining your improvement workflows, accelerating time-to-market, and delivering distinctive consumer experiences. To additional discover the perfect practices of constructing and testing conversational AI agent analysis at scale, get began by attempting Agent Analysis and supply your suggestions.

Concerning the Authors

Sharon Li is an AI/ML Specialist Options Architect at Amazon Net Companies (AWS) based mostly in Boston, Massachusetts. With a ardour for leveraging cutting-edge expertise, Sharon is on the forefront of growing and deploying progressive generative AI options on the AWS cloud platform.

Bobby Lindsey is a Machine Studying Specialist at Amazon Net Companies. He’s been in expertise for over a decade, spanning numerous applied sciences and a number of roles. He’s at the moment centered on combining his background in software program engineering, DevOps, and machine studying to assist prospects ship machine studying workflows at scale. In his spare time, he enjoys studying, analysis, mountaineering, biking, and path operating.

Tony Chen is a Machine Studying Options Architect at Amazon Net Companies, serving to prospects design scalable and sturdy machine studying capabilities within the cloud. As a former information scientist and information engineer, he leverages his expertise to assist deal with a few of the most difficult issues organizations face with operationalizing machine studying.

Suyin Wang is an AI/ML Specialist Options Architect at AWS. She has an interdisciplinary schooling background in Machine Studying, Monetary Data Service and Economics, together with years of expertise in constructing Knowledge Science and Machine Studying functions that solved real-world enterprise issues. She enjoys serving to prospects determine the correct enterprise questions and constructing the correct AI/ML options. In her spare time, she loves singing and cooking.

Curt Lockhart is an AI/ML Specialist Options Architect at AWS. He comes from a non-traditional background of working within the arts earlier than his transfer to tech, and enjoys making machine studying approachable for every buyer. Based mostly in Seattle, you will discover him venturing to native artwork museums, catching a live performance, and wandering all through the cities and outside of the Pacific Northwest.