How Planview constructed a scalable AI Assistant for portfolio and undertaking administration utilizing Amazon Bedrock

This publish is co-written with Lee Rehwinkel from Planview.

Companies right this moment face quite a few challenges in managing intricate tasks and applications, deriving invaluable insights from huge information volumes, and making well timed selections. These hurdles continuously result in productiveness bottlenecks for program managers and executives, hindering their potential to drive organizational success effectively.

Planview, a number one supplier of related work administration options, launched into an formidable plan in 2023 to revolutionize how 3 million international customers work together with their undertaking administration purposes. To understand this imaginative and prescient, Planview developed an AI assistant known as Planview Copilot, utilizing a multi-agent system powered by Amazon Bedrock.

Creating this multi-agent system posed a number of challenges:

• Reliably routing duties to applicable AI brokers
• Accessing information from numerous sources and codecs
• Interacting with a number of software APIs
• Enabling the self-serve creation of latest AI expertise by totally different product groups

To beat these challenges, Planview developed a multi-agent structure constructed utilizing Amazon Bedrock. Amazon Bedrock is a totally managed service that gives API entry to basis fashions (FMs) from Amazon and different main AI startups. This enables builders to decide on the FM that’s finest suited to their use case. This method is each architecturally and organizationally scalable, enabling Planview to quickly develop and deploy new AI expertise to satisfy the evolving wants of their prospects.

This publish focuses totally on the primary problem: routing duties and managing a number of brokers in a generative AI structure. We discover Planview’s method to this problem throughout the improvement of Planview Copilot, sharing insights into the design selections that present environment friendly and dependable activity routing.

We describe custom-made home-grown brokers on this publish as a result of this undertaking was applied earlier than Amazon Bedrock Brokers was typically out there. Nonetheless, Amazon Bedrock Brokers is now the beneficial answer for organizations trying to make use of AI-powered brokers of their operations. Amazon Bedrock Brokers can retain reminiscence throughout interactions, providing extra personalised and seamless consumer experiences. You possibly can profit from improved suggestions and recall of prior context the place required, having fun with a extra cohesive and environment friendly interplay with the agent. We share our learnings in our answer that will help you understanding easy methods to use AWS know-how to construct options to satisfy your targets.

Answer overview

Planview’s multi-agent structure consists of a number of generative AI parts collaborating as a single system. At its core, an orchestrator is liable for routing questions to numerous brokers, amassing the realized data, and offering customers with a synthesized response. The orchestrator is managed by a central improvement crew, and the brokers are managed by every software crew.

The orchestrator includes two principal parts known as the router and responder, that are powered by a giant language mannequin (LLM). The router makes use of AI to intelligently route consumer questions to numerous software brokers with specialised capabilities. The brokers may be categorized into three principal varieties:

• Assist agent – Makes use of Retrieval Augmented Era (RAG) to offer software assist
• Knowledge agent – Dynamically accesses and analyzes buyer information
• Motion agent – Runs actions inside the software on the consumer’s behalf

After the brokers have processed the questions and supplied their responses, the responder, additionally powered by an LLM, synthesizes the realized data and formulates a coherent response to the consumer. This structure permits for a seamless collaboration between the centralized orchestrator and the specialised brokers, which offers customers an correct and complete solutions to their questions. The next diagram illustrates the end-to-end workflow.

Technical overview

Planview used key AWS providers to construct its multi-agent structure. The central Copilot service, powered by Amazon Elastic Kubernetes Service (Amazon EKS), is liable for coordinating actions among the many numerous providers. Its obligations embrace:

• Managing consumer session chat historical past utilizing Amazon Relational Database Service (Amazon RDS)
• Coordinating visitors between the router, software brokers, and responder
• Dealing with logging, monitoring, and amassing user-submitted suggestions

The router and responder are AWS Lambda features that work together with Amazon Bedrock. The router considers the consumer’s query and chat historical past from the central Copilot service, and the responder considers the consumer’s query, chat historical past, and responses from every agent.

Software groups handle their brokers utilizing Lambda features that work together with Amazon Bedrock. For improved visibility, analysis, and monitoring, Planview has adopted a centralized immediate repository service to retailer LLM prompts.

Brokers can work together with purposes utilizing numerous strategies relying on the use case and information availability:

• Current software APIs – Brokers can talk with purposes by their present API endpoints
• Amazon Athena or conventional SQL information shops – Brokers can retrieve information from Amazon Athena or different SQL-based information shops to offer related data
• Amazon Neptune for graph information – Brokers can entry graph information saved in Amazon Neptune to help complicated dependency evaluation
• Amazon OpenSearch Service for doc RAG – Brokers can use Amazon OpenSearch Service to carry out RAG on paperwork

The next diagram illustrates the generative AI assistant structure on AWS.

Router and responder pattern prompts

The router and responder parts work collectively to course of consumer queries and generate applicable responses. The next prompts present illustrative router and responder immediate templates. Extra immediate engineering can be required to enhance reliability for a manufacturing implementation.

First, the out there instruments are described, together with their goal and pattern questions that may be requested of every software. The instance questions assist information the pure language interactions between the orchestrator and the out there brokers, as represented by instruments.

instruments=""'
<software>
<toolName>applicationHelp</toolName>
<toolDescription>
Use this software to reply software assist associated questions.
Instance questions:
How do I reset my password?
How do I add a brand new consumer?
How do I create a activity?
</toolDescription>
</software>
<software>
<toolName>dataQuery</toolName>
<toolDescription>
Use this software to reply questions utilizing software information.
Instance questions:
Which duties are assigned to me?
What number of duties are due subsequent week?
Which activity is most in danger?
</toolDescription>
</software>

Subsequent, the router immediate outlines the rules for the agent to both reply on to consumer queries or request data by particular instruments earlier than formulating a response:

system_prompt_router = f'''
<function>
Your job is to resolve if you happen to want further data to completely reply the Consumer's 
questions.
You obtain your purpose by selecting both 'reply' or 'callTool'.
You've got entry to your chat historical past in <chatHistory></chatHistory> tags.
You even have an inventory of obtainable instruments to help you in <instruments></instruments> tags.
</function>
<chatHistory>
{chatHistory}
</chatHistory>
<instruments>
{instruments}
</instruments>
<guidelines>
- If the chat historical past incorporates adequate data to reply the Consumer's questions, 
select the 'reply' motion.
- To collect extra data earlier than responding, select the 'callTool' motion.
- You a lot solely select from the instruments within the <instruments></instruments> tags.
- If no software can help with the query, select the 'reply' motion.
- Place your chosen motion inside <motion></motion> tags.
- Whenever you selected the 'callTool' motion, present the <toolName> and the <toolQuestion> you
wish to ask.
- Your <toolQuestion> needs to be verbose and keep away from utilizing pronouns.
- Begin by offering your step-by-step considering in <considering></considering> tags.
- Then you'll give your reply in <reply></reply> tags.
- Your reply ought to comply with the format of certainly one of these three examples:
When selecting the 'reply' motion, your reply ought to comply with the under instance EXACTLY:
<instance>
<reply>
<motion>reply</motion>
</reply>
</instance>
When selecting the 'callTool' motion for a single Device:
<instance>
<reply>
<motion>callTool</motion>
<instruments>
<software>
<toolName>applicationHelp</toolName>
<toolQuestion>How do I reset my password?</toolQuestion>
</software>
</instruments>
</reply>
</instance>
Executing the above, would produce the next end result:
You can even name a number of Instruments utilizing this format:
<instance>
<reply>
<motion>callTool</motion>
<instruments>
<software>
<toolName>dataQuery</toolName>
<toolQuestion>What number of duties are assigned to me?</toolQuestion>
</software>
<software>
<toolName>applicationHelp</toolName>
<toolQuestion>How do I add a brand new activity?</toolQuestion>
</software>
</instruments>
</reply>
</instance>
</guidelines>
'''

The next is a pattern response from the router part that initiates the dataQuery software to retrieve and analyze activity assignments for every consumer:

<considering>
To find out who has probably the most duties assigned, I might want to question the applying information. The "dataQuery" software appears most applicable for this query.
</considering>

<reply>
<motion>callTool</motion>
<instruments>
    <software>
        <toolName>dataQuery</toolName>
        <toolQuestion>Which consumer has probably the most duties at present assigned to them?   </toolQuestion>
    </software>
</instruments>
</reply>

The next is a pattern response from the responder part that makes use of the dataQuery software to fetch details about the consumer’s assigned duties. It studies that the consumer has 5 duties assigned to them.

<considering>
Primarily based on the chat historical past, I beforehand known as the dataQuery software to ask "What number of duties are at present assigned to the consumer?". The software responded that the consumer has 5 duties assigned to them.
</considering>

<reply>
Based on the information I queried beforehand, you have got 5 duties assigned to you.
</reply>

Mannequin analysis and choice

Evaluating and monitoring generative AI mannequin efficiency is essential in any AI system. Planview’s multi-agent structure allows evaluation at numerous part ranges, offering complete high quality management regardless of the system’s complexity. Planview evaluates parts at three ranges:

• Prompts – Assessing LLM prompts for effectiveness and accuracy
• AI brokers – Evaluating full immediate chains to keep up optimum activity dealing with and response relevance
• AI system – Testing user-facing interactions to confirm seamless integration of all parts

The next determine illustrates the analysis framework for prompts and scoring.

To conduct these evaluations, Planview makes use of a set of fastidiously crafted check questions that cowl typical consumer queries and edge instances. These evaluations are carried out throughout the improvement part and proceed in manufacturing to trace the standard of responses over time. At present, human evaluators play an important function in scoring responses. To assist within the analysis, Planview has developed an inside analysis software to retailer the library of questions and monitor the responses over time.

To evaluate every part and decide probably the most appropriate Amazon Bedrock mannequin for a given activity, Planview established the next prioritized analysis standards:

• High quality of response – Assuring accuracy, relevance, and helpfulness of system responses
• Time of response – Minimizing latency between consumer queries and system responses
• Scale – Ensuring the system can scale to hundreds of concurrent customers
• Value of response – Optimizing operational prices, together with AWS providers and generative AI fashions, to keep up financial viability

Primarily based on these standards and the present use case, Planview chosen Anthropic’s Claude 3 Sonnet on Amazon Bedrock for the router and responder parts.

Outcomes and affect

Over the previous yr, Planview Copilot’s efficiency has considerably improved by the implementation of a multi-agent structure, improvement of a strong analysis framework, and adoption of the newest FMs out there by Amazon Bedrock. Planview noticed the next outcomes between the primary technology of Planview Copilot developed mid-2023 and the newest model:

• Accuracy – Human-evaluated accuracy has improved from 50% reply acceptance to now exceeding 95%
• Response time – Common response occasions have been diminished from over 1 minute to twenty seconds
• Load testing – The AI assistant has efficiently handed load checks, the place 1,000 questions have been submitted simultaneous with no noticeable affect on response time or high quality
• Value-efficiency – The associated fee per buyer interplay has been slashed to at least one tenth of the preliminary expense
• Time-to-market – New agent improvement and deployment time has been diminished from months to weeks

Conclusion

On this publish, we explored how Planview was in a position to develop a generative AI assistant to deal with complicated work administration course of by adopting the next methods:

• Modular improvement – Planview constructed a multi-agent structure with a centralized orchestrator. The answer allows environment friendly activity dealing with and system scalability, whereas permitting totally different product groups to quickly develop and deploy new AI expertise by specialised brokers.
• Analysis framework – Planview applied a strong analysis course of at a number of ranges, which was essential for sustaining and bettering efficiency.
• Amazon Bedrock integration – Planview used Amazon Bedrock to innovate sooner with broad mannequin alternative and entry to numerous FMs, permitting for versatile mannequin choice primarily based on particular activity necessities.

Planview is migrating to Amazon Bedrock Brokers, which allows the combination of clever autonomous brokers inside their software ecosystem. Amazon Bedrock Brokers automate processes by orchestrating interactions between basis fashions, information sources, purposes, and consumer conversations.

As subsequent steps, you may discover Planview’s AI assistant function constructed on Amazon Bedrock and keep up to date with new Amazon Bedrock options and releases to advance your AI journey on AWS.

About Authors

Sunil Ramachandra is a Senior Options Architect enabling hyper-growth Unbiased Software program Distributors (ISVs) to innovate and speed up on AWS. He companions with prospects to construct extremely scalable and resilient cloud architectures. When not collaborating with prospects, Sunil enjoys spending time with household, operating, meditating, and watching motion pictures on Prime Video.

Benedict Augustine is a thought chief in Generative AI and Machine Studying, serving as a Senior Specialist at AWS. He advises buyer CxOs on AI technique, to construct long-term visions whereas delivering rapid ROI.As VP of Machine Studying, Benedict spent the final decade constructing seven AI-first SaaS merchandise, now utilized by Fortune 100 corporations, driving vital enterprise affect. His work has earned him 5 patents.

Lee Rehwinkel is a Principal Knowledge Scientist at Planview with 20 years of expertise in incorporating AI & ML into Enterprise software program. He holds superior levels from each Carnegie Mellon College and Columbia College. Lee spearheads Planview’s R&D efforts on AI capabilities inside Planview Copilot. Outdoors of labor, he enjoys rowing on Austin’s Girl Fowl Lake.