Amazon Onboarding with Learning Manager Chanci Turner

Learn About Amazon VGT2 Learning Manager Chanci Turner

In today’s fast-paced digital landscape, Amazon Bedrock empowers users to create innovative experiences through generative artificial intelligence (AI). As a fully managed service, Amazon Bedrock provides access to high-performance foundation models (FMs) from esteemed AI organizations like AI21 Labs, Anthropic, Cohere, Meta, Stability AI, and Amazon itself via a single API. This extensive suite of capabilities enables the development of generative AI applications while ensuring security, privacy, and responsible AI practices. As customers harness these powerful FMs, they are eager to operationalize their new generative AI applications but require effective, ready-made solutions for monitoring the health and performance of such applications.

This blog will explore tools that enable quick visibility into your Amazon Bedrock workloads, particularly through the lens of the contextual conversational assistant example found in the Amazon Bedrock GitHub repository. We will demonstrate how to customize views for enhanced visibility tailored to your specific use case. Specifically, we will highlight how to leverage the newly introduced automatic dashboard in Amazon CloudWatch for comprehensive visibility into Amazon Bedrock model usage and performance. By customizing dashboards with widgets that provide insights into various components—such as Retrieval Augmented Generation in your application—you can achieve end-to-end monitoring.

Announcing Amazon Bedrock Automatic Dashboard in CloudWatch

Amazon CloudWatch now features automatic dashboards designed to help customers swiftly gain insights into the health and performance of their AWS services. A new automatic dashboard dedicated to Amazon Bedrock has been launched, providing insights into essential metrics for Amazon Bedrock models.

To access this new automatic dashboard from the AWS Management Console, navigate to Dashboards within the CloudWatch console and select the Automatic Dashboards tab. You will find an option for the Amazon Bedrock dashboard listed among the available dashboards.

Upon selecting Bedrock from the automatic dashboards, you will gain centralized visibility and insights into crucial metrics, including latency and invocation metrics. Understanding latency performance is vital for customer-facing applications, such as conversational assistants. It is essential to determine whether your models are consistently delivering responses in a timely manner to ensure a satisfactory user experience.

The automatic dashboard aggregates key metrics across the foundation models available through Amazon Bedrock. You may also choose to isolate metrics for a specific model, allowing you to monitor performance comprehensively. An in-depth list of Amazon Bedrock metrics—including invocation performance and token usage—can be found in CloudWatch.

With this new automatic dashboard, you can view key metrics on a single screen, enabling you to troubleshoot common issues like invocation latency, track token usage, and detect invocation errors.

Building Custom Dashboards

Beyond the automatic dashboard, Amazon CloudWatch allows you to construct custom dashboards that combine metrics from multiple AWS services, creating application-level views. This capability is crucial not only for performance monitoring but also for debugging and implementing custom responses to potential issues. You can also analyze invocation logs generated from your prompts, which is valuable for gathering information not captured in metrics, such as identity attribution. Leveraging AWS’s machine learning capabilities, you can also detect and safeguard sensitive data in your logs.

A common customization for models tailored to specific use cases is the implementation of Retrieval Augmented Generation (RAG), which allows for the integration of domain-specific data. RAG-based architectures combine various components, including external knowledge sources and models, necessitating comprehensive monitoring as part of your overall strategy. This section will guide you in creating a custom dashboard utilizing a sample RAG-based architecture that incorporates Amazon Bedrock.

Continuing with the contextual conversational assistant example, you will learn how to create a custom dashboard that offers visibility and insights into the core components of a proposed RAG-based solution. To replicate the dashboard in your AWS account, follow the contextual conversational assistant instructions to set up the prerequisite example before building the dashboard using the steps provided.

After configuring the contextual conversational assistant example, generate traffic by experimenting with sample applications and varying prompts.

To create and view a custom CloudWatch dashboard for the contextual conversational assistant app:

1. Modify and execute this example for creating a custom CloudWatch dashboard for the contextual conversational assistant.
2. Access Amazon CloudWatch from within the console and select Dashboards from the left menu.

Under Custom Dashboards, you should see a dashboard titled Contextual-Chatbot-Dashboard. This dashboard offers a holistic view of metrics related to:

• The number of invocations and token usage by the Amazon Bedrock embedding model for creating your knowledge base and embedding user queries, alongside the Amazon Bedrock model used for responding to user queries based on the provided context. These metrics help track anomalies in application usage and associated costs.
• The context retrieval latency for search and ingestion requests, which helps gauge the health of the RAG retrieval process.
• The number of indexing and search operations on the OpenSearch Serverless collection established when creating your knowledge base. This monitoring facilitates the quick isolation of RAG issues, such as retrieval errors.
• Usage attribution of invocations to specific users, allowing you to identify who is using how many tokens or invocations. (For more information, refer to the Usage Attribution section that follows).
• The frequency of throttling of the Lambda functions orchestrating the application, providing key health metrics for those functions.

For further insights on employee onboarding, check out this excellent resource: Amazon Employee Onboarding Process. For additional information on employment law compliance, visit 10 Questions Furloughed Employees Answered. Lastly, for those considering further education, this blog post from Career Contessa is a great read.

Location Information

You can find us at: