In this article, we explore the concept of embeddings, demonstrate their practical application, and examine how they can enhance functionalities like zero-shot classification and semantic search. We will illustrate these features using Amazon Bedrock and language embeddings in a straightforward RSS aggregator application.
Amazon Bedrock is a fully managed service that provides access to foundation models (FMs) from top AI startups and Amazon via an API, allowing you to select the model that best fits your specific needs. With Amazon Bedrock’s serverless infrastructure, you can quickly get started, tailor FMs with your own data, and seamlessly integrate them into your applications using AWS services without the hassle of managing infrastructure. For this discussion, we utilize the Cohere v3 Embed model within Amazon Bedrock to generate our language embeddings.
Use Case: RSS Aggregator
To showcase the potential applications of these language embeddings, we developed a website that aggregates RSS feeds. RSS, or Really Simple Syndication, is a web feed format that allows publications to deliver updates in a standardized, machine-readable format. On our site, users can subscribe to RSS feeds to receive a categorized list of new articles. We leverage embeddings to introduce the following features:
- Zero-shot classification: Articles are categorized into various topics, including default categories such as Technology, Politics, and Health & Wellbeing. Users also have the option to create custom topics.
- Semantic search: Users can perform semantic searches on articles, allowing them to refine their searches not only by topic but also by aspects like tone or style.
This article serves as a reference for the technical implementation of both semantic search and zero-shot classification features.
Solution Overview
Our solution incorporates the following services:
- Amazon API Gateway: The API is accessible via Amazon API Gateway, with caching managed by Amazon CloudFront to reduce database load for certain topics.
- Amazon Bedrock with Cohere v3 Embed: Articles and topics are transformed into embeddings using Amazon Bedrock and Cohere v3 Embed.
- Amazon CloudFront and Amazon S3: The single-page React application is hosted on Amazon S3 and Amazon CloudFront.
- Amazon Cognito: User authentication is handled through Amazon Cognito user pools.
- Amazon EventBridge: New updates are coordinated using Amazon EventBridge and scheduled events.
- AWS Lambda: The API runs as a Fastify application written in TypeScript, hosted on AWS Lambda.
- Amazon Aurora PostgreSQL-Compatible Edition and pgvector: This database is used both for application functionality and as a vector store via pgvector.
- Amazon RDS Proxy: It facilitates connection pooling.
- Amazon SQS: This service queues events, processing one at a time to avoid exceeding the rate limits of Cohere in Amazon Bedrock.
The following diagram illustrates the architecture of our solution.
What Are Embeddings?
This section provides a brief overview of what embeddings are and how they function.
Embeddings are numerical representations of various concepts or objects, such as language or images. In this discussion, we focus on language embeddings. By converting these concepts into numerical forms, we enable computers to comprehend and manipulate them.
For example, consider the words Berlin and Paris. While humans can easily recognize the conceptual connections between these two cities, such as being capitals of their respective countries and both located in Europe, computers lack an innate understanding of these relationships.
To enable computers to recognize these concepts, we transform them into language embeddings. These embeddings are high-dimensional vectors that learn to express their relationships through training within a neural network. During this training, the network is exposed to vast amounts of text and learns patterns based on how words co-occur and relate to one another in various contexts.
Embedding vectors empower computers to model language-based concepts. For example, embedding “Berlin” and “Paris” allows us to conduct mathematical operations on these vectors. We can identify fascinating relationships, such as Paris – France + Germany ≈ Berlin. This result emerges because the embeddings capture the relationships between “Paris” and “France” and between “Germany” and “Berlin,” illustrating that both cities are capital cities.
For our application, we utilize the pre-trained Cohere Embeddings model in Amazon Bedrock, which embeds entire texts instead of individual words. These embeddings encapsulate the meaning of the text and can undergo mathematical operations, making them valuable for assessing text similarity.
Zero-Shot Classification
We utilize language embeddings to assess how closely an article aligns with a given topic. This is achieved by breaking down a topic into multiple related embeddings. For instance, the topic of culture may include embeddings for sports, TV shows, music, and literature. We then embed the titles and descriptions of RSS articles and calculate their similarity to the topic embeddings, allowing us to assign relevant topic labels to each article.
The embeddings generated by Cohere are high-dimensional, containing 1,024 values (or dimensions). For illustrative purposes, we use an algorithm to reduce the dimensionality of these embeddings, specifically t-distributed Stochastic Neighbor Embedding (t-SNE), enabling us to visualize them in two dimensions. This visualization helps demonstrate how topics cluster based on similarity and meaning.
By comparing an article’s embedding to the topic embeddings, we can classify it according to the most closely associated topic. This process employs the k-nearest neighbor (k-NN) algorithm, which allows us to make assumptions about a data point based on its proximity to other points. For example, if an article’s embedding is close to the music topic, we can classify it under the culture topic.
For further insights on this topic, check out another blog post here. Additionally, for authoritative information, visit this resource, they are experts in this field. You can also explore this excellent resource for a deeper understanding.
Leave a Reply