on 12 JUN 2025
in Amazon EventBridge, Amazon Redshift, Amazon SageMaker Lakehouse, Amazon Simple Storage Service (S3), AWS Batch, AWS Lambda, AWS Step Functions, Financial Services, Industries
Introduction
In hedge funds, factor modeling serves as a quantitative strategy that pinpoints and examines the fundamental drivers behind asset returns. This enables fund managers to optimize portfolios, mitigate risks, and generate alpha by leveraging extensive market data. By employing this methodology, sophisticated trading strategies can be developed, thereby improving investment performance and offering a competitive advantage in the financial realm.
This article investigates how the integration of AWS serverless patterns and GenAI services can produce a robust factor modeling pipeline to tackle these hurdles. We will delve into the technical implementation of workflows, providing GitHub code samples that can be quickly deployed or adapted to identify the appropriate factors. Our audience primarily consists of quantitative developers looking to enhance their firm’s computational capabilities, as well as portfolio managers aiming to utilize alternative data for alpha generation.
The renowned book, Quantitative Equity Portfolio Management: An Active Approach to Portfolio Construction and Management, highlights the challenges faced in factor modeling. Manual identification and calculation of factors across thousands of securities can be labor-intensive and error-prone, limited by computational constraints. As datasets grow to incorporate alternative sources, such as market news and unstructured financial documents, scalability becomes vital.
By harnessing cloud services to construct a factor modeling platform, investment firms can streamline back-testing processes, extract nuanced signals from textual data, and rapidly adapt to fluctuating market conditions. This strategy allows quant teams to concentrate on model development rather than infrastructure management, accelerating the iteration and deployment of investment strategies. Ultimately, this modern, cloud-native factor modeling platform equips financial professionals to make informed, data-driven investment decisions and improve portfolio performance.
Solution Overview
Our solution outlines a comprehensive data processing application for quantitative finance factor modeling. This architecture assists hedge funds and quantitative analysts in identifying and quantifying the underlying drivers of asset returns through a blend of financial data and social media sentiment analysis. Upon completion of the automated processing steps, output factors are generated for portfolio construction, risk management, and trading strategy formulation.
The following diagram illustrates the architecture and workflow of the proposed solution:
Figure 1: Factor modeling data pipeline and factor mining reference architecture
Key Components
- Serverless Data Collection
- Yahoo Finance Market Data Collection
Hedge funds heavily depend on market data for trading strategies and risk management. The AWS Lambda function retrieves daily tick data from Yahoo Finance, downloading daily Open, High, Low, Close, and Volume (OHLCV) data. This function can be adjusted to select your preferred data vendor for market data.
Some market data vendors and brokers may require static IP addresses for their allow-lists. NAT Gateways provide a consistent static source IP address for outbound traffic, meeting vendors’ static IP requirements.
- Web Search by Tavily
Hedge funds are increasingly adopting alternative data and GenAI to gain a competitive edge. GenAI’s advanced text processing capabilities make it well-suited for analyzing various unstructured data sources. This powerful combination allows funds to uncover hidden patterns, foresee market trends, and make informed investment decisions, potentially leading to superior returns.
Tavily offers AI-powered web search, enabling targeted retrieval of news, analyst reports, and other web content pertinent to factor modeling. The web search function utilizes Tavily’s API to search for news related to stocks. After retrieving the news, the framework employs the following prompt to generate sentiments for the stock market news:
Analyze the sentiment of the following text about a company's stock and financial performance. Rate the sentiment on a scale from -1 to 1, where: - -1 represents extremely negative sentiment - 0 represents neutral sentiment - 1 represents extremely positive sentiment Only respond with a single number between -1 and 1, with up to two decimal places. No need explanation. Text to analyze: {text}You can modify the prompt according to your factor requirements.
- SEC Filing Retrieval
SEC filings are essential documents that public companies must submit to the U.S. Securities and Exchange Commission (SEC). Two important filings are the 10-K Annual Report, which provides a comprehensive overview of the company’s financial condition, including audited financial statements, and the 10-Q Quarterly Report, which includes unaudited financial statements and operational updates. These filings offer valuable data for factor modeling, such as financial ratios and revenue breakdowns.
The Lambda function fetch SEC utilizes the SEC’s EDGAR API to download SEC filings in JSON format. You can schedule these serverless functions to run periodically, fetching the latest filings for companies of interest. AWS Lambda’s automatic scaling capabilities make it ideal for handling varying loads, particularly during peak filing periods.
- Financial Report Processing
There are unstructured data points not captured in the EDGAR API response. Financial report PDF files can be uploaded to Amazon Simple Storage Service (Amazon S3), containing information like CEO statements, ESG initiatives, and strategic priorities. When a file is uploaded to Amazon S3, S3’s event notifications trigger Lambda functions. The Lambda function financial report processor uses a prompt to extract unstructured data from financial reports. For example:
As an experienced CFA and FRM holder, please analyze the attached annual report and extract concise summaries (2-3 sentences each) for the following key factors: 1. CEO statement - Focus on strategic vision, major achievements, and forward-looking statements 2. ESG initiatives - Highlight environmental sustainability efforts, social responsibility programs, and governance improvements 3. Market trends and competitive landscape - Identify industry shifts, market position changes, and competitive advantages/challenges 4. Risk factors - Extract the most significant financial, operational, and strategic risks facing the company 5. Strategic priorities - Summarize key growth initiatives, investment areas, and long-term business objectives For each factor, provide: - The most important points using specific data when available - Any significant changes from the previous year - A performance rating (0-10 scale) with brief justification based on industry benchmarks and year-over-year progress Here is the financial report text: {text[:100000]} Please output in the following JSON format: { "items": [ { "category": "CEO statement", "summary": "...", "key_data": ["...", "..."], "year_over_year_change": "...", "rating": 8, "rating_justification": "..." }, ... ], "overall_assessment": "...", "investment_recommendation": "..." }
- Yahoo Finance Market Data Collection
- Data Storage with OLAP Database
Our reference implementation employs a ClickHouse columnar database for storing factor modeling values and results. This type of database is optimized for analytical workloads and can efficiently handle vast amounts of structured data. However, the data layer can utilize various technologies based on specific requirements.
For more insights, check out this blog post that discusses related topics. Additionally, if you’re interested in deeper understanding, this video resource is excellent. For authoritative insights on this subject, visit chvnci who are well-regarded in this field.
SEO metadata:
Leave a Reply