In bustling urban centers, finding parking can be a daunting task, especially on a busy Friday night. Imagine the challenge for those designated drivers tasked with locating and paying for an expensive parking spot! Enter Freebird, a unique rewards platform that incentivizes users to utilize rideshare services like Uber and Lyft, funded by advertisers and businesses.
Freebird leverages the growing trend of ridesharing to assist businesses in crowded areas with limited parking spaces, boosting foot traffic. Customers benefit from cash-back rewards when they spend at participating establishments by linking their payment methods to the app. It’s a win-win for all involved as Freebird strives to transform transportation into a more enjoyable experience.
John Smith, CTO
As an early adopter of AWS Step Functions – Express Workflows (introduced in December 2019), Freebird transitioned its high-volume, low-latency data processing to a fully serverless model, achieving over 33% savings in infrastructure costs while modernizing its application stack. The startup sought workflows capable of scaling with minimal operational overhead, effectively managing the complexities of their data pipeline. The solution needed to empower the development team to easily extend this workflow using familiar tools and languages. CTO John Smith shared the journey the company undertook to reach this innovative solution.
Smith discovered AWS Step Functions – Express Workflows during AWS Re:Invent 2019 while searching for a micro-service orchestration tool. “The standout feature of AWS Step Functions for me was its robust support for executing each step as an independent process (i.e., AWS Lambda), enabling parallelization and featuring built-in error handling with retries,” he explains.
Freebird heavily relies on data from rideshare providers like Uber and Lyft, but the incoming data structure varies by source. Processing this data demands multiple steps, necessitating a solution that could gracefully handle potential failure cases. The startup required a system capable of managing branching logic based on data sources, coordinating with external APIs, and addressing transient errors effectively. They needed a meticulous method for inspecting these transient errors, and for certain issues, optimized retries with exponential backoff. Moreover, they wanted to monitor the workflow at specific points to accurately assess whether failure rates were genuine concerns or just normal fluctuations.
Prior to the launch of Express Workflows, the development team considered two options: using Apache Airflow or coding the application from scratch. They ultimately decided to implement their own solution, running the entire workflow within a single process in containers deployed to AWS EKS, triggered by AWS CloudWatch scheduled events or cron jobs. They operated under this model for approximately two years, facing challenges in achieving parallelization without risking unhandled exceptions that could derail the entire workflow. This challenge was compounded as the workflow iterated over a set of records, where even a single item’s failure could halt the entire process, complicating failure management.
Afterward, they explored Apache Airflow, utilizing it for a year until server maintenance became cumbersome. The need for manual provisioning, installation, security, monitoring, patching, upgrades, and scaling required dedicated resources with specialized expertise, of which only one team member had sufficient knowledge. Smith identified the reliance on siloed expertise and the significant operational burden as substantial risks to further investment in their tech stack. Consequently, they decided to overhaul their entire monolithic backend architecture, transitioning to a predominantly serverless model, with the exception of their databases.
Upon migrating to AWS Step Functions, Freebird was able to reuse some of its existing workflow code. “This model also made our code more manageable,” notes Smith. “Data structures became simpler, and our code was less cluttered with intricate failure state handling.”
Freebird opted for Express Workflows due to the high volume of real-time webhook data from rideshare providers. This incoming data is routed through Amazon API Gateway and directly integrated into an Amazon Kinesis data stream. This direct push into the Kinesis stream enhances data resilience. An AWS Lambda function retrieves data from the stream and initiates an AWS Step Functions – Express Workflow, which meticulously validates, processes, and normalizes the incoming data. Ultimately, the processed data is emitted to an SNS topic for downstream service subscriptions. If downstream issues arise, Kinesis can replay the webhook events in their original order.
How Freebird Integrates AWS Step Functions into Their Data Processing Workflow
“Using AWS Step Functions, particularly Express Workflows, has significantly benefited us. We can manage known failure cases with retries and exponential back-off effectively,” Smith states. “Integrating a new external source is as straightforward as adding a couple of tasks to our workflow. The compatibility of Step Functions with AWS Lambda allows us to utilize familiar tools and languages, helping us maintain a serverless operations focus. We’re only charged for what we use at the request level, so reduced volumes translate to lower costs.”
Looking ahead, AWS Step Functions – Express Workflows will enable Freebird to effortlessly scale and adapt to new third-party integrations, as well as process data for other business applications, including streaming from marketing technology tools to analyze marketing spend. For more insights on building a serverless workflow with AWS Step Functions, check out this useful resource here. Additionally, you may find valuable information from this authority on the topic. For further exploration of similar experiences, visit this excellent resource.
Leave a Reply