In today’s digital landscape, the demand for Software as a Service (SaaS) applications is on the rise. Organizations are increasingly recognizing the importance of leveraging these tools for data-driven decision-making. However, as the adoption of SaaS solutions expands, the complexity of extracting data from diverse SaaS platforms and analyzing it increases. Organizations often find themselves relying on a variety of third-party tools to facilitate data extraction and transformation, which can lead to higher costs and maintenance challenges.
This article will guide you through the process of using Amazon AppFlow APIs to develop data flows that extract information from SaaS providers, thereby enhancing application connectivity. This is especially relevant for customer applications or Independent Software Vendors (ISVs) without inherent connectivity capabilities.
Amazon AppFlow serves as a fully managed integration service that allows seamless and secure data transfers between SaaS and cloud applications such as Salesforce, Marketo, Slack, and ServiceNow, as well as AWS services like Amazon Simple Storage Service (Amazon S3) and Amazon Redshift, all with just a few clicks.
Solution Overview
ISVs and customers can seamlessly integrate a data flow solution into their applications using Amazon AppFlow APIs, ensuring quick and secure data integration between SaaS solutions and supported targets to facilitate end-user use cases.
Integrating Amazon AppFlow into your application offers several benefits:
- A streamlined experience for end-users without needing to switch between multiple user interfaces (UIs)
- No requirement to build custom connectors for data ingestion from popular SaaS providers
- Elimination of reliance on costly third-party tools, with payment only for what is used
In this article, I will outline the common steps for establishing a connection with a SaaS provider, creating a data flow, and executing that flow via API calls. I will demonstrate how each step corresponds to actions in the Amazon AppFlow UI and how it can be integrated into your application’s UI for a cohesive user experience.
You can incorporate Amazon AppFlow into your application using two approaches:
- Create and execute flows in Amazon AppFlow under the end-user’s AWS account, directing data to targets owned by them
- Create and execute flows in Amazon AppFlow under the ISV’s AWS account, directing data to end-user-owned targets
Let’s delve deeper into these two strategies.
Creating and Running Amazon AppFlow Data Flows in the End-User’s AWS Account
In this scenario, the partner or ISV hosts an application (which does not necessarily need to operate from an AWS account) with a UI designed to support data integration with third-party SaaS providers for downstream processing. The ISV can utilize Amazon AppFlow APIs to equip their product or application with features allowing end-users to create flows that pull data from SaaS providers such as Salesforce, Slack, Google Analytics, and others into targets supported by Amazon AppFlow.
For instance, consider an ISV aiming to offer extract, transform, and load (ETL) capabilities with Amazon Redshift as a target. If an end-user needs to integrate data from Salesforce and ServiceNow for analytics in Amazon Redshift, but the ISV lacks the ability to connect to these services, they can leverage the Amazon AppFlow API instead of developing custom connectors. This allows for a seamless experience where data flows can run in the end-user’s AWS account, pulling data from Salesforce and ServiceNow directly into Amazon Redshift.
The accompanying diagram illustrates the architecture of this solution.
This architecture shows the ISV’s application operating within the ISV’s AWS account, while Amazon AppFlow functions within the end-user’s AWS account, extracting data from various sources and delivering it to supported targets. The API does not need to be called from a different AWS account. Later in this article, we will provide an example of the authentication mechanism based on the assumption that the end-user-facing application operates on AWS.
To facilitate this approach, since Amazon AppFlow resides in the end-user’s AWS account, an AWS Identity and Access Management (IAM) role must be established. This role needs permissions to list, create, and run flows and connectors, along with cross-account access to the ISV’s AWS account so that the ISV can assume that role and manage Amazon AppFlow.
Creating and Running Amazon AppFlow Data Flows in the ISV’s AWS Account
In this scenario, the partner or ISV operates an application on AWS with a UI that seeks to support data integration with third-party SaaS providers for downstream processing. The ISV can use Amazon AppFlow APIs to integrate functionality within their product, enabling end-users to create flows to pull data directly from SaaS providers like Salesforce, Slack, Google Analytics, and more into targets supported by Amazon AppFlow for downstream processing.
For example, consider an ISV providing analytics capabilities on data stored in Amazon Redshift. If an end-user needs to bring data from Salesforce and ServiceNow into Amazon Redshift for analysis, but the ISV lacks the functionality to connect to these platforms, they can quickly implement this capability using the Amazon AppFlow API. This allows the ISV to offer a seamless experience that enables data flows to run in the ISV’s AWS account, pulling data from Salesforce and ServiceNow into the end-user’s Amazon Redshift cluster for downstream processing.
The following diagram illustrates this architecture.
This architecture depicts the ISV’s application in operation and how Amazon AppFlow data flows are executed from the ISV’s AWS account. The access needed to authenticate and save extracted data into customer-owned destinations must be granted to the ISV. This can be achieved by creating cross-account roles or similar mechanisms, except for Amazon S3, as Amazon AppFlow does not support cross-account access to S3 buckets to prevent unauthorized access.
Creating an Amazon AppFlow Data Flow Using an API
Now, let’s explore a few fundamental steps to create and execute an Amazon AppFlow data flow via API. We will also discuss how to implement these steps within your application’s UI and what it looks like when configuring the flow directly on the Amazon AppFlow console. We will focus on the first design discussed, where the ISV operates Amazon AppFlow data flows in the end-user’s AWS account.
Set Up a Cross-Account Role for Amazon AppFlow Access
Before we begin, if the ISV implementing this solution is utilizing an AWS account to run the application, it is advisable to set up cross-account roles to obtain access for creating and managing Amazon AppFlow data flows. For further details, see this excellent resource on onboarding tips from Amazon.
In this scenario, the ISV will assume the role using AWS Security Token Service (AWS STS) to create resources and manage the lifecycle of Amazon AppFlow. Below is a Python function to assume roles in another account after creation. This function requires only the role ARN to be passed; the role must have the necessary permissions to manage Amazon AppFlow.
def assumed_role_session(role_arn: str, base_session: botocore.session.Session = None):
base_session = base_session or botocore.session.Session()
# Additional code here
For further reading on related topics, you can check out this insightful blog post. It’s a good idea to stay informed on the latest trends and practices in this field.
By leveraging Amazon AppFlow, you can significantly simplify data integration for your applications, making it easier to connect with various SaaS providers and optimize your data workflows.
Leave a Reply