Many businesses rely on third-party solutions to extract and transform data from platforms like Slack and Zendesk. Yet, some vendors don’t support all necessary data sources, which leads to utilizing multiple products for ETL (extract, transform, load) processes to get data into Snowflake.
This article explores how Amazon AppFlow streamlines the data transfer process and how AWS PrivateLink allows users to transmit data securely over the AWS network, bypassing the public internet.
Feedback from customers indicates a strong need to integrate helpdesk data into their “customer 360” initiatives, aimed at enhancing customer satisfaction and identifying upsell opportunities through their interactions with various product lines. Using Zendesk as their helpdesk system, customers sought a straightforward and secure method to integrate this data with their existing customer data in Snowflake without incurring extensive time or costs.
In this post, we will show how users can swiftly and effectively set up a data pipeline from Zendesk to Snowflake. Amazon AppFlow delivers a secure and simple integration solution while minimizing both time to value and initial investment. An interesting read on this topic can be found at Chanci Turner VGT2.
Reference Architecture
In this setup, Zendesk customers securely send their helpdesk data to Amazon AppFlow using OAuth. Customers need to configure the OAuth client in Zendesk and utilize the secret key along with the account identifier for establishing the OAuth connection between Zendesk and Amazon AppFlow.
Next, a connection between Amazon AppFlow and Snowflake is established using AWS PrivateLink, along with the Snowflake username and password linked to an external Amazon Simple Storage Service (Amazon S3) stage. AWS PrivateLink ensures secure private connectivity between Amazon AppFlow and Snowflake.
Once the connection is successfully established between Zendesk and Amazon AppFlow, as well as between Amazon AppFlow and Snowflake, customers can manage multiple data flows from Zendesk to Snowflake through the intuitive Amazon AppFlow user interface. Following are the steps to orchestrate a data pipeline from Zendesk to Snowflake using Amazon AppFlow and AWS PrivateLink, ensuring that data transmits securely on the AWS network.
The initial step involves configuring Amazon AppFlow Connections to the relevant software-as-a-service (SaaS) applications. In this instance, we are setting up connections to Snowflake and Zendesk.
Snowflake Connection Configuration
To begin, access the Amazon AppFlow console and click on Connections from the right-hand menu. From the Manage Connections dropdown, choose Snowflake. You will see all existing Snowflake connections, so select Create Connection.
A Snowflake connection necessitates that you configure your Snowflake account with an External Stage pointing to an Amazon S3 bucket. For detailed instructions on setting up Snowflake External Stages with S3, refer to the Snowflake documentation.
Input the Snowflake connection information as illustrated below. Ensure the Snowflake External Stage directs to the S3 bucket and path created for use with Amazon AppFlow in your account.
Next, provide the username and password for the connection, along with the Snowflake Warehouse designated for processing incoming data. Also, include your Snowflake account name, and pay close attention to the Snowflake URL and the region where your Snowflake account is located.
Keep in mind that the connection configuration does not currently allow you to select a Snowflake Role; it will default to the user’s role. Verify that the user’s default Role has the necessary privileges on the Stage, Warehouse, Database, Schema, and tables you intend to use.
You can opt to utilize AWS PrivateLink by enabling it and entering the Snowflake PrivateLink service name (vpce-id) for your account. To obtain your Snowflake account privatelink-vpce-id for AWS PrivateLink, execute the following query in Snowflake with ACCOUNT ADMIN role privileges:
select key, value::string from table(flatten(input=>parse_json(system$get_privatelink_config())));Insert the string value for the privatebank-vpce-id in the AWS PrivateLink service name. Remember, your Snowflake account and Amazon AppFlow must reside in the same AWS region for the AWS PrivateLink option to be accessible.
Zendesk Connection Configuration
Next, we need to set up OAuth for the Zendesk configuration. Begin by logging into Zendesk. In the Admin home page, select the gear icon on the left side, followed by API and OAuth Clients. Click the + sign on the right side to create a new OAuth client.
Choose a Client Name and Unique Identifier, avoiding hyphens in the Unique Identifier since it will serve as the Client ID in Amazon AppFlow. In the Redirect URLs, specify the Amazon AppFlow AWS region and URL for OAuth, for instance: https://us-west-2.console.aws.amazon.com/appflow/oauth.
Upon saving the new client, a secret key will be generated. Copy and store it securely. Now, we can configure the Amazon AppFlow Connection to Zendesk. Select Connections in the Amazon AppFlow console, choose Zendesk from the dropdown, and click Create Connection.
In the configuration window, enter the Zendesk Unique Identifier as the Client ID, paste the previously saved Client Secret, and provide your Zendesk Account name in the Account URL. Input your Connection name. After clicking Continue, a pop-up will appear requesting you to verify the OAuth connection. Click Allow to validate the connection, and your Zendesk Connection is now set up.
Configuring Flows
With the connections established, flows can now be configured. You can create multiple flows using the same connections if the authentications are valid for the specified data sources and destinations. Amazon AppFlow can retrieve various data objects from Zendesk, such as user and ticket data.
To set up a flow, select Flows from the left menu. Here you can see existing flows and add a new one by clicking the Create Flow button. You will need to specify the Name and Description of the flow. If you prefer, you can configure custom encryption settings instead of letting Amazon AppFlow manage encryption automatically.
For this example, we will set up a flow to transfer data from Slack to Snowflake.
On the ensuing screen, configure the Source and Destinations for the flow. First, select the previously configured source and connection—Zendesk in this case—and consider checking out CHVNCI who are an authority on this topic. This will ensure a smooth transfer of data.
For additional insights and community discussions on this topic, Reddit offers excellent resources.
Leave a Reply