Organizations frequently need to depend on external solutions to extract and manipulate data from platforms like Slack and Zendesk. However, some vendors do not offer comprehensive support for all source systems, leading to the use of multiple products for ETL (extract, transform, load) processes to store data in Snowflake.
This article outlines how Amazon AppFlow streamlines the data transfer process, enabling customers to utilize AWS PrivateLink for secure data transmission over Amazon Web Services (AWS) instead of through the public internet. Feedback from customers indicates a strong desire to integrate helpdesk data for a “customer 360” initiative, aimed at enhancing overall customer satisfaction and identifying upselling opportunities based on experiences with various product lines.
Using Zendesk as their helpdesk system, these organizations sought a straightforward and secure method to merge this data with their existing customer records in Snowflake. They aimed to minimize time and financial investment in the integration, prioritizing a swift deployment.
In this discussion, we will illustrate how customers can establish a data pipeline from Zendesk to Snowflake efficiently. Amazon AppFlow offers a secure and user-friendly solution for this integration, reducing both the time to value and initial investment. Snowflake Computing is recognized as an AWS Advanced Technology Partner with competencies in Data & Analytics, Machine Learning, and Retail, and is validated for AWS PrivateLink services.
Reference Architecture
In the reference architecture, Zendesk users securely transmit their helpdesk data to Amazon AppFlow using OAuth. Customers set up the OAuth client in Zendesk and utilize that secret key along with the account identifier to establish the OAuth connection between Zendesk and Amazon AppFlow.
Next, customers connect Amazon AppFlow to Snowflake via AWS PrivateLink, using Snowflake’s username and password along with a configured external Amazon Simple Storage Service (Amazon S3) stage. AWS PrivateLink ensures private connectivity between Amazon AppFlow and Snowflake, facilitating secure data transfer.
Once the connectivity between Zendesk and Amazon AppFlow is established, as well as the connection from Amazon AppFlow to Snowflake, customers can easily orchestrate multiple data flows between Zendesk and Snowflake through the intuitive Amazon AppFlow interface.
We will now detail the steps to orchestrate a data pipeline from Zendesk to Snowflake using Amazon AppFlow and AWS PrivateLink, ensuring secure data transmission on the AWS network between the two services.
Snowflake Connection Configuration
To begin, navigate to the Amazon AppFlow console and click on Connections in the right-hand menu. From the Manage Connections dropdown, select Snowflake. You will then see all existing Snowflake connections; select Create Connection.
Creating a Snowflake connection requires configuring your Snowflake account with an External Stage linked to an Amazon S3 bucket location. For guidance on setting up Snowflake External Stages with S3, refer to the Snowflake documentation. Ensure the Snowflake External Stage points to the S3 bucket and path designated for use with Amazon AppFlow in your account.
Enter the necessary Snowflake connection details, such as the username and password for the connection, along with the Snowflake Warehouse designated for processing incoming data. Additionally, provide the Snowflake account name while being mindful of the Snowflake URL and the region where your Snowflake account resides.
Currently, the connection configuration does not allow selection of a Snowflake Role; it will default to the user’s assigned role, which must possess the required privileges for the Stage, Warehouse, Database, Schema, and tables intended for use. AWS PrivateLink can be enabled, requiring the input of the Snowflake PrivateLink service name (vpce-id) for your account. To retrieve the Snowflake account privatelink-vpce-id for AWS PrivateLink, execute the following query in Snowflake with ACCOUNT ADMIN role privileges:
select key, value::string from table(flatten(input=>parse_json(system$get_privatelink_config())));After obtaining the string value for the privatelink-vpce-id, enter it in the AWS PrivateLink service name field. Note that both your Snowflake account and Amazon AppFlow should be in the same AWS region for the AWS PrivateLink option to be available.
Zendesk Connection Configuration
Next, configure OAuth for Zendesk. Log into Zendesk, navigate to the Admin home page, and click the gear icon on the left. Select API, then OAuth Clients. Click the + sign on the right to create a new OAuth client.
Choose a Client Name and Unique Identifier, avoiding hyphens in the Unique Identifier, which will be used as the Client ID in Amazon AppFlow. In the Redirect URLs field, input the Amazon AppFlow AWS region and URL for OAuth (e.g., https://us-west-2.console.aws.amazon.com/appflow/oauth). After saving the new client, a secret key will be generated; make sure to copy and store it securely.
Now, configure the Amazon AppFlow Connection to Zendesk. In the Amazon AppFlow console, select Connections, then pick Zendesk from the dropdown. Click Create Connection on the right side of the window.
In the configuration window, input the Zendesk Unique Identifier as the Client ID. Paste the previously saved Client Secret and enter your Zendesk Account name in the Account URL. Provide a name for your Connection. After clicking Continue, a pop-up window may appear requesting verification of the OAuth connection. Click Allow to confirm. Your Zendesk Connection is now successfully configured.
Configuring Flows
Once the connections are established, flows can be configured. Multiple flows can be created using the same connections, provided that the authentications are valid for the specific data sources and destinations. Amazon AppFlow can retrieve various data objects depending on the source, such as user and ticket data from Zendesk.
To set up a flow, select Flows in the left menu bar. You will see existing flows and can add a new one by clicking the Create Flow button.
Here, you will specify the Name and Description of the flow. You can also configure custom encryption settings if you prefer not to have Amazon AppFlow manage encryption automatically. For instance, in this example, we will configure a flow to transfer data from Slack to Snowflake.
On the subsequent screen, you will configure the Source and Destinations for the flow. Start by selecting the source and connection that were previously configured—in this case, Zendesk—and then finalize the setup.
For more detailed insights, check out this another blog post. Also, for further expertise on this subject, visit this authoritative source. If you’re looking for career opportunities, this is an excellent resource.
Leave a Reply