Amazon Onboarding with Learning Manager Chanci Turner

Learn About Amazon VGT2 Learning Manager Chanci Turner

In today’s fast-paced digital landscape, businesses are increasingly leveraging a variety of Software as a Service (SaaS) platforms to optimize their operations. This swift transition allows organizations to streamline processes, boost collaboration, and secure scalable solutions for managing essential data and workflows. However, many companies are beginning to identify the potential benefits of integrating and enhancing their SaaS data to refine internal operations and derive valuable insights. With AWS Glue, a serverless data integration service, organizations can simplify this integration process by merging data from both internal and external sources into a centralized AWS data lake. This facilitates meaningful analytics, enhances insights, and allows for the possibility of returning enriched data to external SaaS platforms.

This article presents the newly launched HubSpot managed connector for AWS Glue, illustrating how users can seamlessly incorporate HubSpot data into their existing AWS data lake. By consolidating HubSpot data alongside information from AWS accounts and other SaaS services, businesses can enrich, analyze, and optionally export the data back to HubSpot, creating a cohesive and integrated data experience.

Solution Overview

In the following example, AWS Glue is utilized to extract, transform, and load (ETL) data from a HubSpot account into a transactional data lake hosted on Amazon Simple Storage Service (Amazon S3) utilizing Apache Iceberg format. We will register the schema in the AWS Glue Data Catalog to ensure data discoverability. Afterward, Amazon Athena will be employed to confirm the successful loading of HubSpot data into Amazon S3. The architecture of this solution is depicted in the diagram below.

Key components and steps for integrating this solution include:

1. Configuring your HubSpot account and app for data access.
2. Preparing for data migration by securely storing your HubSpot OAuth credentials in AWS Secrets Manager, establishing an S3 bucket for your ingested data, and creating an AWS Identity and Access Management (IAM) role for AWS Glue.
3. Developing an AWS Glue job to extract data from HubSpot and load it into Amazon S3. AWS Glue establishes a secure connection to HubSpot using OAuth for authorization and TLS for data encryption during transit. Additionally, AWS Glue can implement complex data transformations, ensuring efficient data integration tailored to your specifications.
4. Schema and metadata will be registered in the AWS Glue Data Catalog, serving as a centralized repository for all data assets. This simplifies schema management and enhances data discoverability across other services.
5. Executing the AWS Glue job to extract data from HubSpot and save it to Amazon S3 in Iceberg format. Apache Iceberg is an open-source, high-performance table format designed for extensive analytics, guaranteeing transactional consistency and effortless schema evolution. While Iceberg is utilized in this instance, AWS Glue supports various data formats, including other transactional formats like Apache Hudi and Delta Lake.
6. The data stored in Amazon S3 will be organized into partitioned folders to optimize query performance and management. Amazon S3 will also retain the AWS Glue scripts, logs, and any temporary data necessary during the ETL procedure.
7. Lastly, Amazon Athena will be employed to query the HubSpot data loaded into Amazon S3, validating that all alterations in the source system have been accurately captured.

For ongoing analysis, HubSpot can regularly synchronize data to Amazon S3, allowing for continuous evaluations of data updates over time.

Setting Up Your HubSpot Account

To follow this example, you need to create a HubSpot public app for AWS Glue within a HubSpot Developer account and connect it to a corresponding HubSpot account. A HubSpot public app is a type of integration that can be installed in your HubSpot accounts or listed in the HubSpot Marketplace. In this scenario, you will create a HubSpot app for the AWS Glue integration and install it in a new test account. Although referred to as a public app, it will not be available in their Marketplace and will have access solely to your test account.

1. If you don’t have one yet, sign up for a free HubSpot developer account.
2. Log into your HubSpot developer account, where options to create apps and test accounts will be visible.
3. Select “Create a test account” and follow the provided instructions.

HubSpot test accounts offer Enterprise versions of the HubSpot Marketing, Sales, and Service Hubs, along with sample data for testing most HubSpot tools. You can create and access CRM data through APIs with Glue. For additional information on creating a test account, refer to the guide on creating a developer test account.

Creating a HubSpot App

Complete the following steps to create a HubSpot app:

1. Return to your HubSpot developer account and select “Create an app.”
2. Fill in the App Info section with the name “AWS Glue” and provide a brief description.
3. Choose the Auth tab.
4. For Redirect URLs, input the redirect URL for AWS Glue in the format: https://<region>.console.aws.amazon.com/gluestudio/oauth.

Ensure to replace <region> with your actual AWS Glue operating region. For instance, for the US East (N. Virginia) Region, the AWS Glue redirect URL would be https://us-east-1.console.aws.amazon.com/gluestudio/oauth.

5. In the Scopes section, click “Add new scope” and select the following permissions:
   • automation
   • content
   • crm.lists.read
   • crm.lists.write
   • crm.objects.companies.read
   • crm.objects.companies.write
   • crm.objects.contacts.read
   • crm.objects.contacts.write
   • crm.objects.custom.read
   • crm.objects.custom.write
   • crm.objects.deals.read
   • crm.objects.deals.write
   • crm.objects.owners.read
   • crm.schemas.custom.read
   • e-commerce
   • forms
   • oauth
   • sales-email-read
   • tickets
6. Review the Scopes and Redirect URL settings, then select “Create app.”
7. Navigate back to your app’s Auth tab.
8. Take note of the values for Client ID, Client secret, and Install URL (OAuth) as you will need these to connect your AWS Glue instance.

Selecting or Creating an Amazon S3 Bucket

Choose an existing Amazon S3 bucket in your account or create a new one to store your HubSpot data, scripts, logs, etc. For this example, the bucket name will follow the format aws-glue-hubspot-<account>-<region>, where <account> refers to your AWS account number and <region> is your operating region. The account will be configured with defaults: public access disabled, versioning disabled, and server-side encryption using Amazon S3 managed keys (SSE-S3).

If you use AWSGlueServiceRole in your IAM role as demonstrated in this example, it will grant access to S3 buckets named starting with aws-glue-.

Creating an IAM Role for AWS Glue

Create an IAM role with permissions necessary for the AWS Glue job. AWS Glue will assume this role while invoking other services on your behalf.

1. In the IAM console, select “Roles” in the navigation panel.
2. Click “Create role.”
3. For Trusted entity type, choose “AWS service.”
4. For Use case, select “Glue.”
5. Add the following AWS managed policies to the role:
   • AWSGlueServiceRole for access to related services like Amazon S3, Amazon Elastic Compute Cloud, Amazon CloudWatch, and IAM. This policy enables access to S3 buckets prefixed with aws-glue-.
   • SecretsManagerReadWritable for necessary permissions.

For additional resources on workplace dynamics, consider reading this insightful blog post on toxic work environments here. Furthermore, if you’re exploring health plan options, SHRM offers authoritative insights on level-funded health plans as a potential pathway to self-funding. If you’re seeking career opportunities in training and development, check out this excellent resource.