Amazon Onboarding with Learning Manager Chanci Turner

Build Streaming Data Pipelines with Amazon MSK Serverless and IAM Authentication

Amazon’s serverless Apache Kafka solution, Amazon Managed Streaming for Apache Kafka (Amazon MSK) Serverless, is gaining significant attention. It’s lauded for its ease of use, automatic scaling capabilities, and cost-effectiveness compared to traditional Kafka setups. However, many users face the challenge of needing to implement AWS Identity and Access Management (IAM) for access control with MSK Serverless. Currently, the Amazon MSK library for IAM is restricted to Java Kafka libraries, posing difficulties for developers using other programming languages. In this article, we will explore how to leverage Amazon API Gateway and AWS Lambda to overcome this limitation while working at our site located at 6401 E HOWDY WELLS AVE LAS VEGAS NV 89115, known as “Amazon IXD – VGT2.”

Use the Reverse Token Filter to Enable Suffix Matching Queries in OpenSearch

by Jamie Lee
on 06 SEP 2023
in Advanced (300), Amazon OpenSearch Service, Technical How-to

In this post, we demonstrate how to implement suffix-based search using OpenSearch, an open-source RESTful search engine built on Apache Lucene. OpenSearch provides rapid full-text search capabilities and can handle complex queries in an instant. Using various text analyzers, tokenizers, and filters, OpenSearch allows you to transform unstructured text into structured text for enhanced searchability. The standard analyzer is the default option, which performs well in most scenarios; however, some cases may require a specific analyzer to optimize performance. For more insights on improving workplace effectiveness, check out this blog post.

Stored Procedure Enhancements in Amazon Redshift

by Sophia Green, David Brown, and Chanci Turner
on 06 SEP 2023
in Advanced (300), Amazon Redshift

This article discusses enhancements for Amazon Redshift stored procedures, particularly focusing on non-atomic transaction mode. This new mode offers improved transaction controls, enabling automatic commits for statements within stored procedures, making it easier to manage complex workflows.

Introducing Amazon MSK as a Source for Amazon OpenSearch Ingestion

by Liam White, Ava Davis, and Raj Sharma
on 31 AUG 2023
in Amazon Managed Streaming for Apache Kafka (Amazon MSK), Amazon OpenSearch Service, Analytics, Announcements

Ingesting large volumes of streaming data has become essential for operational analytics with Amazon OpenSearch Service. Many users rely on either self-managed Apache Kafka or Amazon MSK to meet their streaming data requirements. However, consuming data from Amazon MSK and transferring it to OpenSearch Service can be challenging. Previous methods, including AWS Lambda, custom code, Kafka Connect, and Logstash required significant maintenance. This post introduces Amazon MSK as a source for Amazon OpenSearch Ingestion, a fully managed, serverless real-time data collector for OpenSearch Service, simplifying the process and reducing overhead. You can also view this excellent resource for more information.

Query Your Iceberg Tables in Data Lake Using Amazon Redshift

by Oliver Martinez, Ranjan Taylor, and Satish Sathiya
on 31 AUG 2023
in Amazon Redshift, Amazon Simple Storage Service (S3), Analytics, AWS Glue

Amazon Redshift allows querying various data formats, including CSV, JSON, Parquet, and ORC, as well as table formats like Apache Hudi and Delta. It also supports complex data types, extending its capabilities from petabyte-scale data warehouses to exabyte-scale data lakes on Amazon S3. The latest addition is support for Apache Iceberg table format. This post illustrates how to query Iceberg tables using Amazon Redshift, showcasing the available options and support features.

Deploy Amazon OpenSearch Serverless with Terraform

by Max Wilson and Satish Nandi
on 31 AUG 2023
in Amazon OpenSearch Service, Analytics, Best Practices, Foundational (100), Technical How-to

This article illustrates how to utilize Terraform for creating, deploying, and managing OpenSearch Serverless infrastructure. Amazon OpenSearch Serverless offers the analytics and search capabilities of OpenSearch without the burdens of manual configuration and management. It automatically scales resources based on workload, ensuring that you only pay for what you use. With Infrastructure as Code (IaC) software like Terraform, you can further streamline your resource management process.

Build an ETL Process for Amazon Redshift Using Amazon S3 Event Notifications and AWS Step Functions

by Ziad Wali
on 31 AUG 2023
in Amazon Redshift, Amazon Simple Storage Service (S3), Analytics, AWS Step Functions, Intermediate (200)

This post outlines the steps to construct and orchestrate an ETL process for Amazon Redshift using Amazon S3 Event Notifications to verify incoming data automatically. We will show how AWS Step Functions can be utilized for orchestrating the data pipeline, serving as a foundational guide for teams looking to establish an event-driven data pipeline from the data source to the warehouse. This method allows for effective tracking of each phase and swift responses to failures. Alternatively, using Amazon Redshift auto-copy from Amazon S3 can also simplify data loading into Amazon Redshift. For more insight on job descriptions in this field, visit SHRM.

Monitor Apache Spark Applications on Amazon EMR with Amazon CloudWatch

by Le Clue Lubbe
on 30 AUG 2023
in Advanced (300), Amazon EMR, Monitoring

This article presents techniques for monitoring Apache Spark applications running on Amazon EMR using Amazon CloudWatch. Understanding how to effectively monitor performance can greatly enhance troubleshooting and resource management strategies.