Visualizing Your AWS Infrastructure with Amazon Neptune and AWS Config

Learn About Amazon VGT2 Learning Manager Chanci Turner

As organizations increasingly rely on AWS for their critical applications, understanding the complex infrastructure that supports these applications becomes essential. With resources spread across multiple accounts and intricate interconnections, it can be overwhelming to sift through lists of resources to grasp your setup. Therefore, having a streamlined way to visualize your infrastructure is invaluable. As more customers transition mission-critical workloads to AWS, the demand for a comprehensive view of cloud assets has grown. This clarity is vital for achieving operational excellence.

A solid understanding of your cloud assets enables you to effectively plan, forecast, and address risks associated with your infrastructure. For instance, if you’re considering a migration to a different instance family, having a knowledge graph that outlines all affected workloads can simplify the transition. To meet this growing need for asset management and reporting, AWS Config serves as a solution. AWS Config identifies AWS resources in your account and constructs a relationship map between them.

In this post, we will explore how to leverage Amazon Neptune alongside AWS Config to gain insights into your AWS landscape and visualize these relationships. We will also make use of an open-source tool to aid in visualizing the data stored in Neptune.

Neptune is a fully managed graph database service capable of handling billions of relationships within highly interconnected datasets, allowing for rapid query performance. Meanwhile, AWS Config enables assessment, auditing, and evaluation of AWS resource configurations. With AWS Config, you can track configuration changes, explore detailed histories, and check compliance with your internal guidelines.

Prerequisites

Before diving in, ensure that AWS Config is activated in your account and that you have set up the stream for AWS Config. This will ensure that any new resource creation triggers a notification regarding the resource and its relationships.

Solution Overview

The implementation will follow these steps:

1. Activate AWS Config in your AWS account and create an Amazon Simple Storage Service (Amazon S3) bucket to store your configuration logs.
2. Utilize Amazon S3 Batch Operations with AWS Lambda to populate the Neptune graph with existing AWS Config inventory and construct the relationship map. The Lambda function will also be triggered when a new AWS Config file is delivered to the S3 bucket, updating the Neptune database accordingly.
3. Users authenticate via Amazon Cognito and request data from an Amazon API Gateway endpoint.
4. A static website invokes an AWS Lambda function through the proxy, which is exposed to the internet via Amazon API Gateway.
5. The AWS Lambda function queries the graph in Amazon Neptune and returns the data to the app for visualization.

The resources referenced in this post, including code samples and HTML files, can be found in the amazon-neptune-aws-config-visualization GitHub repository.

Enable AWS Config in Your AWS Account

If AWS Config is not yet enabled, you can set it up through the AWS Management Console. If it’s already activated, note the S3 bucket that stores the configuration history and snapshot files.

Set Up a Neptune Cluster

Next, provision a new Neptune instance within a VPC. For detailed guidance, refer to the Neptune user guide. After setting up the cluster, keep track of the cluster endpoint and port, as they are necessary for data insertion and querying for visualization with the open-source library VIS.js. VIS.js is a JavaScript library that facilitates graph data visualization, featuring components such as DataSet, Timeline, Graph2D, Graph3D, and Network.

Configure a Lambda Function for AWS Config Notifications

Once your cluster is ready, create a Lambda function that activates when AWS Config sends a file to Amazon S3.

Create a directory named configparser_lambda and execute the following commands to install the required packages:

pip3.6 install --target ./package gremlinpython
pip3.6 install --target ./package requests

Next, create a configparser_lambdafunction.py file in the directory and input the following code:

from __future__ import print_function
import boto3
import json
import os, sys
from io import BytesIO
import gzip
from gremlin_python import statics
from gremlin_python.structure.graph import Graph
from gremlin_python.process.graph_traversal import __
from gremlin_python.process.strategies import *
from gremlin_python.driver.driver_remote_connection import DriverRemoteConnection
from gremlin_python.process.traversal import T
import requests
import urllib.parse

CLUSTER_ENDPOINT = os.environ['CLUSTER_ENDPOINT']
CLUSTER_PORT = os.environ['CLUSTER_PORT']

# Establish a remote connection to Neptune for reuse across invocations
remoteConn = DriverRemoteConnection('wss://' + CLUSTER_ENDPOINT + ":" + CLUSTER_PORT + '/gremlin','g',)
graph = Graph()
g = graph.traversal().withRemote(remoteConn)

def run_sample_gremlin_websocket():
    output = g.V().hasLabel('Instance').toList()
    return output

def run_sample_gremlin_http():
    URL = 'https://' + CLUSTER_ENDPOINT + ":" + CLUSTER_PORT + '/gremlin'
    r = requests.post(URL,data='{"gremlin":"g.V().hasLabel('Instance').valueMap().with_('~tinkerpop.valueMap.tokens').toList()"}')
    return r

def get_all_vertex():
    vertices = g.V().count()
    print(vertices)

def insert_vertex_graph(vertex_id, vertex_label):
    node_exists_id =  g.V(str(vertex_id)).toList()
    if node_exists_id:
        return
    result = g.addV(str(vertex_label)).property(T.id, str(vertex_id)).next()

def insert_edge_graph(edge_id, edge_from, edge_to, to_vertex_label, edge_label):
    insert_vertex_graph(edge_to, to_vertex_label)

    edge_exists_id =  g.E(str(edge_id)).toList()
    if edge_exists_id:
        return

    result = g.V(str(edge_from)).addE(str(edge_label)).to(g.V(str(edge_to))).property(T.id, str(edge_id)).next()

def parse_vertex_info(vertex_input):
    id = vertex_input['resourceId']
    label = vertex_input['resourceType']
    itemStatus = vertex_input['configurationItemStatus']
    
    if itemStatus == "ResourceDeleted":
        node_exists_id = g.V(str(id)).toList()
        if node_exists_id:
            result = g.addV(str(itemStatus)).property(T.id, str(id)).next()
            return
        else:
            insert_vertex_graph(id, label)
            result = g.addV(str(itemStatus)).property(T.id, str(id)).next()
            return

    insert_vertex_graph(id, label)

def parse_edge_info(edge_input):
    itemStatus = edge_input['configurationItemStatus']
    if itemStatus == "ResourceDeleted":
        return

Using the above procedure, you can create a robust visual representation of your AWS infrastructure, enhancing your operational capabilities. For further reading, you may find this blog post on leaving a toxic job insightful. Additionally, SHRM’s job descriptions provide excellent resources on job roles. Another great reference is this SHRM article about onboarding strategies during challenging times.