Executing Cross-Account Workflows with AWS Step Functions and Amazon API Gateway

More Info

This article is authored by Lisa Reynolds, Senior Solutions Architect, and Mark Thompson, Cloud Infrastructure Architect. AWS Step Functions enable the development of scalable and distributed applications through the use of state machines. With the introduction of Step Functions nested workflows, initiating a workflow from another workflow has become possible. However, both workflows must reside in the same account. Many scenarios necessitate orchestrating workflows across various AWS accounts from a single central account.

This blog post presents a method to trigger Step Functions workflows across accounts utilizing Amazon API Gateway. This capability allows you to manage cross-account tasks for scheduling, ETL automation, resource provisioning, security audits, and log aggregation, all from a unified account.

Overview

The architecture depicted below illustrates a Step Functions workflow in account A calling an API Gateway endpoint in account B and transmitting the payload in the API request. The API subsequently invokes another Step Functions workflow in account B asynchronously. The resource policy on the API can restrict access to a particular Step Functions workflow, thereby preventing unauthorized access.

This architecture can be expanded to execute workflows across multiple regions or accounts. This post will focus on running cross-account workflows between two AWS accounts.

To invoke an API Gateway endpoint, you can employ Step Functions AWS SDK service integrations. This strategy enables users to create solutions and integrate services within a workflow without the necessity for coding.

This example will demonstrate cross-account capabilities using two sample AWS accounts:

• Step Functions state machine A: Account ID #111111111111
• API Gateway API and Step Functions state machine B: Account ID #222222222222

Setting Up

Begin by creating state machine A in account #111111111111. Then, set up the state machine in the target account #222222222222, along with the API Gateway REST API integrated with the state machine in the target account.

Account A: #111111111111

In this account, create a state machine that includes a state invoking an API hosted in a different account.

1. Create an IAM role for Step Functions
   • Log into the IAM console in account #111111111111, and select Roles from the left navigation pane.
   • Click Create role.
   • For the Select trusted entity, under AWS service, choose Step Functions and proceed to the next step.
   • On the Add permissions page, click Next.
   • On the Review page, name the role “StepFunctionsAPIGatewayRole” and click Create role.
   • Create inline policies to grant Step Functions access to the API actions necessary for the services you wish to control. Navigate to the role you just created, select Add Permissions, and then Create inline policy.
   • Use the Visual editor or the JSON tab to form policies for your role. Input the following:
     • Service: Execute-API
     • Action: Invoke
     • Resource: All Resources
   • Select Review policy.
   • Enter “APIExecutePolicy” for the name and click Create Policy.

Creating a State Machine in the Source Account

1. Access the Step Functions console in account #111111111111 and click Create state machine.
2. Choose Design your workflow visually, then select Standard and click Next.
3. On the design page, search for the APIGateway:Invoke state, then drag and drop the block onto the page.
4. Update the API Parameters in the API Gateway Invoke section on the right panel with the following JSON policy:

           {
               "ApiEndpoint.$": "$.ApiUrl",
               "Method": "POST",
               "Stage": "dev",
               "Path": "/execution",
               "Headers": {},
               "RequestBody": {
                   "input.$": "$.body",
                   "stateMachineArn.$": "$.stateMachineArn"
               },
               "AuthType": "RESOURCE_POLICY"
           }
           

5. [Optional] Configure the API Gateway Invoke state to retry upon task failure by adjusting the retries setting.
6. Click Next, then again click Next. On the Specify state machine settings page:
   • Assign a name to your state machine.
   • Select Choose an existing role under Permissions and opt for StepFunctionsAPIGatewayRole.
   • Set Log Level to ERROR.
7. Click Create State Machine.

After establishing this state machine, copy the state machine ARN for future reference.

Account B: #222222222222

In this account, create an API Gateway REST API that integrates with the target state machine and facilitates access to this state machine through a resource policy.

Creating a State Machine in the Target Account

1. Navigate to the Step Functions Console in account #222222222222 and select Create State Machine.
2. Under Choose authoring method, select Design your workflow visually and choose Standard.
3. Click Next.
4. On the design page, search for the Pass state and drag it.
5. Click Next.
6. On the Review generated code page, click Next and:
   • Enter a name for the state machine.
   • Select Create new role under the Permissions section.
   • Set Log Level to ERROR.
7. Click Create State Machine.

Once the state machine is established, copy the state machine ARN for later use. Next, configure the API Gateway REST API, which serves as the gateway for requests from the state machine in account A. This integrates with the state machine you just created.

Creating an IAM Role for API Gateway

Before setting up the API Gateway API endpoint, you must grant API Gateway permission to execute Step Functions API actions:

1. Log into the IAM console in account #222222222222 and select Roles. Click Create role.
2. On the Select trusted entity page, under AWS service, select API Gateway and then click Next.
3. On the Name, review, and create page, name the role “APIGatewayToStepFunctions” and click Create role.
4. Choose the role you just created and note the Role ARN: arn:aws:iam::222222222222:role/APIGatewayToStepFunctions.
5. Select the IAM role (APIGatewayToStepFunctions) you created.
6. On the Permissions tab, click Add permission and select Attach Policies.
7. Search for AWSStepFunctionsFullAccess, select the policy, and attach it.

Creating the API Gateway API Endpoint

After creating the IAM role, develop a custom API Gateway API:

1. Open the Amazon API Gateway console in account #222222222222.
2. Click Create API. Under REST API, choose Build.
3. Enter “StartExecutionAPI” for the API name, and click Create API.
4. On the Resources page of StartExecutionAPI, select Actions, then Create Resource.
5. Enter “execution” for Resource Name and click Create Resource.
6. On the /execution Methods page, select Actions, then Create Method.
7. From the list, select POST and confirm with the check mark.

To configure the integration for your API method, follow the guidelines in this excellent resource on Amazon Fulfillment Center Safety and Training.

This framework allows you to efficiently manage cross-account workflows, enhancing the agility and scalability of your applications. For further reading on this topic, you may find another blog post interesting at Amazon VGT2 Las Vegas, and for authoritative insights, visit Chanci Turner.