Today AWS Step Functions expands the number of supported AWS services from 17 to over 200 and AWS API Actions from 46 to over 9,000 with its new capability AWS SDK Service Integrations.
When developers build distributed architectures, one of the patterns they use is the workflow-based orchestration pattern. This pattern is helpful for workflow automation inside a service to perform distributed transactions. An example of a distributed transaction is all the tasks required to handle an order and keep track of the transaction status at all times.
Step Functions is a low-code visual workflow service used for workflow automation, to orchestrate services, and help you to apply this pattern. Developers use Step Functions with managed services such as Artificial Intelligence services, Amazon Simple Storage Service (Amazon S3), and Amazon DynamoDB.
Introducing Step Functions AWS SDK Service Integrations
Until today, when developers were building workflows that integrate with AWS services, they had to choose from the 46 supported services integrations that Step Functions provided. If the service integration was not available, they had to code the integration in an AWS Lambda function. This is not ideal as it added more complexity and costs to the application.
Now with Step Functions AWS SDK Service Integrations, developers can integrate their state machines directly to AWS service that has AWS SDK support.
You can create state machines that use AWS SDK Service Integrations with Amazon States Language (ASL), AWS Cloud Development Kit (AWS CDK), or visually using AWS Step Function Workflow Studio. To get started, create a new Task state. Then call AWS SDK services directly from the ASL in the resource field of a task state. To do this, use the following syntax.
arn:aws:states:::aws-sdk:serviceName:apiAction.[serviceIntegrationPattern]
Let me show you how to get started with a demo.
Demo
In this demo, you are building an application that, when given a video file stored in S3, transcribes it and translates from English to Spanish.
Let’s build this demo with Step Functions. The state machine, with the service integrations, integrates directly to S3, Amazon Transcribe, and Amazon Translate. The API for transcribing is asynchronous. To verify that the transcribing job is completed, you need a polling loop, which waits for it to be ready.
Create the state machine
To follow this demo along, you need to complete these prerequisites:
- An S3 bucket where you will put the original file that you want to process
- A video or audio file in English stored in that bucket
- An S3 bucket where you want the processing to happen
I will show you how to do this demo using the AWS Management Console. If you want to deploy this demo as infrastructure as code, deploy the AWS CloudFormation template for this project.
To get started with this demo, create a new standard state machine. Choose the option Write your workflow in code to build the state machine using ASL. Create a name for the state machine and create a new role.
Start a transcription job
To get started working on the state machine definition, you can Edit the state machine.
The following piece of ASL code is a state machine with two tasks that are using the new AWS SDK Service Integrations capability. The first task is copying the file from one S3 bucket to another, and the second task is starting the transcription job by directly calling Amazon Transcribe.
For using this new capability from Step Functions, the state type needs to be a Task. You need to specify the service name and API action using this syntax: “arn:aws:states:::aws-sdk:serviceName:apiAction.<serviceIntegrationPattern>”. Use camelCase for apiAction names in the Resource field, such as “copyObject”, and use PascalCase for parameter names in the Parameters field, such as “CopySource”.
For the parameters, find the name and required parameters in the AWS API documentation for this service and API action.
{
"Comment": "A State Machine that process a video file",
"StartAt": "GetSampleVideo",
"States": {
"GetSampleVideo": {
"Type": "Task",
"Resource": "arn:aws:states:::aws-sdk:s3:copyObject",
"Parameters": {
"Bucket.$": "$.S3BucketName",
"Key.$": "$.SampleDataInputKey",
"CopySource.$": "States.Format('{}/{}',$.SampleDataBucketName,$.SampleDataInputKey)"
},
"ResultPath": null,
"Next": "StartTranscriptionJob"
},
"StartTranscriptionJob": {
"Type": "Task",
"Resource": "arn:aws:states:::aws-sdk:transcribe:startTranscriptionJob",
"Parameters": {
"Media": {
"MediaFileUri.$": "States.Format('s3://{}/{}',$.S3BucketName,$.SampleDataInputKey)"
},
"TranscriptionJobName.$": "$$.Execution.Name",
"LanguageCode": "en-US",
"OutputBucketName.$": "$.S3BucketName",
"OutputKey": "transcribe.json"
},
"ResultPath": "$.transcription",
"End": true
}
}
}In the previous piece of code, you can see an interesting use case of the intrinsic functions that ASL provides. You can construct a string using different parameters. Using intrinsic functions in combination with AWS SDK Service Integrations allows you to manipulate data without the needing a Lambda function. For example, this line:
"MediaFileUri.$": "States.Format('s3://{}/{}',$.S3BucketName,$.SampleDataInputKey)"Give permissions to the state machine
If you start the execution of the state machine now, it will fail. This state machine doesn’t have permissions to access the S3 buckets or use Amazon Transcribe. Step Functions can’t autogenerate IAM policies for most AWS SDK Service Integrations, so you need to add those to the role manually.
Add those permissions to the IAM role that was created for this state machine. You can find a quick link to the role in the state machine details. Attach the “AmazonTranscribeFullAccess” and the “AmazonS3FullAccess” policies to the role.
Running the state machine for the first time
Now that the permissions are in place, you can run this state machine. This state machine takes as an input the S3 bucket name where the original video is uploaded, the name for the file and the name of the S3 bucket where you want to store this file and do all the processing.
For this to work, this file needs to be a video or audio file and it needs to be in English. When the transcription job is done, it saves the result in the bucket you specify in the input with the name transcribe.json.
{
"SampleDataBucketName": "<name of the bucket where the original file is>",
"SampleDataInputKey": "<name of the original file>",
"S3BucketName": "<name of the bucket where the processing will happen>"
}As StartTranscriptionJob is an asynchronous call, you won’t see the results right away. The state machine is only calling the API, and then it completes. You need to wait until the transcription job is ready and then see the results in the output bucket in the file transcribe.json.
Adding a polling loop
Because you want to translate the text using your transcriptions results, your state machine needs to wait for the transcription job to complete. For building an API poller in a state machine, you can use a Task, Wait, and Choice state.
- Task state gets the job status. In your case, it is calling the service Amazon Transcribe and the API getTranscriptionJob.
- Wait state waits for 20 seconds, as the transcription job’s length depends on the size of the input file.
- Choice state moves to the right step based on the result of the job status. If the job is completed, it moves to the next step in the machine, and if not, it keeps on waiting.
Wait state
The first of the states you are going to add is the Wait state. This is a simple state that waits for 20 seconds.
"Wait20Seconds": {
"Type": "Wait",
"Seconds": 20,
"Next": "CheckIfTranscriptionDone"
},Task state
The next state to add is the Task state, which calls the API getTranscriptionJob. For calling this API, you need to pass the transcription job name. This state returns the job status that is the input of the Choice state.
"CheckIfTranscriptionDone": {
"Type": "Task",
"Resource": "arn:aws:states:::aws-sdk:transcribe:getTranscriptionJob",
"Parameters": {
"TranscriptionJobName.$": "$.transcription.TranscriptionJob.TranscriptionJobName"
},
"ResultPath": "$.transcription",
"Next": "IsTranscriptionDone?"
},Choice state
The Choice state has one rule that checks if the transcription job status is completed. If that rule is true, then it goes to the next state. If not, it goes to the Wait state.
"IsTranscriptionDone?": {
"Type": "Choice",
"Choices": [
{
"Variable": "$.transcription.TranscriptionJob.TranscriptionJobStatus",
"StringEquals": "COMPLETED",
"Next": "GetTranscriptionText"
}
],
"Default": "Wait20Seconds"
},Getting the transcription text
In this step you are extracting only the transcription text from the output file returned by the transcription job. You need only the transcribed text, as the result file has a lot of metadata that makes the file too long and confusing to translate.
This is a step that you would generally do with a Lambda function. But you can do it directly from the state machine using ASL.
First you need to create a state using AWS SDK Service Integration that gets the result file from S3. Then use another ASL intrinsic function to convert the file text from a String to JSON.
In the next state you can process the file as a JSON object. This state is a Pass state, which cleans the output from the previous state to get only the transcribed text.
"GetTranscriptionText": {
"Type": "Task",
"Resource": "arn:aws:states:::aws-sdk:s3:getObject",
"Parameters": {
"Bucket.$": "$.S3BucketName",
"Key": "transcribe.json"
},
"ResultSelector": {
"filecontent.$": "States.StringToJson($.Body)"
},
"ResultPath": "$.transcription",
"Next": "PrepareTranscriptTest"
},
"PrepareTranscriptTest" : {
"Type": "Pass",
"Parameters": {
"transcript.$": "$.transcription.filecontent.results.transcripts[0].transcript"
},
"Next": "TranslateText"
},Translating the text
After preparing the transcribed text, you can translate it. For that you will use Amazon Translate API translateText directly from the state machine. This will be the last state for the state machine and it will return the translated text in the output of this state.
"TranslateText": {
"Type": "Task",
"Resource": "arn:aws:states:::aws-sdk:translate:translateText",
"Parameters": {
"SourceLanguageCode": "en",
"TargetLanguageCode": "es",
"Text.$": "$.transcript"
},
"ResultPath": "$.translate",
"End": true
}Add the permissions to the state machine to call the Translate API, by attaching the managed policy “TranslateReadOnly”.
Now with all these in place, you can run your state machine. When the state machine finishes running, you will see the translated text in the output of the last state.
Important things to know
Here are some things that will help you to use AWS SDK Service Integration:
- Call AWS SDK services directly from the ASL in the resource field of a task state. To do this, use the following syntax: arn:aws:states:::aws-sdk:serviceName:apiAction.[serviceIntegrationPattern]
- Use camelCase for apiAction names in the Resource field, such as “copyObject”, and use PascalCase for parameter names in the Parameters field, such as “CopySource”.
- Step Functions can’t autogenerate IAM policies for most AWS SDK Service Integrations, so you need to add those to the IAM role of the state machine manually.
- Take advantage of ASL intrinsic functions, as those allow you to manipulate the data and avoid using Lambda functions for simple transformations.
Get started today!
AWS SDK Service Integration is generally available in the following regions: US East (N. Virginia), US East (Ohio), US West (Oregon), Canada (Central), Europe (Ireland), Europe (Milan), Africa (Cape Town) and Asia Pacific (Tokyo). It will be generally available in all other commercial regions where Step Functions is available in the coming days.
Learn more about this new capability by reading its documentation.
— Marcia