Amazon selling partner API (SPAPI) — How to implement reports generation workflow with AWS Step Function (Part 1)

Today I will share with you an interesting topic that I had the possibility to implement several months ago on Scrappie. Our main target for today will be implementing a reports generation workflow that will automatically generate/check status/parse data from Amazon reports and save information to a database.

If you just begin your journey with AWS SPAPI I recommend you to check the article on how to get access to spapi and how to make your first API request to SPAPI

Requirements:

1. Have an active AWS account
2. Have an active developer application on Amazon Seller Account
3. Have at least minimal knowledge of AWS Step Function
4. Know how to sign and make calls to spapi correctly
5. Basic understanding of NodeJS

Let’s get started!

I assume that you already know how to make API requests to SPAPI, know how to generate a refresh token and access token, if you don’t know how to generate credentials, please check my article where I show how to do this correctly (API request to SPAPI)

Most up-to-date information about Amazon SPAPI for Reports you can find here on official documentation.

Amazon Reports API

In this article, I will implement the possibility to generate GET_MERCHANT_LISTINGS_ALL_DATA report, which is a kind of report that gives us tab-delimited flat file detailed all listings report.

More types of reports, you can find here

AWS Step Functions

Before we start writing any piece of code, we need to understand what AWS Step Function is.

AWS Step Functions is a low-code visual workflow service used to orchestrate AWS services, automate business processes, and build serverless applications. Workflows manage failures, retries, parallelization, service integrations, and observability so developers can focus on higher-value business logic.

In our case, the Step Functions will provide the possibility to create an automated reports generation workflow with the possibility to retry operation in case of server failure and finally to store data into a database. Everything will be automatic without any kind of user involvement.

If you want to know more, here is the link to the official documentation.

Let's create our Step Function

I assume that you already registered on aws.com.

You have several ways to create Step Functions, everything depends on your knowledge and preferences. I prefer to use a Serverless Framework, but you can easily use an AWS console, Terraform, or CloudFormation.

In this article, I will create a Step Function and workflow using a serverless framework.

Workflow modeling

Before coding workflow we need to understand the flow, what processes we want to automate, what outcome we expect?

workflow in step function designer

Each lambda step will handle requests to amazon spapi, a “choice” step for checking report status, and a “wait” step to decrease the number of requests for report status.

So let's go through each of the steps.

1. Create report — will call lambda function that will create a report with type GET_MERCHANT_LISTINGS_ALL_DATA, and pass report metadata into the next step.
2. Check report status — will call lambda that will retrieve the report status, if report status equals “IN_QUEUE” or “IN_PROGRESS” then we will move to the “Wait for the report” state. If the status equals “DONE” then we move to the next step.
3. Insert report data — this step is invoked when report status equals “DONE”. This step will invoke a lambda function that will retrieve a report document, parse a document from CSV to JSON, and insert data into your database.

Extra settings of tasks.

“Create report” and “Check Report Status” tasks have error handling settings. “TimeOut” and “TaskFailed” exceptions will forward to the “Exit” task. Also, those tasks have a ttl equal to 60 seconds.

Now we are ready to write some code.

Workflow implementation

First of all, I will share with you the JSON definition and add a link to the YAML definition at the end of this section.

Let's go through step function definition

2 line — Starter point definition and name

3–103 line — States list

5–35–55–76–81–101 line — tasks type. All list of types you can find here

6 line — Resource definition, in our case you should put here your lambda ARN that will handle task processing

13–31 line — Task settings. Retry configuration and exception configuration

32— Task ttl configuration

55 — Choice task

56–72–Report status processing, each status represents different behavior that will be run in case of equality truthy

75–79 — Task will wait 2 seconds before step into “Check report status”

80–99 — Data processing “Task” that downloads report converts it from CSV to JSON, and store data into database

Click here to open the step function definition file.

Lambda functions

In this part we will create simple lambda functions that will not make any calls to amazon API, our goal is to create skeleton architecture that will be improved in the next parts of this guide.

So our lambda functions will simulate success behavior when we trigger our state machine.

create-report lambda

Then report-status lambda goes

And finally process-report

You can deploy those lambda functions using any way you prefer, I will use Serverless Framework.

Here is the serverless.yaml that deploys my stack.

“functions” section define CreateReport, ReportStatus, ProcessReport lambda.

The “stepFunctions” section defines step machines, the definition is referencing an external file that you can find below

So what you should expect after deploying serverless yaml?

1. 3 lambda functions
2. 1 Step Function
3. Working state machine (I mean when you trigger it manually it should be successfully completed)

Source code of serverless deployment you can find here

What next?

In the next part, we will add Amazon Seller partner API support

What is the use case?

To implement automation and remove any sort of manual work, I use state machines in Scrappie to automate report generation when users connect amazon integration and also when I need to set up scheduled report processing.

Scrappie?

A month ago I finally release the first version of my product, check it on producthunt. Everything that is related to Amazon Selling Partner Api, data processing, and monitoring of e-commerce data that I will post here will be connected somehow with Scrappie. So don’t hesitate and ask questions in the comments.

So what Scrappie is and what is its main purpose?

Scrappie is a platform that enables tracking product information from e-commerce websites. You don’t need to know any programming or markup language and there is no need to run servers as everything is handled on our side. We only need a link to the product to monitor or get information about. We collect historical data about every product so that you can use the information to analyze for example when and why there was a big spike in the price in a specific period of time.

We can also notify you about product changes using webhooks. If you sell on Amazon, you can easily connect your inventory to suppliers (aka monitored products) and set how your inventory should react to suppliers’ changes to products.

Thank you for reading, looking for your feedback/comments.