Simplifying Step Functions and Stepwise: Lessons Learned and a New Approach

At Motiva, we use AWS Step Functions to manage workflows for email delivery, data integration, and machine learning decisions. To make this simpler, we developed Stepwise, an open-source Clojure library that helps us coordinate the workers for these state machines. By using Step Functions, we can handle complex event-driven processes and monitor our operations.

After using Step Functions and Stepwise in production for a couple years, we found that creating a new state machine still requires us to work in three places:

1. We define the state machine in EDN using Amazon States Language, including :start-at, :next, and :result-path. I still need to refer to the ASL specification while writing it.
2. We register the workers that do the actual work in another file. Each worker is matched to its state machine definition by keyword.
3. In a separate infrastructure repository, we use Terraform to configure the EventBridge cron, IAM roles, and CloudWatch alarms. These resources refer to the state machine by name inside an ARN string.

We don't have anything that checks whether all three match.

How much does this slow us down? One of our state machines went from scaffolding to working in two days. That didn't mean we were done. We fixed a :result-path on the same day we added a step, added another step the week after, added a retry policy seven weeks later, and registered its alarms in the infrastructure repository along the way.

The bigger problem is that we sometimes avoided Step Functions altogether. The same service runs fifteen SQS queue listeners alongside its one state machine. Most of them manage workflows: heartbeat crons, a deadletter handler, code that marks a batch as skipped so a later heartbeat can reschedule it, and a health check for batches stuck in processing. Step Functions can already track an execution that stalls. We wrote these ourselves anyway because adding a queue listener was easier than adding a state machine.

We only have four developers at Motiva. As customers ask for more, how fast we can turn a new business workflow into a state machine sets how fast we can ship.

To improve this, we want to go Clojure all the way instead of splitting the work across Amazon States Language and Terraform. What do we mean by Clojure all the way? We have been designing a new interface to define Step Functions state machines without needing to write ASL or Terraform. Let's illustrate this with a pizza-making workflow:

(def pizza-making-state-machine
  (sfn/->> request
           (sfn/parallel (make-dough)
                         (make-sauce)
                         (sfn/map {:iterate-over :ingredients} prepare-ingredients))
           (put-ingredients-on-dough)
           (bake)
           (sfn/wait 2 :minutes)
           (sfn/choice (comp not is-pizza-acceptable?)
                       ;; branch off to this fn if condition is true
                       (sfn/fail))
           (serve)))

(sfn/ensure-state-machine client :pauls-pizza-making-machine pizza-making-state-machine)

This looks like normal Clojure code, but it's actually a workflow. This is because AWS Step Functions keeps the state of your workflow at all times. So, if your server, processes, or any of your workers go down during any workflow execution, your execution will continue where it left off once your system comes back online.

We can define the workers for each step of the workflow in Clojure as follows:

(defn make-dough
  ;; SFN error handling configuration as metadata
  {:retry [{:error-equals     [:sfn.timeout]
            :interval-seconds 60
            :max-attempts     3}]}
  [coll]
  coll)

(defn put-ingredients-on-dough [coll] coll)

(defn bake [coll] coll)

(defn make-sauce [coll] coll)

(sfn/def-choice-predicate-fn is-pizza-acceptable? [m]
  true)

These functions define the steps in the pizza-making workflow. We can choose where to run these workers at runtime using the following code:

(sfn/run-here client
              {make-dough               {:concurrency 2}
               put-ingredients-on-dough {:concurrency 1}
               bake                     {:concurrency 1}})


(sfn/run-on-lambda client
                   {make-sauce {:timeout         40
                                :memory-size     512
                                :max-concurrency 50}})

Since the workers are defined as Clojure functions, we can choose to run them in containers or serverless functions at runtime.

Say, what if make-dough turns out to be an infrequent but bursty process that would be better suited for a serverless function? But make-sauce takes too long and Lambda times out. We can switch the two like so:

(sfn/run-here client
              {make-sauce               {:concurrency 5}
               put-ingredients-on-dough {:concurrency 1}
               bake                     {:concurrency 1}})


(sfn/run-on-lambda client
                   {make-dough {:timeout         30
                                :memory-size     512
                                :max-concurrency 100}})

We can't switch between the two this easily with the current version of Stepwise. Our workers can only run in the service's container, and concurrency is still hardcoded next to a TODO to make it configurable. Moving one step to Lambda would require a new deployment artifact and more Terraform.

We're still in the design phase of this new library. If this interests you, please get in touch with me.