Dealing with On-Call Anxiety at a Startup

25 December 2021 · teams

Being on-call 24/7 for 18 months straight was one of the most anxiety-inducing responsibilities that I've ever had. I had to make sure that I always had my phone within reach, my laptop accessible, and my mind ready to fix any critical system failure. I sometimes jolted up in the middle of the night, believing there was a page on my phone when there wasn't. It took months to realize I had a problem, and only because my partner accused me of being irritable lately and it dawned on me that she was right.

How I ended up on-call for 18 months

As a serial technical co-founder, I often find myself taking on-call responsibility for months at a time, simply because there's no one else available. What was different this time is that we had been around for 4 years already when this stretch started. It happened because our only other senior backend developer left us in mid-2020.

People come and go. It was my fault for letting myself be stuck in this situation. We have other developers on the team, but they've been working on more interesting and valuable parts of our product. I let myself be left behind as the only one responsible for our legacy, but still mission-critical, components.

Motiva A.I. orchestrates digital marketing campaigns for our enterprise clients. Imagine a scenario where you sign up for a newsletter at Verizon for Business (one of our customers) and expect to receive an email from them. Well, not if Motiva is down. On-call ensures that we can immediately respond to any service interruption.

"I can't wait to spend my weekend waiting for a system outage!", said nobody.

If you want to work at a small startup, on-call is part of the deal. Something mission-critical will eventually fail, and somebody needs to fix it. In a team of ten, that somebody is you more often than you'd like. I'd rather we have a clear on-call schedule with clear expectations than scramble to fight the fire. So this post is about how to survive it.

Keep in mind that on-call is a last line of defense to keep things running. If high availability is a business requirement, then that consideration should be an integral part of the product development process right from the start. More on that later.

What the anxiety actually felt like

The weird thing about my anxiety is that I found myself less anxious when there were more alerts happening regularly. One of the worst times was when I was on a family vacation for the first time in a long while, and there hadn't been any alert for weeks prior. The longer it stayed quiet, the more I felt a page coming.

According to the American Psychological Association, anxiety differs from stress in that anxiety is a persistent worry that won't go away, whereas stress is typically caused by an external trigger. That maps exactly to what I felt. I was anxious about the prospect of our system going down. Once it was actually down, I was stressed, debugging the failed system while our customers were pounding on our support team. In fact, I gave a talk about the mental stress of debugging production applications.

Surviving the stretch

If you're feeling anxious about your on-call duty, my first advice is to seek a workplace therapist. Second to that, I recommend working through the Anxiety & Worry Workbook by Clark and Beck.

What ultimately helped me mentally is accepting that the worst that can happen if I miss an alarm or two isn't so bad after all. We might lose a customer or two, which is pretty bad for an enterprise startup with only a handful of customers. But what's worse is getting burnt out and not being able to function anymore. Growing a startup is a marathon, and I shouldn't lose the race over any single bump. Having said that, stepping back is easier said than done. That's why the therapist and the workbook come first — it took me months of actively working on easing my mind before finding my peace.

While I worked on my mental health, I made some small adjustments along the way:

changed my pager tone to something more relaxing
took it easy when I got paged — resolved it when I could, and did not fret about jumping onto my laptop immediately
blocked out certain hours in our on-call scheduling system during expected slower weeknights. Getting woken up in the middle of the night was my biggest anxiety, and this gave me peace of mind.
this wasn't an option for me, but I wish I could have opted out of on-call duties for a couple of months when I was going crazy

I also changed how I handled the incidents themselves. During an incident, don't try to do too much. Focus on getting the system back up, or at least the mission-critical parts, even if they'll be limping. Leave the fixing for the team during work hours. A couple of times I would just pause our affected workers before spending any more time debugging. It's easier to explain to our customers that the system stopped working than to explain that it did the wrong thing. This is a debugging tip, but it helped with my anxiety because it substantially lowered the bar for what needed to be done at 3am — from fixing the system to pausing it gracefully.

And it would suck to keep getting paged for the same issue. We follow every incident with a retrospective: a one-page incident report covering who it affected, why it happened, and suggestions for future prevention, with actual tickets scheduled to be worked on. This gives the product team visibility into delivery expectations.

The breaking point

None of that fixed the underlying problem: I was the only person who could answer the pager, for systems I didn't trust. The lasting remedy was building confidence in our systems so that I'd know we'd be fine even if I were unavailable.

But when? Between scaling our system to handle a 400% increase in load this year, hiring a replacement, building new features, reviewing code, mentoring... how do you find time to address technical debt that's been known for years? We didn't. It took me getting to the point where I couldn't take it anymore, and I had to convince the rest of the company that we had to stop everything. For two quarters in 2021, we directed all engineering resources to rebuilding the most vulnerable parts of our system as high-availability workflows.

Developers that feel the pain caused by the system they built will build better systems.

I adhere to that pain-driven development mentality, and this was the most painful example of it in my career.

The rules I run the team by now

Having lived through this, I wouldn't put it on anyone else on my team. Here is what we do differently:

The more mission-critical a component is, the more eyes should be on it. Bus factor of one on a mission-critical system is how I got into this mess. We now categorize our application workflows as i) mission critical, ii) essential, or iii) everything else. Only a couple of mission-critical workflows are allowed to raise on-call alarms outside of work hours. This reduces support fatigue and focuses our attention on what matters.
Nobody is on-call alone. We have a clear escalation procedure so that whoever is on-call feels supported, and I make sure the team knows I'm always available at the top of that ladder. That's the unfortunate responsibility of being the tech lead at a small startup. Luckily, my team only had to call me once outside of my own on-call schedule this year.
Push work down the escalation ladder, not just up. In a team of ten, there are only so many people who can play musical chairs on the rotation. So we built semi-automated diagnostic tools for the non-developers on our team. The more issues our customer support team can resolve, the fewer tickets get escalated to our devs.
Spread the team across time zones. Motiva has been remote-first from the start, and one thing we did well early on is hiring across time zones. My previous senior engineer was on the other side of the world from me (not deliberately to such an extreme; suitable partners are just hard to find). That worked wonders for our on-call and support coverage, though it meant relying heavily on asynchronous collaboration for actual development work.
Set clear expectations for new engineers. Anyone new to our on-call rotation knows they are not expected to fix the world, that it's ok to take it easy, and that they have our support. If anything, err on the side of escalating early for the first few incidents, so that we can pair on them together.