Deciding to Roll Back or Fix Forward During an Incident

crisis-leadershipMid10–15 min
How to Use This: Run this roleplay with 2–4 people, or try it solo by taking on each role. Add curveballs to test adaptability. Reflect afterward to see what worked well and what didn’t.

Roleplay Scenario

Scenario Overview

You've just released a major update, and now there's a bug causing trouble with a core feature. The team needs to decide quickly: should you roll back the release or push a fix forward? This scenario tests your ability to lead under pressure, making decisions that balance technical feasibility and business impact.

Roles & Setup

Role A – Incident Lead (You)
You're guiding the conversation to help the team make a well-informed decision.
Your goal: Facilitate a thorough discussion and drive towards a consensus, ensuring everyone is on the same page.

Role B – On-Call Engineer
You're knee-deep in logs, assessing the situation for a rollback or a fix.
Your goal: Provide technical insights to guide the decision, considering all potential risks and timelines.

Role C – Product Manager
You're worried about user experience and keeping stakeholders informed.
Your goal: Balance user impact with release goals and keep everyone updated.

Role D – Senior Engineer (Optional)
You're here to lend your expertise and help keep the team focused.
Your goal: Offer strategic insights and ensure technical details aren't overlooked.

Suggested Openers

Incident Lead:

  • “Thanks for jumping on this quickly. We need to decide if we should roll back or fix forward. Let’s weigh our options and see what makes sense.”
  • “This is a big deal — I want everyone’s thoughts before we make the call.”

On-Call Engineer:

  • “I’m seeing a major issue with [Feature X]. Rolling back is doable, but fixing forward might be faster if we can nail it down.”
  • “Here’s what I’ve found: [describe issue]. Need your thoughts on whether a fix is feasible in the short term.”

Product Manager:

  • “This is affecting a lot of users. What happens if we don’t roll back right away?”
  • “I need to update stakeholders. How confident are we in fixing this on the fly?”

Senior Engineer:

  • “We’ve been here before. Let’s think about what’s best long-term, not just a quick fix.”
  • “I’m here to help. Let’s make sure we consider all angles.”

Sample Roleplay in Action

Incident Lead:
“Thanks for coming together so fast. We’ve got a major issue with [Feature X]. Do we roll back or fix it forward? Let’s break down the pros and cons.”

On-Call Engineer:
“The bug is causing [describe issue] and hitting a lot of users. Rolling back is straightforward but undoes other features like [list key features]. A fix might be ready in a couple of hours if we all pitch in.”

Product Manager:
“Reverting means delaying those features, which isn’t great for our release schedule. How sure are we that a fix can be pushed that quickly?”

Senior Engineer:
“A rollback is safer for stability, but we need to check for any changes that might complicate it. What’s our contingency plan if the fix hits a snag?”

Incident Lead:
“Alright, let’s weigh fixing forward against rolling back. What’s the best way to minimize impact on users and stakeholders while keeping the team on track?”

On-Call Engineer:
“With a bit more testing support, I’m confident we can fix forward and keep disruption low.”

Product Manager:
“If we’re transparent with stakeholders about our fix-forward plan, we might get their buy-in. I’ll draft a communication plan.”

Senior Engineer:
“Let’s document our decision process and prepare for both outcomes. I’ll help with testing if we go with the fix.”

Incident Lead:
“Sounds like we’re leaning towards fixing forward, with rollback as our backup. I’ll coordinate efforts to keep us moving.”

Post-Scenario Tools

Curveball Mode (Optional)

Throw one of these into the mix to see how the team handles unexpected challenges:

  • A new dependency complicates the fix.
  • Stakeholders demand an immediate resolution.
  • Another bug shows up during the discussion.

Reflection Checklist

As the Incident Lead:

  • Did you keep the discussion balanced?
  • Did you ensure all perspectives were heard?
  • Did you lead the team to a clear decision?

As a Participant:

  • Did you provide clear, relevant information?
  • Were you open to different solutions?
  • Did you communicate potential risks effectively?

Common Mistakes to Avoid

  • Making a hasty decision without enough information
  • Letting one viewpoint dominate the discussion
  • Not documenting the decision-making process

Pro Tip

In high-pressure situations, staying calm and ensuring a structured decision process can significantly reduce risks and maintain team confidence.