Addressing Systemic Issues from Repeat Incidents

crisis-leadershipSenior15–20 min
How to Use This: Try this roleplay with 2–4 people, or go solo by taking on each role. Throw in curveballs for extra challenge. Reflect afterward to see how it went.

Roleplay Scenario

Scenario Overview

Your team’s been facing a string of incidents in a key system area. They’re not major, but they’re happening often enough to be a pain and attract some attention. It's time to talk about what's really causing these and how to fix them for good.

Roles & Setup

Role A – Incident Leader (You)
You’re steering the discussion, aiming to figure out the root problems and get everyone on board with a plan to fix them.
Your goal: Keep the talk focused on finding real solutions that’ll last.

Role B – On-Call Engineer
You’ve been dealing with these issues firsthand and know the system quirks well.
Your goal: Share what you’ve seen and work with the team to pinpoint the root causes and solutions.

Role C – Senior Engineer (Optional)
You bring a wealth of experience and can help assess the feasibility of different solutions.
Your goal: Offer guidance and make sure solutions are solid.

Role D – Product Manager (Optional)
You’re keeping an eye on how these issues affect customers and the business.
Your goal: Ensure solutions align with business needs and keep customers happy.

Suggested Openers

Incident Leader:

  • “Thanks for joining, everyone. We’ve seen these incidents in [Core System] popping up too often. Let’s dig into what’s really going on and how we can stop it from happening again.”
  • “I’m hoping we can focus on solutions that’ll stick. What patterns or gaps are we noticing?”

On-Call Engineer:

  • “These incidents seem to happen after certain updates. I think there might be something missing in our checks.”
  • “I’ve noticed a pattern with [specific condition]. Maybe our tests aren’t catching everything?”

Senior Engineer:

  • “From what I’ve seen before, we might need to tweak our architecture or improve processes.”
  • “Let’s check if our deployment or monitoring processes are part of the problem.”

Product Manager:

  • “These incidents are starting to affect how users see us. Let’s make sure we’re fixing things in a way that helps them.”
  • “We need to balance fixing these issues with our broader goals. How can we do that best?”

Sample Roleplay in Action

Incident Leader:
“Thanks for being here. We’ve had a few too many incidents with [Core System]. Let’s figure out why they keep happening and come up with a plan that’ll work long term.”

On-Call Engineer:
“I’ve noticed these incidents usually happen after updates in [specific module]. I’m thinking maybe our integration tests aren’t catching everything.”

Senior Engineer:
“That’s worth looking into. We might need to beef up our testing and review our pipeline for weak spots.”

Product Manager:
“I’m worried about how this is affecting our users. We need to make sure our fixes align with what we’re trying to achieve and manage expectations.”

Incident Leader:
“Let’s map out what these incidents have in common and see where our processes might be falling short. We should look to improve our testing and monitoring.”

On-Call Engineer:
“I can put together a plan to expand our tests and automate more checks.”

Senior Engineer:
“I’ll review our architecture to see if there are changes we can make to prevent these issues.”

Product Manager:
“I’ll make sure any changes fit our strategic goals and keep everyone updated.”

Incident Leader:
“Great, let’s touch base next week to review our plans and get moving. Thanks for your input, everyone.”

Post-Scenario Tools

Curveball Mode (Optional)

Introduce one of these challenges during the roleplay to see how participants handle unexpected situations:

  • A new incident happens during your discussion, adding pressure.
  • Stakeholders push for quick fixes, conflicting with long-term goals.
  • Technical constraints make some solutions hard to implement.

Reflection Checklist

As the Incident Leader:

  • Did you guide the conversation towards lasting solutions?
  • Did everyone have a chance to contribute?
  • Did you help the team decide on concrete next steps?

As a Participant:

  • Did you share your insights clearly?
  • Did you suggest practical solutions?
  • Did you balance technical needs with business priorities?

Common Mistakes to Avoid

  • Focusing only on quick fixes without addressing the real problem
  • Ignoring stakeholder concerns or business impact
  • Letting the conversation drift without action items

Pro Tip

Solving systemic issues takes teamwork and foresight. Encourage different viewpoints and aim for solutions that work technically and meet business goals.