The Power Of Lessons Learned In Incident Response

Our cyber team has spent two years transforming and maturing the incident response function within our cyber security program. While two years is an eternity in cyber time. We aren’t done. That’s because continuous improvement isn’t just an idea anymore. At this point, continuous improvement within our program has been operationalized through a fresh take on “lessons learned” which has become a fundamental part of the incident response program.

Two years in. We could have simply shrugged and collectively accepted things as they are and faced the same issues over and over. But, instead, we made a lot of progress. We’ve identified many of the processes that slowed down our incident response capability as a team. We’ve had sufficient time now to budget, implement, and mature controls that protect our network, systems, and data. We’ve added playbooks.

These didn’t happen overnight but has been incremental over two annual budget cycles. And, still, two years later, we still find opportunities for improvement in every incident. Some meaty list of lessons learned still accompanies each and every incident. Why? Not because the incident response program remains immature but because we still actively look for lessons that can be applied.

Want more resources? Lessons learned that are detailed, uncompromising, and actionable can be a weighty resource driver with executives. Properly presented, executives can immediately see the linkage and the benefits. You won’t have to explain some entirely theoretical potential act occurring on the other side of the globe or outside of your industry. Executives will have real data coming from initial unfiltered summaries of incidents involving their own systems. Accompanying the normal incident summary will be the first clear set of lessons and an alternative future outcome to that lesson that executives can understand, internalize, and act upon. These make powerful statements and be the start to very compelling resourcing conversations.

They are made even more powerful when the initial summary generated within a hour or so of the end of the incident. The fast commentary sets an expectation that the lessons learned are fresh, unfiltered, and raw. You can wordsmith in a more formal version later if you need to.

Obviously, the summary for a given incident includes an overview of the incident and a high level view of the times/details of activities. That said, in your standardized incident response summary template, you should also have a section entitled, “Lessons Learned And Areas For Improvement”. This second section ideally will be filled with intentionally raw and unfiltered unapologetic bullet points….

• What control(s) weren’t in place or didn’t work as advertised?
• What needs to change in the order of operations or nature of the team’s response?
• What response steps or critical information was missed during this response that shouldn’t have been?
• What key information or files couldn’t we get to fast enough?
• What should have be automated that isn’t?
• What was the chokepoint this time?
• Who should we have brought in sooner?
• Was this incident different enough to warrant its own playbook?

By focusing on the process in the second section rather than the incident summary details, each incident can be part of an ongoing narrative that identifies the gaps in your incident response plan or playbooks for different types of incidents. In short, the obstacles that keep the team from responding better and faster than they just did.

Example lessons learned:

• “manual searching across all email servers meant that the weaponized emails were still being downloaded by other employees before the email could be found and deleted.”
• “Since this was the first incident on X, the team didn’t have the right set of permissions to delete files from X and had to wait until an administrator from that area could be contacted”
• “We did not have a control that could detect [description of detection required]”
• “A control was in place but the expected configuration had been changed or disabled without the knowledge of the cyber security team.”

Every organization has something to learn or areas to improve regardless of the level of funding. The summaries aren’t just to administratively file away. They’ll automatically become part of your executive communication toolbox because they are part of the incident summaries. You’ll want to build on these lessons as future conversation starters with executives because they make great levers for an action plan and request for resources to correct the issues.

In our own program, we used initial lessons learned to more quickly identify missing controls, automation, and internal process issues that were keeping the organization at significant risk. These were noted whenever they were causal factors in every incident summary to execs. When the time came to budget for the remedies to these issues, the justification was easy. The execs were usually already aligned around the issues when the ask came in. That’s the best way to start a resourcing conversation.

You have the right tools in your toolbox. Use every tool at your disposal to improve.