Managing Risk

IT Risk management is an ever expanding area of interest throughout the software development community. Traditional risk management practices are applied to ensure organizations can withstand unfortunate events like power outages and natural disasters. Additional processes are layered on to deal with events particular to production software systems, such as security vulnerabilities or severe bugs.

In years past, a specialized security team usually did the deep thinking around these issues. The watchful eyes of an operations center monitored production systems state. Developers were left to write code and not worry too much about the underlying systems and latest threats. Developers were not the people to call when something bad happened at 2am.

That division of responsibility is rapidly disappearing as system complexity, exploits and hacks increase. Software developers are recognizing that writing careful, secure code is the first-line of defense against malicious actors. Developers, especially in small shops, are expected to understand the fundamentals of operations, and to take a turn at the monitoring station. Devop skills are often required to setup and navigate self-service cloud-based architectures. Developers are also expected to have a holistic view of their systems and intimately understand the architecture.

Here at the CTL, we’ve spent a lot of time thinking about risk management over the past year. A comprehensive assessment challenged us to become more aware of the risks we face as we develop technology solutions for students and faculty. The process touched all areas of our organization, ensuring that we engage in best practices on a multitude of measures. Not that we were slouchers to begin with, but our current standings show a much stronger awareness around security, a trove of new processes and more solid documentation.

As many shops are engaging in this type of risk management exercise, I thought I’d share a few basic things that got us through a period of intense introspection.

Stay open

Taking a deep look at existing practices can evoke feelings of defensiveness. Breaking out of old patterns is scary. Crafting new patterns takes a lot of work. Ugh.

For us, maintaining a collaborative, open mindset was key to getting through the “ugh” stage. Admitting ignorance, asking lots of questions, playing with solutions and making mistakes transformed our experience. Discussions around NIDS (network-based intrusion detection systems) and HIDS (host-based intrusion detection systems) definitely produced a blank look on my face and an intense desire to procrastinate. I took the needed time to read the basics, and now have a good understanding of how these systems work.

Lean on lightweight, living documentation

We’re lucky here to have a long history of collaborative documentation. We use MediaWiki to record everything from our staff meeting schedule to project particulars. We turned again to our wiki to organize and enhance existing documentation for this exercise. The wiki format makes it easy to build on our past notes, quickly correct mistakes and smooth out poor instructions. We also continue to expand on the checklists approach for repetitive tasks, like the daily systems checklist.

A great example of new documentation emerging from this exercise was around vendor management. Unwritten steps around third-party relationships were captured in detail, including where to stash important documents. An on-boarding checklist was generated that guarantees we record data export procedures, support numbers and billing details. During a recent vendor outage, I was able to easily remember where to find necessary information and get a resolution.

Not “if”, but “when”

The idea behind risk management is not just to prevent data breaches and system downtime, but to understand how to cope with these events and recover from them. The question is not “if” an event will happen, but when. And when. And when. New threats, new patterns and new vulnerabilities continue to emerge daily.

We understand now that the risk management exercise will never be complete. That understanding is probably the most valuable thing we’ve gained. We’ve done a good job of getting our house in better order this year. But, we’ve also built in regular reviews to maintain our introspective posture. At least we’ve made a beginning.