October 2016 saw several DDoS (Distributed Denial of Service) attacks of unprecedented magnitude and impact. When I am impacted by such things, I’m always a little irritated. “Well, if I were administering that (network|data center|server),” I tell myself, “this would never have happened.”
But let’s be real. I can’t say that. Not with any degree of certainty, anyway, and most certainly not without a LOT of hubris. Even with the best centralized logging and diligent engineering teams, during events like the October 2016 attack, any analysis or determination of causality will likely be post-mortem. Mitigation measures have proven themselves effective to varying degrees, but they all have one thing in common: they’re reactive, not proactive.
In a world where even refrigerators are connected to the Internet and bandwidth is cheaper than the compressed air I breathe while SCUBA diving, the opportunities for (and avenues of) attack are simply too large in number for the best of sysadmins, engineers, and teams to anticipate. We cannot act proactively much of the time; this because attack vectors simply can’t be predicated with a high level of certainty.
Security is essential, that stands without further discussion. Logging is a critical part of security, but I’m not going to touch on it in this post. This post is about logging’s more insightful, painful-to-implement, more-difficult-to-manage cousin: monitoring.
Monitoring is increasingly important, not just for the obvious reasons (performance, trending, etc.) but also because well-designed, well-implemented, and scalable monitoring may serve as an early warning system in terms of recognizing hostile activity. Of course, this assumes your monitoring system(s) are not themselves vulnerable to such activities. It does no good to monitor your systems and networks if your monitoring systems can be taken down along with the rest of your systems and networks. So some planning is definitely key.
I brought this up to the folks in Linux and DevOps team here at Linux Academy. I’m writing the curriculum for the Nagios Professional Certification, and I came across a somewhat novel idea to demonstrate how Nagios can be used for exactly the purposes I’ve described above: war games.
That reminds me a bit of the Cold War and Matthew Broderick in the 1980s, though, so I’m calling the whole affair “Nerd Wars.” Sounds a lot more fun.
I’m not going to divulge too many details here, at least not yet. But suffice it to say that there are two teams: me and THEM. “They” are my colleagues at Linux Academy, each formidable in his own right. As a group? Well… I think I might be doomed from the get-go, but we’ll see.
So what is the scenario? Let’s frame the battlefield for a moment with a quick walk through history. The City of Rome was invaded by the Visigoths (folks from Northern Europe) in AD 410. I’m Rome. Or rather: I’m going to set up a fake blog (lots of Loris Ipsum, and so on) and my servers are Rome. My colleagues? Yup, you guessed it: they’re the Visigoths.
My goals are three:
- Keep the fake blog up.
- Keep the systems from being compromised, keep them accessible.
- Anticipate, detect, and if possible, neutralize whatever attacks and tricks are thrown at me.
- In short: just stay “alive”
My teammates have considerably less to worry about and more to work with:
- Their servers don’t have to stay up.
- There are no holds barred in terms of what they use.
- Their servers don’t have to stick with a fixed configuration in order to maintain uptime.
- They can attack at any time without warning.
- They may act collectively or unilaterally.
Now first and foremost, this is going to be lots of fun. Lots of fun to set up, participate in, and of course, we are all well aware that Bragging Rights in the IT Industry are like titles among the aristocracy. You can pass them on to your children and they entail a certain degree of privilege.
But this exercise is serious as well. It is a case study of monitoring – in particular, of monitoring’s efficacy as an early-warning and detection tool. And of course, it will be interesting to see to what degree monitoring provides us information and tools to mitigate or prevent attacks.
There will be more to come on the Nerd Wars – keep your eyes peeled for blog postings with that title. In the meantime, if you’ve got ideas about how we should (or should not) go about the Nerd Wars, leave a comment.
Until next time!