Devopssecfail
- 1. DevOpsSecFAIL DevOps Security Anti Patterns Dr. Constantine Aaron Cois Carnegie Mellon University
- 2. Me @aaroncois www.codehenge.net github.com/cacois Disclaimer: Though I am an employee of the Software Engineering Institute at Carnegie Mellon University, this work was not funded by the SEI and does not reflect the work or opinions of the SEI or its customers.
- 3. DevOps
- 4. DevOps
- 5. DevOpsSec DevOps is a Risk Mitigation strategy, built on Situational Awareness, Automation, and Repetition
- 6. DevOpsSec But security is where a lot of DevOps implementations Fall Down
- 9. TheException You automate… …builds …functional tests …deployment …reporting …the coffee machine Image: https://lh4.ggpht.com/z_w-yCMvUcrqZd_6eXlt7E24YvSHEak1k5lNvk5GGNYmzMaBQkH1oe3emhZk0scIWg=w300
- 10. TheException But security testing is still manual pen testing, done only on release
- 11. Automate Security Testing removes… …Human error …infrequent execution …Excuses
- 12. There are great projects out there OWASP ZAP https://www.owasp.org/index.php/OWAS P_Zed_Attack_Proxy_Project http://gauntlt.org/ GAUNTLT BE MEAN TO YOUR CODE AND LIKE IT Help improve them!
- 14. Image: https://artcritique.files.wordpress.com/2014/01/multiverse-and-schrodingers-cat-in-play.png
- 27. When nothing looks the same you can never be sure your app will behave the same
- 28. When nothing looks the same you can never be sure your app security features will behave the same
- 30. TheConfigurator Manual configuration, done buy your best and brightest... Image: http://2vga1o5mew51s6gu7x0mnk7kf.wpengine.netdna-cdn.com/wp-content/uploads/main/2013_06/A-Cat-Snatching-Wires-Out-of-a-Server.jpg
- 31. TheConfigurator …will still lead to an unmanageable, unpredictable, and unrepeatable solution Image: http://assorted-images.s3.amazonaws.com/datacenterinfrastructure/messy%20data%20center.png
- 32. TheConfigurator If it’s not Automated it’s not Done
- 33. TheConfigurator If it’s not Automated it’s not Done Secure
- 35. TheInfiltrator He sneaks in… …and alters production …but he works for you! http://blog.landesk.com/wp-content/uploads/sites/4/2012/05/ninja.jpg
- 36. There is always a reason to make a manual changes But don’t do it!
- 37. Unexpected manual changes are often… Undocumented, Unauditable, Unrepeatable
- 38. Unexpected manual changes are often… Undocumented, Unauditable, Unrepeatable Insecure
- 39. Protip Configure production to alert the entire team when manually accessed. Transparency is key
- 41. We’ve all been there… Intrusions overnight… …lock it down… …cascading system failures… …it’s all crashing…
- 42. It feels like…
- 43. But it ends You survive You’re out of the woods. Just glad its over. Going to go sleep for 18 hours… …and then back to normal.
- 44. Survivor mentality defeats continuous improvement When do we analyze what went wrong? How do we prevent similar failures in the future? All failures must result in codified change to DevOps process.
- 45. This attitude persists… …when we don’t expect failure. We should always expect failure. Be ready for it. Plan for it
- 46. After action rules for failure Understand exactly what went wrong Never let the same failure happen twice Propagate fixes across the enterprise Ensure that you teach the next generation
- 49. TheCollegeParty 99% of Global 2000 companies will be using open source code in mission-critical apps by 20161 1 http://www.zdnet.com/article/scan-open-source-use-to-minimize-risks-optimize-benefits/
- 50. TheCollegeParty Do you know what’s in your app? Code we wrote Code someone else wrote Image: http://acardiac.blogspot.com/
- 53. TheSkydiver Once you jump, you can’t return to the plane. You are committed. Permanently. This is not how we should model our deployments.
- 54. TheSkydiver Rollback is essential Never be left without an escape route to completely working software.
- 55. QUESTIONS? Any
Editor's Notes
- #5: In case you aren’t good at Where’s Waldo, here is security. It was mentioned twice in the wikipedia article, which is three less times than the word ‘Patrick’, but one more time than the word ‘portmandeau’.
- #16: We start with dev. The first environment for our app.
- #17: But then, of course, we have a pipeline of environments that get us to production. At very least, we need a test (or QA) environment and a staging environment.
- #18: And our app migrates from dev…
- #19: …to test…
- #20: …to staging…
- #21: …and eventually to production.
- #22: A really common pitfall in devops comes when we don’t maintain rigorous, 100% parity between these environments – or when we create a multiverse. Let’s say the testers open a few new ports on servers in the test environment, to access some specialized logs.
- #23: Some late-night firefighting causes some IT ops and networking folks to change some configurations in prod to restore functionality. InfoSec also locks down a few networking settings, in response to some potential attacks they are seeing. This is all done at 4am after 8 hours of fighting, so it’s not immediately recorded, but the guys involved know what happened.
- #24: And, of course, the developers need more access in their dev environment. Some more ports open…
- #25: …some more packages and libraries installed…
- #26: …and update some libraries to the latest version…
- #27: But now we have a problem – all of these changes mean our app has the potential to work in some of our environments, but not in others. This may be an obvious failure, leading to a failed deployment, or it may just lead to an undetected security hole in production, that doesn’t exist in any of the other environments. Without complete environmental parity, all of our testing on non-production environments dramatically loses value.
- #28: What caused this? MANUAL CONFIGURATION. ANY AND ALL MANUAL CONFIGURATION IS DANGEROUS. Extremely dangerous. This means that all production firefighting has to go through automated channels to make changes? Sound like it will slow you down? It may, especially if everyone isn’t used to (and on board with) the process and the tools. But if you have full parity in all prior environments, and a solid automated deployment strategy that everyone is comfortable using, it won’t. And you will always have maximal confidence that anything that works in dev will work in prod, including security features.
- #29: What caused this? MANUAL CONFIGURATION. ANY AND ALL MANUAL CONFIGURATION IS DANGEROUS. Extremely dangerous. This means that all production firefighting has to go through automated channels to make changes? Sound like it will slow you down? It may, especially if everyone isn’t used to (and on board with) the process and the tools. But if you have full parity in all prior environments, and a solid automated deployment strategy that everyone is comfortable using, it won’t. And you will always have maximal confidence that anything that works in dev will work in prod, including security features.
- #33: By which I mean something automated. We can’t keep the tacit knowledge in the humans. We don’t want to just create expert humans, we want to create a customized, expert process for our project.
- #34: By which I mean something automated. We can’t keep the tacit knowledge in the humans. We don’t want to just create expert humans, we want to create a customized, expert process for our project.
- #36: While we’re at it, let’s talk about those “unavoidable”, ‘necessary” changes to production to keep systems up and running
- #40: PINK SOMBRERO – Babcock Jenkins
- #45: By which I mean something automated. We can’t keep the tacit knowledge in the humans. We don’t want to just create expert humans, we want to create a customized, expert process for our project. This tends to happen when we don’t expect failure. We should always expect failure, and be ready for it. PLAN for it. And by the way…How did you fix it? Are those changes propagated back into the automated pipeline?
- #46: By which I mean something automated. We can’t keep the tacit knowledge in the humans. We don’t want to just create expert humans, we want to create a customized, expert process for our project. This tends to happen when we don’t expect failure. We should always expect failure, and be ready for it. Plan for it. USE it.
- #47: By which I mean something automated. We can’t keep the tacit knowledge in the humans. We don’t want to just create expert humans, we want to create a customized, expert process for our project.
- #51: Now, I say this for two reasons. 1) it probably scares you a little bit, and I enjoy that. But 2) do not make the mistake of trying prohibition. It won’t work, and it will make it harder to keep track and regulate your security posture. InfoSec needs to work with developers and ops to mitigate this risk. This is important – InfoSec needs to understand DevOps, and work within a devops process to address this risk. That means working within CI, working with automation. If you say “We need 2 weeks to assess and approve any new open source library you want to use”, guess what? You lose! You are now the primary bottleneck in the DevOps pipeline, and everyone else will be trying to mitigate YOU.