Table of Contents
We’re priviliged to have so many great conferences from which to choose in the tech industry. If we’re able to attend even half of them, it still means we miss a considerable number of quality conferences. We want to make it easier for you to catch up beyond the videos by providing transcripts from our favorite microservices talks. This week, we selected a talk by Laura Bell, Founder and CEO at safestack.io who shared some security tips at GOTO Chicago. We hope you enjoy the presentation as much as we did!
Practical Microservice Security
Presented at GOTO Chicago
Alright, good morning. That was my nice mic check, it sounds loud to me. Can you all hear me? All right, fabulous. Are you all caffeinated? Yes, good, awesome. That’s even better. My name is Laura. If you like there are screens on each side. You can find me on the Twitter as @lady_nerd. I mention this because the underscore is important. The real lady does not understand microservices, or security, or like people who find me after a conference so please find the right one.
This talk is being recorded, so please excuse the slide; this is for the people who are going to watch videos at home. We are in a fast-paced industry; really, really fast-paced. This is all relevant for today, all of the examples are pulled from the last couple of months, some of it from the last couple of days. If you’re watching this at home, please make sure you’re watching the latest advice for whatever field you are in because seriously there are some scary stuff out there.
Who the hell am I?
Why am I on a stage here? You can tell by the accent I’m not from around here. I live in New Zealand, I’m from the UK originally and I run a small security firm called SafeStack. I’m not here to sell to you, which is a beautiful thing because I live thousands of miles on the other side of the planet. What do I do? I’ve spent the last 10 years doing bad things to good people’s code, to good people in general: lying, cheating and stealing, and doing a bit of software development on the side.
This is my window in my office. It’s my Kanban board. We spend 50% of our time doing software development and building things, and 50% of our time helping very fast-moving companies do security. That means doing security in a way that doesn’t get in the way of doing awesome things because … Please raise your hand if you think security is the cool fun part of your job. Just me, right, good, all right.
In this talk
What are we going to do today? We are going to look at some fundamentals obviously, we’ve got to start at the same place and I’m not going to make assumptions about where you are all at. We are not all the people who wrote the book on a subject, we don’t all get to use unicorn technologies and we don’t all go cloud first all of the time. I know this because I work with people just like you. We are going to talk about prevention and how we prevent some of the attacks that are going to happen against your microservice architectures. Then we are going to talk about detection because actually prevention is an impossible puzzle. You’re never going to be 100% secure, we might as well give that up just right now. If you came thinking I will be safe at the end of this, nope, sorry, you’re screwed. We are going to talk about how we know that we are in trouble because that’s just as important.
Let’s start at the beginning. This is how we used to build applications. I don’t like putting code slides up very often because most of you spent time looking at my syntax and I’m what they call an Ops coder, which means that I’ve written some of the dirtiest C++ you’ll ever see in your life and I’m proud of it. I write Python day-to-day. You all are just going to judge me by language choice. So this is a monolith. It was a big piece of naturally evolving source code, which basically meant we gave some interns a project to write once. They write a really good little project, we put it live on the Internet, and 15 years later we inherited it. Somehow that is still live and in production. We’ve now split that into lots of little bits and pieces, each with its own age and story, sometimes each with its own technology. That’s where we are right now.
[3:54] Why is this important for a fundamental? Because what we are talking about is we’ve changed our attack surface, we’ve changed our architecture. We should know the basics of microservices by now. You’ve literally got the person who wrote the book in the room running this kind of track today. I don’t need to go into this in-depth, but we know the benefits of that that are there. For someone like me, however, someone who has lived in the dark place on the Internet, has spent a lot of time breaking things, it changes the way we have to approach security a little. You’ll have to excuse the voice, I’ve flown in from New Zealand and somehow I caught airplane plague.
What do we care about in security?
Before we get into the prevention and the detection, all of the practice stuff you want to get on with, let’s get a couple of foundations straight. There are three things I care about as a security person. I care about confidentiality. If you trust me with your information and you only trust me to give it to one person in the entire world, then that is my duty of care. That’s what I have to do.
How many of you work for an organization that stores data from other people? Good, awesome, most of you in software jobs then. That’s good. What we do day-in day-out, we are secret keepers for all sorts of people, whether that’s card information or personal information, health information, it doesn’t matter what we are storing. You could be writing Tinder for French poodles, it doesn’t matter, you are storing something for someone else.
Integrity. If we start with our data in form A, we want it to be form A at the end, or at least we need to know what form it’s going to be in. Availability, let’s not forget, a lot of people say, “Well, a secure system, just turn it off and wrap it in concrete and put it at the bottom of the ocean. That’s your secure system.” It’s just not true. Secure systems are available systems, they are systems you can actually get to and do the job you want to do. Security people, we are not actually the main attraction here, we are like side cast. We are here to make sure what you do is amazing and people can use it because if they’re not using it, we don’t get paid, and I quite like being paid.
[6:03] The last bit of our foundation is the types of things we are going to care about in this session. If any of you have done threat modeling before, you’ll notice this is a set of stride threats that we look at. It’s the common six security risks faced by most applications, so spoofing, tampering, repudiation, information disclosure, denial of service, escalation of privilege. These are the things that are our path as an attacker to getting the good stuff. This is what I am doing when I’m attacking an application to try and get to your data or to other systems you control. Don’t forget, I might be doing these against you, I might be doing it against your user base, I might be doing it against one person to get to another. It’s not all as simple as we like to make out.
Let’s start with prevention. How on earth do we stop getting attacked in a microservice space? Because hopefully we get all of that a bit perfect, we never get attacked and then we never have to worry. How many of you’ve said that, “I’ve never been attacked so it’s awesome, I must be doing it right already.” Anyone? There is normally one person, even if in your secrets, you are like, “Yeah, I’ve got this.”
Prevention is actually a big area, and we spend almost all our resources on it, but is what we call an asymmetric problem. For every developer out there, there are ten attackers doing vulnerable to research or looking for some way to forward their objectives and motivations. When we try to prevent them, we don’t have as much resource or time as our attackers do. It’s a flawed problem, we can never actually get it to the point where we could fend them all off.
Bad advice is bad
Let’s talk about some basics first. Oh my God, the Internet is a terrible place. How many of you have ever written some code and got stuck part way through and gone on to StackOverflow for a solution? Anyone not raising their hand is a liar. I’m not saying you can’t do this, not by any stretch of the imagination, we all do this.
In fact places like StackOverflow have become a good source of crowd-sourced information. However, not always true, this little gem came from StackOverflow this morning recommending that our lovely question, the original question producer should just use MD5 to store his passwords. Great, because anything else is just being paranoid.
Now how many of you have a graphics card in your computer? Yes, all of you do. Most of you, I hope. There is a reason we don’t like MD5 anymore is because that little graphics card in most home computers is now fast enough to crack passwords using MD5 that even when they are assaulted, this is not a control, but the internet said so, so somebody somewhere is still building a system like this.
Firstly, raise your hand, be honest if you don’t know what OWASP is. All right, hi, welcome, this is the most important lesson you’ll take from today. OWASP is the Open Web Application Security Project. It is free, it is a global community of security people who are working to give free resources to the community: tools, guidebooks, cheat sheets, all about security. It’s awesome, you should be getting involved. There is a link on this slide, the slides will be shared afterwards, but just go and dig in now, it comes with a caveat. Their website does look like it escaped from 1998. That’s not my fault, I’m sorry, but the advice is still very good. That’s the last of our places to go for information, so StackOverflow can be good, but use some common sense, check authoritative sources first.
Auth vs. Auth
Or why acronyms make you less secure
[10:31] Now this is my personal crusade so. What does Auth stand for? Can we all be clear for a second? We have two auth words in our life, we have authentication and authorization and they are not the same thing. Now when we are dealing with microservices, it becomes really important to get this straight because the components that are doing your authentication may not be the same components that are doing your authorization. Your authorization might be happening differently in different places of your application.
Whenever I see a specification that says “Do Auth stuff” I get a little bit concerned because often we generalize and we put them into the same bucket. My first thing I ask you your homework, any time you go and see a requirement spec and it says “Do Auth”, please just push it back with your red pen just go, big cross through it, say try again try harder because this is the root of our problem in security. We use buzzwords, we use abbreviations and we don’t really consider them.
Now in microservices we have the luxury of breaking down our components and actually doing this in individual places. Now we are going to come back to orchestration layers afterwards because they are fairly significant in this space, but you don’t have to have one. You can just create your little services to either be atomic just going to work on their own, or you could just have one of your little services that does authentication and another one does authorization. You’ve got ultimate flexibility. That’s part of the beauty of this, part of why it can scale and part of why you can send it over multiple teams. At the same time, it means you don’t have an even surface when it comes to your authentication or your authorization.
Now if we look at authentication first, we might identify central components like the little white square on this one and say, we’ll have two factor on this one, it’s definitely important in the middle. What we don’t factor in is there is implicit trust between our microservices, that while I uses my authenticate and authorize at border points where they interact, there are little trust mechanisms going on inside the app that mean you can go from a low trust application to a high trust application without really noticing you’ve done it. Ooh, I’m losing microphone. It’s good, awesome.
The thing with authentication, it has to be consistent, and it has to be planned. Now this might seem like common sense, but a lot of the time I’m seeing microservice architecture spring up in places that have a large legacy code base. How many of you are working with the legacy code base? How many of you have still got .net God knows what version old in your life? Java 1.4? Oh, I’m sorry. This is the world we live. Our services are not all written in latest node with React UIs and with Scala over here and Rust over there. That’s not how this works for most of us. You end up with components that are limited by the technologies that they are actually written in. Old versions of Java and .net do not have the same features as new versions of some of the later languages. We have to have a consistent planned approach and we have to be able to test it.
[13:44] Authorization, now authorization is your next step. Your authentication is you are who you say you are. Great, wonderful, I’m going to let you in the door. But am I going to let you do anything? Probably not by default hopefully. Think of this like your sliding doors going into the lobby of somewhere. Sliding doors have a tendency to fail open. If they break they are going to let you in and out because if there was a fire or something they are going to let you leave the building and not have a horrible ending. In security, we actually want to go the other way. You do not want to be your sliding doors, you want to be a prison. If things are going wrong, if you’ve not made a decision it should be closed.
Many frameworks give you this, you can use decorators for your services or you can use something, you need a general configuration. Have your base one, the one it’s going to fall through to be, “Nope, get out, nothing for you here” because failing open is really dangerous for us. We don’t want to have public facing APIs failing open. This has to be applied to every object, every method, every endpoint in your architecture.
Now that’s fine when you are doing like Go one-on-one microservices and you’ve got three microservices and you are feeling really spectacularly good in the world. I’ve got customers who’ve got 300+ microservices. That’s not easy to do on that kind of scale.
Storage, Quality, Length, Lifecycle
The keys to token success
It also gets challenging when we start looking at the mechanisms we are using for our authentication or authorization, the tokenization. There is lot of different schemas you can use for your tokens, you can homebrew. Absolutely, anything you can write, they can definitely write it, then do it or consistent with it. I probably wouldn’t, but you can. The problems we are seeing commonly … Oh goodness, I do apologize, one second. I’m going to end up holding this.
[15:45] The problems we commonly see in the authentication-authorization space is that the tokens are not being well managed, they are not high-quality. They survive much, much longer than they should. This is not a new problem, this is an old problem, we’ve had this for sessions for a very long time, but now we’ve got different tokens so the only thing that we are passing back and forth, this is very large token.
[16:09] Now things you need to consider, where they are being stored. Where are your tokens typically being stored in your applications now? It could be nowhere, it could be you are just writing curl requests or using post- and it’s just getting copied and pasted in. It could also be that you are storing them in local storage or somewhere in a mobile device. Remember a lot of these storage places are not secure by default, these are not encrypted stores.
If you’ve got Chrome on your laptop, you can interfere with your local storage and mess around with those tokens. If you can do it as an engineer, you have to assume that your users can do that too and your attackers. You need to be handling that token very carefully and aggressively monitoring for strange behavior or strange usage of that token because if that usage pattern changes, you need to go, “Nope, I don’t trust you right now, we are going to need to re-authenticate, we’ll need to check that again.”
We need to do input validation now. It’s been a long, long time since I’ve been in this field and the one lesson we keep saying again and again and again and again and again and again is please validate your input. Yet the OWASP’s Top 10 has remained largely unchanged for about ten years because input validation sucks, we are not good at it, at all.
Now many of your frameworks do really good stuff in the standard setup, so .net does some amazing stuff, C# does some amazing stuff. Django, which I spent a lot of time in has some really good defaults out of the box, but the second you start writing custom APIs or things that don’t use the big bulky generic framework stuff, and you just try and write something lightweight on the side, you forgo some of this.
The other thing that happens is we expect all of our languages to do this for us. If you’ve come from say a C# background and you moved to something like Node, for example, you often assume that if it’s using the same word, it means the same thing, that it really isn’t. I’m going to come back to that when we talk about the languages we are using and how they are affecting our security.
[18:11] Validation has to happen on every header, every request on every method. That sounds really common sense, but if you couple this with the fact we very rarely put the right content type on our returns, that validation is stopping us returning scripts and things to our UIs, which don’t know any better, they are just going to render things onscreen.
You’ve got a monolith, great, awesome, fantastic, means you are in business, and that’s great. You are going to break it down into small pieces and you are going to ship it onto the internet as microservices and the world will be a great place. Awesome. Regardless of how many books you read and how many videos and how any tutorials you watch, this actually isn’t that easy first time around, especially if you’re dealing with a complex existing application. We tend to split it, we’ll code our small service, start deploying them out and then we move onto the next one. It’s very hard to come back to the old ones to have a look at them.
Now that has an interesting effect when we start looking at the security of our applications. This is a crude example again seen in the wild. Obviously no code, don’t get confused, it’s the same lesson just without the code though. We have three microservices, two of them using JWT tokens for their authentication stuff, great. Simple, good labors in frameworks.
Last one, what’s that? Base64. If you are not familiar or it’s been awhile since you look to the difference between encoding, hashing and encrypting, that’s not particularly secure. Also if you were to decode it, it’s probably not a high-quality token either. Remember the way to the heart of an application as an attacker is through its weakest part. Normally the oldest service you’ve written, the one you wrote when you were still watching the tutorials from Adrian Cockcroft or something on the internet. Go back and look at those old bits of code. I know that I have gremlins in my olds bit of code, I’m probably sure you do too.
Size and complexity
[20:25] We all know that microservices is supposed to be small and simple. That’s quite straightforward. Awesome. How many of you’ve come across a monster microservice? A microservice that is pretending very, very hard that it’s a cool kid and it really wants to come to the party, but actually you took a monolithic developer and you said, “Make it an API.” Lo and behold they did, but it’s giant, it’s 500+ lines of massively complex highly coupled code that probably does three or four different functions when it should be doing one.
Yep, this is microservices, and I’m not here to tell you the glorious future of microservices, I’m here to show you the skeletons that I see every day. That’s okay, don’t worry, if you see that kind of stuff it means you are in a normal organization. You haven’t quite got there yet. That’s all good. Call it out. We all know the rules if you are doing a code review, if you are given 10,000 lines of code to review, you will spot how many bugs? 10,000 lines, zero. You’ll say, “It’s fine, ship it.” If I give you six lines, you’ll find ten bugs. Treat your microservices in the same way. If you’ve got thousands of lines of code, it’s not a microservice. It’s okay. Do it in a blameless, nice kind of way, but just say, come back again, try again.
Remember that while we are breaking down our services, there is going to naturally be some coupling between them. It’s a highly unusual for you to be able to make microservices that can work entirely atomatically all on their own. Now that’s interesting when you start breaking out some that do heavier lifting than others, for example, batch processing or dealing with any kind of large data transformation or say your authentication system because if you don’t consider this coupling you end up in denial of service territory. If I can’t get my objective so if I want to take money or have problems … Oh, my microphone again. There we go, awesome. Oh, I’m going to get cello tape, this is fantastic. If we have got services that connect to each other, and I can’t get to my objective, so my objective is your intellectual property or children’s data or whatever I’m trying to get to, then sometimes just causing a denial of service can be an effect in and of itself. I can destroy your reputation in the industry. I can interrupt a crucial transaction that’s going on.
Making sure that the coupling in your services means that if one fails that the others don’t go down with it. How many of you do any kind of load testing or failure testing in that way? If you don’t it’s a really fun way to do testing. If you think testing is boring, this is the one to get started in because basically you are going to destroy your entire architecture. It’s all for good things. You can get KPIs and things for this.
Scaling and resource exhaustion
[23:23] The other thing to know is it may not be your code that fails under pressure. Now we’ve all decided that using the cloud is good and we’ll come back to that. I like the cloud, don’t worry about the cloud. How many of us love AWS’s documentation? That’ll be no one. Sometimes it’s not our code that is going to cripple our microservices, it’s whatever we’ve hosted it on and however we’ve configured it. Whether it’s going toward to autoscale or not autoscale, whether you’ve set a billing limit on it and you’re going to hit that billing limit, and actually lock out all of your accounts for all time, which is the Azure approach?
We have managed to, it’s caused three-week outages in systems by triggering inbuilt controls on platform-as-a-service, not because we intended to obviously, but you don’t want to live in an industry where three weeks outage is a thing that you could actually happen, that shouldn’t be happening to you, right? How many of you think your boss is going to be happy if three weeks later your code is still not running? Nobody is going to be happy.
Orchestration layer attacks
We started breaking down our services, we’ve decided we don’t want to put authentication, authorization in each of them. We are going to outsource out this to an orchestration layer. Fantastic, there are great ones out there and many available, I’m not here to shill for a vendor, I don’t care which one you choose. Choose one that has nice stickers as a starting point, I think that’s how it works.
One component to rule them all?
Great, you are going to have a really powerful component and lots of little tiny components, awesome. How many of you feel slightly worried about this? You should, this is worrying because claims like this happen. This was taken from a vendor website this morning. They are promising not only to do your orchestration but protect you from all of the OWASP Top 10 and adaptive threats such as bad bots. I don’t know what bad bots are and I’m a security person. If you can figure that one out, please do let me know. I’d be interested to know what that actually means.
[25:25] There is a worry here. What we’ve done is we said, “Ah, the security thing, I I don’t really need to worry about that because my orchestration layer is going to do it for me.” Yes. That is really dangerous. That creates an approach in our development teams of, “Not my problem, somebody else can deal with that bit. We just do our core pure functionality.” This is rubbish. At some point you are going to change your orchestration layer, or your orchestration layer is going to let you down in some way, and you are going to be really, really in trouble at this point. If it looks like magic, it’s probably bullshit.
Attackers like simple
Other things you should know, people like me attackers, people who do what I do but don’t look like me, we are incredibly lazy, incredibly lazy. If it’s a choice between doing some Ocean’s Eleven kind of heist that’s going to take three weeks of planning and I need a [inaudible] person and somebody who is really good at talking to humans, or I can just steal your laptop in the lobby, I’m going to steal your laptop.
Now same goes for orchestration layer attacks. If you’ve got 300 microservices, but I know your orchestration layer is the gateway to every single one of them, I’m not going after your microservices, I’m going after your orchestration layer. Now for many of us, that scenario we don’t even control. That might be a third-party service, a software-as-a-service product. It could be something we have internally in our network, but for most of us it’s somebody else entirely.
Features that scare me
Now these are the features I’m starting to see in orchestration layers that scare me. Impersonation, somebody in the support team for orchestration service number three can pretend to be me at any time and show up in the logs as me. Now that’s really scary. I want to be able to see if an admin or super-admin logs in, I want to see what they do specifically and it shouldn’t ever be disguised as me.
We are seeing investigation mode or SSL interception. SSL is a pain in the ass to configure at the best of times. If you haven’t run your tools through something like SSL scan or use SSL Labs to test what your SSL status is, then please go do that because you’ll start to get a feeling for how complex something like SSL is to configure and maintain day-to-day.
[27:45] I do not want some other vendor on the internet meddling with my SSL traffic. That kind of breaks the whole point of it being encrypted. Now if you’re putting any services into your network, say, “Oh, it’s all right, we man-in-the-middle the SSL traffic and then we rebuild it and we send it on its way.” Always ask yourself, who is looking at that point? Who actually can see that data? You’ll see why that’s important in just a second.
Password sins of the third-party provider. I am unlucky enough to have done so many SaaS reviews of security in the last few months that I can probably do them in my sleep. How many of you run or work for a SaaS product? Three or four of you. Please I’m asking you, please just once I’m going to ask you nicely as ambassadors for your industry, please get better at passwords, please start caring about things like two-factor authentication, please start caring about having good password lengths, do not have a max length on a password, it’s not a good idea.
How do we do this? We need to choose appropriate technologies. It’s okay to say no to new hipster unicorn technologies because they are not ready yet. It’s hard, you won’t feel like a cool kid but it’s okay. Restrict access to these things. Your orchestration layer is the most powerful thing in your architecture at this point, so not everyone needs access to it.
Monitor it aggressively and configure it well. If it doesn’t come with any, “Awesome, I will work with your monitoring system”, here is a little of a tip for you. If it’s got an API, query the API, just use whatever scripting language you like and use AWS Lambda. Ship the logs to whatever monitoring system everything else goes into. You can be up and running in less than 40 minutes as long as you know your script works okay and Lambda honors your versioning, which doesn’t always happen, but you’re going to have actual monitoring that you can actually interact with in the same place as everything goes. Don’t assume anything, don’t assume just because their marketing website is super slick and has all the right buzzwords in it and it means anything. I don’t give a rat’s if you say you are a PCI compliant, that you have [inaudible - socks?] written all over the website, I really don’t.
What I care about is that you care about my security, that you are going to take steps on your behalf and then are going to roll out to all of your staff, not just one person in your team. Then I want you need to test these people regularly. I want you to make sure that they are getting pen tested. If you are you working on applications that’s going onto the internet that isn’t getting penetration tested or being put in a bug bounty program or having a responsible disclosure process, you are doing it wrong.
The golden rule
[30:26] Now in this space there is a weird misconception that security vendors know what they’re doing when it comes to secure development. Now I want to ask you all as better engineers than anyone in my community will ever be, please challenge us on this. I want to show you just the last two weeks from one of our researchers in the community. This is Tavis, Tavis has a problem with security vendors. He has been systematically doing vulnerability research over every antivirus and security product he can find, and it is raining vulnerabilities.
Now this was last week, this is the week before that. It goes on and on and on. Now these are, we are talking about Symantec, Norton, Kaspersky. I’m going to get in trouble with someone because this is on the internet, but don’t assume if it’s a security product they know what they’re doing. Challenge them, push back, push back harder than you would on yourself because they have a greater duty of care than you do.
Also if you have a Fitbit set of scales, you should probably patch it, which I think is hilarious that we live in a world where you are patching your scales, but please do that.
Identity and access management
We’ve got our orchestration layer, it’s going to help with that. We hopefully haven’t decided to homebrew any of this, we are going to use some nice practices. Things we need care about, principle of least privilege. You work for an application development firm or on an application that has a God mode or a super-user or a super-admin mode. Most applications have it somewhere hidden in the backend. It’s the backdoor route that you get in and you do raw database queries and edits. It’s the one where it’s all gone horribly, horribly wrong and there is still a way to get in. I know about these things, I’ve seen them, I’ve written them.
Principle of least privilege
Your sins and my sins, we are in this together. We have to get used to the idea, it’s okay to say no. That it’s okay to say, actually you only need this permission. That’s going to mean that your roles available might get more diverse. You might end up with more of them. That’s okay. You don’t have to just have one role that can do all of the things, but to do this you need to define your roles well. What are the people actually using your application for?
Now how many of you have worked with marketing teams and have analytics tools in your application? Hopefully lots of you because that’s how we know what our customers are actually using. Any code that we are not using with the customer is dead code and needs to be deleted, which hurts us because we put our time and money and love into it, but still delete the code. It’s the best thing to do.
Now use those analytics tools, I’m going to be a security person that tells you to go use crazy marketing tools to figure out what people are using your application because then you can do set theory because maths is fun and look at what roles exist. What groupings actually are being used, what functionality means stuff to different users?
Roles vs. Fine Grain Permissions
[33:15] I’m going to ask you categorically please do not go, it’s okay we’ve got fine-grained permissions. Fine-grained permissions is the pathway to dragons, nothing good comes from this. I work with a client, they have 987 different fine-grained permissions in their application, which means if you talk about the possible permutations that make up roles there, you are talking in the tens of thousands of the permutations of roles in that application.
You don’t need this, I’m going to say it, engineers were really good at this but the users don’t care. These will be happy if you just make their life simple. Don’t go down the road of fine-grained permissions, always calculate how many permutations does that mean because every permutation needs monitoring and love in its own way.
Then audit the heck out of this. You really, really do need so much logging and monitoring your application, it’s unbelievable. I commonly see people using SumoLogic and [Raygun] from the vendor hall doing great stuff with their application logic, but they ignore the security stuff around the outside. Access logs. If you are integrating with something like active directory, all of these things need to be pulled together.
How many of you have ever been near a web application firewall? A couple of you, I’m sorry. Sad realities, web application firewalls are normally the primary defense we have in our organizations against bad things happening. However, they are given to the security team which is someone like me or someone who is a sysadmin and they go protect it, so we put it into monitor mode. You put it into monitor mode and it observes things and it goes, “If I was in real mode, I’d be doing stuff right now and here is what I’d be doing.”
Now in a perfect world, we would then go and look at these logs and we go, “All right, yes, that one can go on and that one doesn’t, and that one is a false positive and that one is okay and we turn it onto full mode and it’ll be fine.” Eight out of ten of the companies I’ve worked with in the last 12 months who have web application firewalls are still in monitoring mode 12 months after deployment. They are generating in excess of 67,000 alerts a day that are never getting looked up and processed.
[35:25] Why is this happening? Well, because your security team don’t know your code base. They don’t know what’s normal and what’s not. They don’t know what super awesome microservice number three is supposed to look like. They don’t know what crazy parameters you’ve set are okay, so then it looks like JSON being fired around here, there and everywhere, hopefully and not XML. Please if you are in an environment that is outsourced all of its security monitoring to another team that knows nothing about your application, go and ask them and give them help. Go and pull those logs to where you can see it because the engineers are much more likely to able to use this than anyone else.
Cloud Platform as a Service may* make you more secure
Mature groups and role assistance
Now just to reiterate what I said about AWS, Azure, any other cloud platform as a service, these are awesome that the number of controls you are going to get in this space compared to running around an infrastructure is great. However, this is a complex space, this is not you want to just dive headfirst into. Please consult a grown up before trying to configure security groups and roles and test them in any of these things.
It’s really tricky to make sure you’ve understood all of the complexities of the different components that are coming together. There was an interesting article recently calling Amazon’s offering more like having blocks that you can string together rather than tools. I think that’s a fair assessment. They are not things that you can just intuitively use on their own. You need to know about 13 different components all string together in different ways where their documentation is written in a different style, where you may not know all of the things that you need to configure in advance. Please don’t think the cloud is going to save you, go in with the knowledge you are going to have to do some learning or find someone who has already done that for you.
How many of us are playing with the Vagrants and Dockers and all of that kind of stuff in the world? That’s awesome, it’s really, really great, and I’m going to tell you why. I’m going to tell you why it’s great as a defender, why it sucks as an attacker that you are doing that. Please keep doing it.
Auditable host configurations are a good thing
First things first, we all probably live in some form of compliance environment, and that sucks. I really don’t like compliance very much. It can be a waste of energy and it distracts us and it gives us a certificate so we feel like we’ve done stuff. That’s how we stay in business, so we are going to accept that. Things you are going to like about the space is they are going to be auditable. There are nice open source tools like that that can show you where the security issues are with your configurations, especially for things like Docker which are very fast evolving. Just in the last week Docker released security review for its containers that are held up in the registry, so this is really, really good. There is a Docker audit tool that has been around awhile, there are similar things as well for Vagrant.
Avoids configuration creep
The most important bit, not the audit-ability, its configuration creep that it’s getting rid of because it turns out you’re not the only one that’s turning your survey into a special snowflake, you might be doing it, but in reality it’s actually an attacker who is going to do the most interesting changes. Immutable architectures get rid of this for you. I want to give you a bit of perspective as to what actually happens when we compromise your host because for most of us this is just fuzzy black cloud of doom.
Contrary to popular belief we don’t just go in there, ransack a bit and steal the data and get out. Most of the time that’s actually number three or four on our list. First thing we do, make sure we can come back and play another day because everyone likes to be able to come back and get the stuff you forgot last time or come back outside of working hours or you might want to use that host for other things like mining Bitcoins or sending up to pivot into another network. Attacking someone from someone you’ve already attacked is super-useful.
You are going to install tools there, so it’s very rare that all of the tools I need in a job are installed already on the box. If you have PowerShell, it can pretty much do a lot, but Linux hosts hopefully if they are configured well don’t have a lot installed on them, so I might need to put SSH on and I might need to put endmap on, all sorts of fun things. Nothing particularly exciting, they are not crazy magic wand tools but I want to see them.
Next thing, I’m probably going to clean up anyone else who has been there before me. Now as an attacker we are parasitic, we need the host to be healthy to continue. If we get into a box, and we find someone who was already there, then we might actually patch the vulnerability that they got in via and kick them out because I don’t like sharing. Sharing isn’t that much fun. We’ll clean up, we might patch, we might change if it is a configuration, lock someone else out and then we are happy going about our business finding data.
Our next thing, last thing, we are going to trying and pivot. We are try and go from this host, which might be interesting to this one over here which might be even more interesting. In your microservices this means going from one to another to another to see if I can find different datasets, accesses to different systems.
Now it’s really, really hard to persist access and to keep all of these nice snowflake changes going as an attacker if you’re using something like Lambda, or if you’re using immutable architectures that are destroyed every 24 hours because you’ve done a tweak or a change. Don’t just see them as a cool thing that you could be using because it’ll save you time, there is actually a massive security benefits in this. There is nothing more irritating than trying to attack on an organization that’s infrastructure does not exist unless you are sending a query to it. It’s a beautiful thing. Please I want you to infuriate all of the penetration testers that ever come near your architecture and make it non-persistent.
Heterogeneous language and technology spaces
All right, that’s a fancy word for saying, we’ve been doing some silly things with languages. One of the beauties of microservices is that we can use any language we like. That’s great, awesome, fabulous, but we are like kids in candy shops right now. That means people are going nuts, going, “I’ve got some node over here and then I watch this talk about Go, we’ve got a bit of Java over there because rocking the old school, and we decided to fork, grab it and queue so there’s an Erlang stack somewhere over here.” Before you know bad things start to happen.
Choose the right tools
Bad things are starting to happen because we are not always using the right tools for the right jobs. Not all of these tools are as mature as each other, and not all of them are designed for certain purposes. If you find yourself using a language just because you watched a cool talk about it or because you saw a really whizzy demo, always ask, “Is it the right one?” There was a really interesting article from [inaudible] talking about how they hire engineers and he was talking about, actually they don’t want to bring more tools into their architecture. A good engineer can take a small subset of pre-vetted tools and do amazing things with it.
Now from a security perspective, this is even more important because not all of our technologies are equal when it comes to what they’re providing you. Now how many of you do Node, any Node developers in? No. That’s really good actually, please yeah, awesome. I’m sorry, node people, but I’m talking about Leftpad, I’m talking about the things we’ve learned in the last few months about interdependency inside our package management systems.
Leftpad was taken out. The developers said, “No, no more.” What happened? We expected it to just silently go into the night and refine. What happened is most of the Node failed, it’s because node MPM is a highly connected, highly dependent package system. Most of the packages are less than 20 lines of code in length. What we’ve got is a cascading failure situation where you can put code in or take code out and it breaks everything around it.
[43:10] This is not the same for all of the languages. When we have so much variety in our language space, we are opening ourselves to risk from the package management systems themselves. Other interesting things about node, you don’t actually need to do any validation or have your code checked to have it in MPM. You sign-up, you create an account, you push, gone, done.
Now if you are someone like me, what we see is the ability to then push malicious code in and have it adopted by many, many, many other dependencies and get into this highly connected mesh of packages. Now this is basically the same way that worms work. I could put malicious code in, have the three things become dependent on it. Each of those have ten things that are dependent on those. Before you know it I’m touching every piece of the MPM infrastructure or the ecosystem. That’s scary.
How many of you have a good process for vetting your packages you are pulling from these places? I’m not seeing anything in the wild yet that actually you could use this and vet whether these packages are safe or not safe. We are seeing CVE checkers and things, but we are not seeing anything that goes, “All right, do I actually need this? Is this a good code? Is this sensible engineering?”
Now to end this little section on some horrifically trite analogies, think of it like puppies. Every language and every package you bring into your system is like a puppy. It’s cute, you get to play with it, do awesome cool things you’ve not done before and that’s awesome. It’s going to poop occasionally, and that’s okay because it’s a puppy and you just scoop it up and you deal with it. Great. Then you get 300 puppies, now suddenly that poop problem seems a lot bigger problem than it was previously. Please before you start bringing in more puppies into your life, consider what you are going to do with the poop.
I’ll wrap-up with detection. Prevention is not going to get you the whole way. Please log all the things, we’ve mentioned this previously. I’m going to spend this section how many things I’ve already talked about because it’s the most important. It takes up to 18 months to spot if you’ve been compromised and in that time all sorts of fun things will have happened. If you’ve only got 14 minutes of visibility in your logging, you are in for bad times when it comes to investigating what went wrong.
You have to store your log somewhere that’s immutable itself, that’s secure. It seems like commonsense, but what we see most of the times, it’s stored on hosts, it’s fine. Now, do you remember that thing where I told you what I’m going to do when I compromise your host, part of the cleanup is I’m going to flush myself from your logs. Your logs are going to be crisper and cleaner than they’ve ever been in their life. Your logs and poorly managed logs are actually a really good way to denial of service things. If you’re send them all to a poorly configured endpoint, that’s a really good way to take you down as if you take away that endpoint. Please be careful.
Another bit of homework, how many of you’ve had heard of Hacking Team? Hacking Team was a prolific group of vulnerability researchers out of Italy, and there was a report put on the internet very recently from the guy who compromised them and add all of their dirty laundry online. He compromised them and this is a very security focused company, by attacking their centralized middleware, so their logging systems, their backup systems, their systems health check, anyone is still using Nagios, this is where we are going to come after.
Please bear in mind when you’ve got this very fragmented infrastructure like this architecture, then we are going to look for anything that’s shared between all of your components. We are not afraid to go after things like Nagios. Most of these systems have really, really elevated privileges too. Then watch your logs.
How many of you filter the alerts that come into your inbox into a folder and eventually get to them? How many of you don’t even get the alerts because you unsubscribed? How many of you had the slack channel of doom where you’re logging goes to and one person goes in occasionally, but most the time it’s just noise and then occasional gifs. Yep, we are crap at this. Your biggest, the biggest thing you can do is to not worry about which hipster language you want to do next is get this stuff working.
[47:18] Automate the heck out of your logging and get it monitored. Investigate things and go find out. You’ll find more scaling of performance issues looking at security logs than you’re aware of at this point. I can find security flaws looking at slow query logs in MySQL. Your logs are super-valuable but we don’t look at them unless somebody tells us to look at them, or we’ve broken something.
It’s a bit of a whistlestop tour, I appreciate that. When I say demos of things, everyone expects me to go, “Laura, here is I popped a shell for you.” You are not going to learn that way, it’s a magic show. All of the examples I’ve given you are from real systems, from things in the wild and written by real engineers, not unicorns who write books and presented things.
Please go into your world, do simple things, keep things simple and keep them reviewed, keep them logged because it’s all very well moving from the big monolith that’s costing us a lot and it’s painful to deal with the lovely awesome microservices that we can be much more flexible with. Remember that as we break things in small pieces, the surface area increases. Surface area is all of the points and security that we can attack. If there are more points to attack, the more successful I’m going to be.
If you have any questions please let me know. Obviously there is an application that you’re supposed to give me a feedback scoring. I did not at any point tell you to try and give me the score of purple, so that never happened, just for the record. Thank you very much for your time.
About the speaker
With almost a decade of experience in software development and information security, Laura Bell specializes in bringing security survival skills, practices, and culture into fast paced organisations of every shape and size. An experienced conference speaker, trainer, and regular panel member, Laura has spoken at a range of events such as BlackHat USA, Velocity, OSCON, Kiwicon, Linux Conf AU, and Microsoft TechEd on the subjects of privacy, covert communications, agile security, and security mindset.
Laura is the founder of SafeStack, a specialist security training, development, and consultancy firm. She lives in Auckland, New Zealand with her husband and daughter.