Cybersecurity Effectiveness Podcast

back to podcasts
Listen on:
Subscribe:
stay up to date

Michael Allgeier

We don't get a whole lot of opportunities to really just start off with a contested environment simulated. The red team had full domain credentials, admin credentials, on the domain in the virtual environment. We started off really, really hampered and that's the way we train. We train how we fight, and we fight how we train. To do anything less is not a good way to go.

Michael Allgeier is the Director of Critical Infrastructure Security for The Electric Reliability Council of Texas (ERCOT).

Brian Contos:                

Welcome to the Cybersecurity Effectiveness Podcast, sponsored by Verodin. The Verodin SecurityInstrumentation platform is the only business platform for security that helps you manage, measure, improve, and communicate security effectiveness. I'm your host, Brian Contos, and we've got a very, very special guest today. Joining me is Michael Allgeier. Welcome to the podcast, Mike.

Michael Allgeier:          

Hey, thank you, Brian.

Brian Contos:                

Hey, Mike, before we get going, could you give us a little bit of background about the path you took in yourc areer and eventually what lead to you ending up at ERCOT?

Michael Allgeier:          

Yeah, sure, no problem. I joined the Army right after high school and, well, I needed to get off the farm. I worked in the intelligence community for about 12 years as an analyst at first, and then switched over to counterintelligence special agent primarily focused on investigations and operations involving the high-tech aspects of those investigations and operations. I moved back down to Texas after leaving the service and spent about 10 years in the utility industry. Along the way, I rejoined in the Army, but in the National Guard. Then had a short stint of working another industry before joining ERCOT a couple of years ago.

Brian Contos:                

Very cool. Well, first off, thank you for your service.

Michael Allgeier:          

Thank you.

Brian Contos:                

We had met, I think, about three or four months ago, I believe, down in Texas. We had some drinks and had dinner. You were sharing some of your stories about what you had been doing at ERCOT, and I thought it was really, really fascinating. Before we actually get into some of that, maybe give us a little bit of background on what ERCOT is and what they do. By the way, for our listeners, that's E-R-C-O-T, but what's ERCOT all about?

Michael Allgeier:          

Yeah, so the Electric Reliability Council of Texas manages the flow of electricity for more than 25 million Texas customers, representing just about 90% of the state's electric load. As the independent system operator for the region, ERCOT schedules power on the electric grid that connects more than 46,000 miles of transmission lines, and about 650 generation units. Those are like power plants and solar farms and wind farms. It also performs financial settlement for the competitive wholesale bulk power market and administers retail switching for about 7 million premises in competitive choice areas. That's like in Texas, if you're not in a muni or a co-op, you have the power to choose where you buy your electricity from in Texas, and, see, we manage all that. ERCOT's kind of a different animal. We're a membership-based 501(c)(4) non-profit corporation, governed by the Board of Directors and we're subject to oversight by the Public Utility Commission of Texas and the Texas legislator, so there's all that.

Michael Allgeier:          

There [are] really three grids that service the United States. The independent system operator, which is ERCOT, one of them, we're kind of like the air traffic controller for an airport. You have planes and the runway strips and the pilots and the crew. We don't own the airport or landing, the runways. We don't own the planes, and we don't manage the pilots, but we ensure that things run smoothly.

Brian Contos:                

That's awesome. Sounds like a big job, actually. I didn't realize it was that massive, but then you walkthrough and it's like, "Holy cow."

Michael Allgeier:          

Yeah, Texas is kind of a big...

Brian Contos:                

Kind of a big place.

Michael Allgeier:          

Yeah, it's almost a country, right?

Brian Contos:                

Awesome.

Michael Allgeier:          

It used to be.

Brian Contos:                

Well, and when we were talking, I mentioned earlier we had met face-to-face for a bit. We were talking about GridEx, which is getting a lot of buzz right now. I was hoping maybe you could give folks a little bit of background. What is GridEx and more to the point, why should people care?

Michael Allgeier:          

Yeah, Grid Exercise. It started back in 2011, and it's a bi-annual exercise for physical and cybersecurity. It's managed by the E-ISAC, which is the electric ISAC. There's other ISACs out there that provide information sharing and analysis, like the multi-state ISAC, the financial ISAC, and others. The electric grid, we have our own ISAC that we share information back and forth with, and that's sometimes our conduit to the federal government. When they say, "Hey, look out for this new pack,"or something like that, that's often how we get it. It's an exercise that happens every other year. It also includes many federal and state and local government agencies and other critical infrastructures.

Brian Contos:                

Who's it help? Who really gets value from this kind of exercise?

Michael Allgeier:          

Yeah, so, obviously, it's the electric companies, but it also helps the information sharing coordination between federal, state, and local government entities, first responders, and for other... maybe fusion centers that you might have in the state or in a county. It's really ... we'll get into this maybe later on, but it's growing. It used to only be tabletop exercises within the power company. You have your local power company and they may kind of dust off the instant response plans and playbooks and things and kind of go through it. Well, it's gotten quite a bit more mature for a lot of companies where just last GridEx, GridEx IV... which was a year and a half ago, in November.

Michael Allgeier:          

We really set up a whole virtual environment which mimicked our corporate and also some other sensitive systems. We hired a company to act as the red team guys and we had our defenders. We really actually had live play that followed the script of the overall exercise. I tell you, usually the FBI and others that show up the first day and they kind of leave after a few hours because it's a tabletop exercise, and we've all been through a bunch of those and they kind of leave. They stay the two days of the exercise, the whole time really engaged and kind of walking back from looking at it with the blue team that the defenders are doing. Then walking over to the other room where the red team, the red cells are trying to hack into the systems and going back and forth.

Michael Allgeier:          

I guess it was entertaining for 'em, but it was actually really, really important that we had that type of interaction because we're able to kind of do an administration halt towards the end. It kind of walked through the red team and blue team together, kind of like a purple team of, "Okay, so when this happened I was doing this hack." They're like, "Oh, well I was defending it this way" or "I found out this." They're able to really understand how the attackers look and understand how that looks like on the network and their tools.

Brian Contos:                

Yeah, I love that idea, especially... well, first off, just going beyond tabletop exercise to actually bring it up so you're getting red teams and blue teams involved. You can look at an attack through the lens of the red team, but also seeing what that looks like in your environment through the lens of the blue team. That whole purple team notion, I really think that's where everyone's going and hopefully this continues to expand as you mentioned earlier. I'm wondering, do you have some interesting examples or some lessons learned, maybe from the offensive side, things that you've gleaned from the red team activities?

Michael Allgeier:          

Yeah, so there's a couple things specific to our exercise that occurs all... I'll preface with, we've corrected these things. This is a great opportunity to actually truly practice your plans. It's okay to find fault. If you just run through it all perfect, whatever then, if you're not really pushing all your employees and the plans to the limit, then you're setting yourself up for danger later on. Like I said, we did fix these, but we have, of course, multi-factor authentication for remote access. We were using some of our live corporate system, so we're talking email and things like that, nothing sensitive, in combination with our lastGridEx.

Michael Allgeier:          

And so, we allowed the red team to start walking out corporate accounts remotely from the Internet. They did it because they're just bad passwords and user ID and bad passwords. After so many tries, they get locked out and then the help desk is trying to unlock them, this and that. We found a setting that we fixed it to where it'll lock out the multi-factor and not the main credentials. That's something that ... A lot of people have multi-factor authentication, but they need to verify the settings to make sure that it's going to lock out a token or that other factor, before it starts locking out your domain creds.

Brian Contos:                

It's just such a great real-life example because that's not going to be on the top of anybody's security list to validate, right?

Michael Allgeier:          

No, it surprised us. We were like,"Whoops." The cool thing is, literally they fixed it that day. I mean, that's... we just did an emergency control change and put it through everything and made the change right then and there. We're like, "Wow. Good thing we did. This is pretty cool."

Brian Contos:                

Yeah, yeah. To me, I'm still amazed when people go through a validation process of their security tools to see what's working, what's not. What would happen in this scenario in here? There's always, "Oh my gosh." Because it's not because you have bad people or bad tools, there's a lot of stuff. There's a lot of layers of complexity, and a lot of dependency. It's just the way complex solutions work.

Michael Allgeier:          

Yeah. It's literally sometimes "click this checkbox."

Brian Contos:                

That's right, that's right. Sometimes it's a five-dollar fix to address a million-dollar problem, right? Well let's flip that around. We talked a little bit about sort of the red teamers there. How about some cool lessons learned or examples from the blue team side, the defensive side?

Michael Allgeier:          

Yeah, so on the defense side, we also found one cool little tidbit for application whitelisting. We have some EDR and some response tools, and also some protection tools on systems. The blue team found the hack and saw the services. They threw it in there to go ahead and block this, and there were all high fives all around. All right, we found it and we blocked it, we're good. That service is still running and when you put in for a block, it's for a future. It hadn't learned that this thing was bad yet. It's just, again, we're kind of like--which is nice--the Maytag repairmen where usually things don't break, which I hope is okay.

Michael Allgeier:          

We don't get a whole lot of opportunity to really just start off with a contested environment simulated. The red team had full domain credentials, admin credentials, on the domain in the virtual environment. We started off really, really hampered and that's the way we train. We train how we fight, and we fight how we train. To do anything less is not a good way to go.

Brian Contos:                

It's funny, of course, I work with a lot of people like yourself with a military background. This fight like we train analogy that's applied to cyber right now, I just think is just right on point because I don't think there's another way to address this. We have to fight like we train, right?

Michael Allgeier:          

It's now, for sure.

Brian Contos:                

GridEx has been around for a little while. I think you said the first one was around 2011. It's this bi-annual event. How do you see it evolving over the coming years? Is it making further strides? Are more organizations getting involved?

Michael Allgeier:          

Yeah, so just as far as the quality of exercise, the scenarios that we all base off of is very good. Across the United States and Canada and Mexico, we all have this exercise over two days at the exact same time. We coordinate with each other and report things up, so we practice all that which is good. There's a lot of companies that have started actually doing some live play or mixing in... like, we kicked off last GridEx with a phishing scam sent to the entire company. Just to set the tone of, "Wow. Okay, we're really doing this and seeing how it works." That was all tied into the overall theme of exercise and it's really moving to where a lot of other companies are installing, end up getting some virtualized environments to actually train how you fight and fight how you train.

Michael Allgeier:          

There's that aspect, but there's also another aspect of... we're including a lot more other critical infrastructure, just not power plants and transmission companies and things like that and independent system operators like us. It's also good for government entities to partner and get some time with the other critical infrastructures out there. We had a really good discussion last GridEx with the local fire department about projected cloud plumes from hazards in the county and things like that. That spawned a really good conversation between the County Emergency Operation Center Manager and the Fire Chief. That kicked off a whole new collaborative information sharing aspect right there.

Michael Allgeier:          

Also, when you have your local first responders in your building, give them a tour. Invite them, "Hey, what would you guys do in here if there was an active shooter scenario?" It's just not even cyber and it's everything. You really start to know who your local first responders are, and then what about your state emergency operations center? There's all these other... there's the local national guard, they can help out. We're expanding to a lot of these other areas, and really building a lot of collaboration between us.

Brian Contos:                

Yeah, it sounds almost like a fusion center, right, and you're bridging cyber and non-cyber groups and issues like that. A lot of moving parts, but for what you do, I don't think there's any other way to truly have a holistic solution unless you start bringing in all these groups, so that definitely sounds like a great evolution.

Michael Allgeier:          

Oh, for sure. Don't forget the other aspects of the company. You have human resources, you have corporate communications, you have facilities, you have all these other departments that have a big role when there is a significant synchronized attach of cyber and physical on some of our critical infrastructure. There's messaging, there's coordination with state and local government, there's politicians and it just goes on and on. The messaging of social media is important because we also included social media as basically an attack vector to send incorrect information. Our corporate cons are having to go and find out really what was going on and then kind of respond to calm people's fears. Of course, this is all within the exercise. We kind of threw everything at it, the kitchen sink and all of it. We've planned the spring, we're going to do that this time, too.

Brian Contos:                

Well, it makes me so happy to know that there's people like you and organizations like this that really push the envelope. At the end of the day, this is the type of thing that is the core to society, right? Without the grid, a lot of things collapse. As we wrap up here, I've got one question I'd like to ask. This is something we ask for everybody on the show and that's, who's your favorite superhero or super villain and why?

Michael Allgeier:          

Yeah, so superhero, Superman. Why? I don't know. Check off all the right things, right? First off, he can fly. All right. I mean, he's also really strong and he can go into outer space and not have to worry about having a mask on or something like that. He's impervious to bullets and a bunch of other stuff. More importantly, he's also an alien, so that's kind of cool, too.

Brian Contos:                

Right, right.

Michael Allgeier:          

He seemed kind of like all the boxes were checked, I think, for Superman.

Brian Contos:                

I'll tell you what, for a superhero that was created in 1938, pretty good stand power for somebody today to say it's still their favorite. That's awesome. Well, Mike, thanks so much for joining today. For everyone else, please be sure to check our other episodes of the Cybersecurity Effectiveness Podcast, sponsored by Verodin.

download transcript (PDF)
back to podcasts
Follow:
Subscribe:
join the list
X
Business Need
technology
company
resources
blog