Security Confidential S5 E1 Charles Herring

Manoj Tandon:
Hello everyone, and welcome to another episode of Dark Rhino Security Confidential. I’m your host, Manoj Tandon, and today we are honored to have as our guest Charles Herring from Witfoo. We’re really looking forward to this conversation.

Charles has a great background and a lot of experience in InfoSec. His dedication to maturing the craft of InfoSec is built on a diverse career path across the industry. He started his career in InfoSec in the U.S. Navy back in 2002, serving as the Network Security Officer at the U.S. Naval Postgraduate School. He was a contributing product reviewer for InfoWorld magazine and spent seven years running Herring Consulting, a company dedicated to process orchestration.

He dedicated his career to process orchestration and did a lot of work with veterans, law enforcement, and government officials. He used that experience and his learnings to form Witfoo, which is Charles’s organization today.

We’re honored to have you, Charles.

Charles Herring:
Thanks for having me. I appreciate it. Glad to be here. Happy to spend the time with you.

Manoj Tandon:
I’m glad that you’re here as well.

And another little bit: we’re going to talk about information security, but we’re also going to talk about diving, because we haven’t had a real diver on this show. We’ve had yoga coaches, laughter coaches, security architects, and a lot of CISOs, but you’re the first diver.

Charles Herring:
So I’m pretty good at it. My number of ascents matches my number of descents, which tells you I’m pretty good.

Manoj Tandon:
Excellent, excellent. So, Charles, let’s start this discussion off at the beginning. Our listeners are always curious about how people got their start in cybersecurity. For us, this is a two-part question.

We’d like to understand how you got started, but also your transition from the military to civilian life. That’s a topic that we get a lot of questions on, and we’d love to get your insights as a veteran on that, and perhaps help other veterans become very successful.

Charles Herring:
Yeah, that’s a great question. It was really stressful for me. Coming out of the military, you have a lot of great things: a great support system, a focused mission, leaders who have grown up being taught regularly how to be better leaders, a set of core values, and a team around you that cares about all aspects of your life—making sure you’re good at work, safe at work, taken care of, and willing to put in with you. They’re willing to own things with you.

Leaving that and coming into the corporate world is a whole different animal. That support structure is gone. We tend to live in this dichotomy of work life and home life, and so we have these casual relationships with our coworkers instead of the deeper relationships that we shared with our fellow servicemembers in the military.

It was a little hard for me. The first thing was trying to figure out how to get the support system in place. Who do I talk to? Where do I go? Who can commiserate with me? Where can I bounce ideas off of? How do I get better?

One of the great things that virtually all veterans have is adaptability—the ability to understand the new theater they’re in and adapt to it. But it’s challenging without the support system. Frankly, it’s terrifying.

So the first thing was getting involved in some veteran organizations like the American Legion and VFW, to have some veterans to bounce ideas off of and have that level of support. On the personal level, you have to be grounded there when you come home and have some help.

Then really, adaptability is a whole different thing. The way organizations tend to run in general are loose missions. The mission can change. The interpretation of the mission can change. Values, if they’re defined, aren’t necessarily widely adopted in the organization.

So being patient is another thing we learned in the service. We stand a lot of watch, and so staring and waiting is something we really have to lean into because things don’t happen very fast and there’s a lot of politics.

Relearning how to interact with people who are not so in lockstep was a big challenge. Talking to other veterans who had been through that helped—just on the small details like, “I’m having a problem with so-and-so at work. I think they hate me. Am I doing this wrong? Am I pushing too hard?”

A lot of it is interpersonal skills. Going through the hiring process is tough because veterans want to charge the hill—we want to solve the problem. Sometimes corporations don’t want to run that fast, or there are emotions and politics that we’re generally not accustomed to.

Definitely having your new “command,” with a PCS or permanent change of station inside the military, and realizing that you’re effectively PCSing into a new command outside of the military—you need to get the support structure set up.

But I will say, in cybersecurity, every problem you have in transition is worse because it’s an immature craft. We have problems defining what the real outcomes should be, and we can’t agree on that. We have problems communicating amongst ourselves inside the different parts of cybersecurity. We have a huge problem communicating with the rest of the business.

So it can be a very frustrating craft to be in in general. Learning how to be useful and patient while helping navigate what is currently an immature craft and helping it mature is daunting.

Fortunately, most of my team are veterans, and they’re fantastic. They’re adaptable, they’re able to change, and when I talk to employers who don’t get it and are having struggles understanding the value of a veteran, I say: just give them some room. Let them run with it. Give them the ability to make some transformation. They’ll deliver.

Organizations that want to change should do that, and I think it’s also important for veterans to connect and network with each other about which companies are mission-focused and have a culture around values, because you don’t want to jump right into some dog-eat-dog scenario.

The main things for me were: make sure you get a support group up. We have the same problem when we leave the military; we often go back to our hometowns where there’s not necessarily a base or the infrastructure. So finding like-minded veterans is a huge help just to bounce ideas off of. The support system is probably the most important piece, because there is going to be change. It’s different than any other change of command that we’ve had before.

Manoj Tandon:
What is a little surprising to me is that there’s nothing in military operations that’s static, especially for those who have served on the front line in the field. The situation is continually evolving. It’s changing. You have to adapt and make decisions on split-second notice.

I would think coming into the civilian world it would be a heck of a lot easier. Those decisions being made in field operations are very difficult decisions. On the civilian side of things, I would think it would be easy as pie.

Charles Herring:
Most of the things we’re doing in the military are very empirical, concrete objectives—things you can see, things you can interact with. They’re dangerous or volatile, and the consequences are challenging, so they’re empirically more stressful. But the second thing is, we have well-defined rules and operations.

When I enlisted in the Navy, my craft was aviation electronics technician. I fixed weapons and electronic systems on F-18 Hornets. I thought I was going to go in and it was me, my big brain, against this billion-dollar aircraft, troubleshooting it.

But after I got through training, I found out: you push a button in the wheel well, a maintenance code comes out, I look that maintenance code up in a publication, and I work a flowchart. When I’m done, the plane’s fixed. Then QA happens, we test the aircraft, and turn it back around.

Even though it’s very expensive and complicated, and often done on a flight deck in the Persian Gulf or somewhere else, the structure—how we do things, what’s acceptable, the steps we take, how we QA it—there’s so much structure.

After 9/11, when I was detailed to the Naval Postgraduate School to spin up the Network Security Group, that was my big “aha” moment that not everything is that way. It was just after 9/11, about 20 years ago now, and the command had to figure out how to do cybersecurity. There were no publications, no flowcharts, no training, no professional development. It really was: whatever I thought was right was right. It was disorienting.

Manoj Tandon:
So there was no playbook for it, right? What you’re describing is that there is a playbook on an F-18. But cyber—even when we have playbooks on security orchestration—they evolve based on the situation. I think we’re going to get into that a little bit.

But before we get there, do you have any message for employers that might have a bias or any kind of difficulty in considering a veteran for a position in their organization?

Charles Herring:
In general, especially in cybersecurity, we need diversity—particularly occupational diversity. A lot of the problems we have in the craft are because we’re doing the same thing the same way over and over again and nothing’s getting better. We’re just sort of resigned to the fact that things are going to get worse and worse, which is ridiculous.

One of the most important pools of occupational diversity is military veterans. They’re adaptable, they understand outcomes, they understand the investigative mindset, the troubleshooting mindset of solving problems, and understanding rules of engagement.

What I would say is if an organization wants to get healthy, they need to onboard people who have the ability to solve for the outcomes of cybersecurity. Veterans are going to be in that mix. I don’t think it’s possible to build out a security practice that’s going to create deterrence against cybercriminals, secure the network, and meet risk requirements without the occupational diversity that comes with military veterans.

First align your outcome to wanting success, and then as soon as you do that, you’re going to realize you can’t possibly deliver that without the occupational diversity that military veterans bring.

Manoj Tandon:
I think that’s an excellent point, and I hope people listening take note of that.

Now getting back to the playbook: you started off in the Navy and got into cybersecurity, and there was no playbook. Even today, people think there are finite steps one, two, and three, and not everybody can agree on what those steps should be or how they should be done or communicated.

When you kicked this off in the Navy, I take it this program was in its infancy?

Charles Herring:
Yeah, absolutely. The challenge for most of my life over the last 20 years has been about studying the craft of cybersecurity. Crafts have very concrete things.

Take naval aviation. The manuals that drive naval aviation—the ones I learned from—are called NATOPS. We say they are written in blood because of the criticality. Errors meant losing airplanes and losing lives. So the need to get it right—to figure out what process is safe, repeatable, and trainable—happened in the first hundred years of building that craft, really the first 50 years of building out naval aviation.

We don’t have that criticality in cybersecurity operations. We also don’t have that much time; we haven’t been 50 years into it.

I study what I call the seven unstable conversations in cybersecurity. The craft needs to communicate internally and externally. Data is a building block for a conversation, but to have a conversation you need to understand the audience, the intent, and the outcome.

The first part is establishing a unit of work. In sales, the unit is an opportunity. In Uber, it’s a trip. In a restaurant, it’s a ticket or table. In aviation, I had a maintenance action form.

The fundamental challenge I had when spinning up the Network Security Group was that I had seven sailors working for me, we had budget and all these things, but I couldn’t figure out what the unit was. What is the work?

Much later on, I realized it had to be in the context of a crime: someone is trying to steal patient records, someone is trying to extort money via ransomware. Those words don’t just make sense to cybersecurity people; they make sense to every board member, whether they’re a CISSP or not.

There was this pivot. The reason I was there, and the funding, didn’t come out of IT; it came out of national security. So the cybersecurity group was a security group focused on cyber. My boss was the Director of Security for the base. My peers weren’t just server people and IT people; it was the Chief of Police and the Intel Officer.

We carried sidearms in the cybersecurity group at the postgraduate school because it was security. Our outcome was: someone is trying to harm us, we need to stop them, we need to deter them.

But I also had IT meetings with peers where we were talking about patches and firewall blocks and packets—things that make no sense to most business leaders. I realized I was trying to use the same data one way in security and another way in IT. But when I’m talking to the Admiral, it was always the security language he understood.

It didn’t make sense to run it through an IT filter and reinterpret it back into business language. There is something innately understandable about security when we talk about it as crime instead of talking about it as risk management alone.

Manoj Tandon:
When you’re presenting to the Admiral, is the unit you’re talking about the number of crimes prevented or intrusions prevented? Was that the unit?

Charles Herring:
Yes. When we talk about crime, you have civilian crime, national security issues, and policy violations—students or faculty doing something in violation of command policy. Those are mostly true across every organization.

Someone is trying to commit a crime—steal data, disrupt services, or extort money. They’re doing a crime that lines up to some legal code that can be prosecuted.

That was the unit: how many attempts did we have? How many attempts did we stop? What’s the nature of the adversary? What can we do to further deter them? What can we do to shore up our defenses so they’re less likely to succeed?

Think about law enforcement. Police officers don’t normally get to stop crimes.

Manoj Tandon:
No, they’re reactive after the fact, right? A lot of cyber is like that too.

Charles Herring:
That’s a challenge. One of the main outcomes we should be driving toward in cybersecurity is increasing deterrence for criminals.

One thing that’s becoming important is that the interaction, especially in the financial sector, with the FBI has gotten much better. They’re sharing information with law enforcement. They’ve figured out how to package it. They’re not looking at logs as data; they’re looking at logs as evidence.

So they’re collecting and packaging it in a way that lets them produce an affidavit to the FBI. The FBI recovered north of 70% of wire fraud. The financial sector has figured out that the code of silence doesn’t lead to a safer world. So the wire crimes are starting to go down.

But in virtually every other vertical, we’re not calling the police, we’re not calling each other, and we’re not being good citizens because, frankly, we don’t know how yet.

Manoj Tandon:
I think it’s a major problem. A lot of groups have set up ISACs, which are intended as information-sharing warehouses, but you’re saying that’s certainly not enough.

Charles Herring:
Right. The collaborative market sharing is important. Vendors like me have to find ways of sharing information without bankrupting ourselves. There’s proprietary information we want to protect.

One of the things our community does is if we have customers using something like Palo Alto and Palo Alto is detecting a given set of IOCs, that gets anonymously submitted into our cloud service. The guys using Cisco ASAs that aren’t currently detecting those IOCs are at least logging that the connections are happening, and they can match against that threat feed.

Now the ASAs are able to benefit from the intelligence from Palo Alto because mutual customers are sharing that intelligence.

One of the things we learned in working with law enforcement is how to deal with anonymous tips. Sometimes too many tips are too much work. So how do you vet the quality of threat intelligence? One of the ways physical law enforcement does that is corroboration.

If multiple sources report the same thing, and especially if they are multiple types of sources, that provides corroboration of a fact and makes each one of them stronger.

The same thing applies in cyber. If CrowdStrike detects something and Carbon Black also detects it, the corroboration of detection methodologies plus the corroboration of submitting organizations gives you a confidence level that it is a legitimate tip.

Manoj Tandon:
Going into what we’re seeing as threats across our monitored platforms, one place I don’t see much information sharing is what the red teams are doing.

Charles Herring:
The couple of things that work really well in catching unknown unknowns are honeypots and deception technologies. If something starts talking to a box that’s not supposed to be there, there’s no false-positive scenario—someone’s doing something they shouldn’t be doing.

Then there’s the investigative mindset. You have a trigger, something happens, and then there’s an investigative process of going through it. This starts with developing a theory.

A homicide detective advisor of ours explained it well. If he drives to a house and there are 20 bullets in the wall and shell casings on the ground, he draws a big circle in his mind around the scene and starts with a theory. Is it a bank robbery? No, probably not—there’s no bank. Is it a carjacking? No car. Could be a drive-by shooting.

In cybersecurity, what we tend to do is pick up one shell casing and assign a guy to each one, trying to think through what each thing could mean. We start at the micro-level trying to find our way to a theory.

In law enforcement, you start with a theory, and as you follow the evidence, the evidence either proves the theory or disproves it.

Manoj Tandon:
So hypothesis-based threat hunting.

Charles Herring:
Exactly.

The process of detecting net-new zero-day stuff is always going to be a challenge. Whitelisting, process whitelisting, traffic whitelisting, Zero Trust—these are things that create a very small space for a zero-day exploit to get through.

Even when we’re working with red teams or automated attack simulation tools, it’s important for the blue team—the defenders and responders—to understand where they are and to synthesize actual attacks from simulations, because there are different response actions for those.

The bigger issue is getting away from triage. Because of the volume of signals we have, unlike physical law enforcement, I can’t come back later and set up detection after the crime happens. In digital, you have to have all the collection and processing set up before the event occurs.

That means we’re processing huge amounts of data. A lot of organizations I work with are dealing with terabytes up to petabytes a day of signals. So what we’ve had to do as a craft is triage which signals are important.

Because we’re triaging, we’re missing things. And a healthy craft doesn’t triage forever. Triage happens during crisis, but at some point the broken arm has to get fixed.

Computers can process every signal in the context of every theory. Then humans get involved where it matters—asking questions, doing digital forensics, interrogating the evidence.

Manoj Tandon:
You mentioned there are seven conversations. We’ve talked about one. Can you summarize the others?

Charles Herring:
Sure.

The first conversation is about building a unit of work that delivers clarity for the investigators, so the investigator can comprehend their work.

The second is that the managers of investigators need to be able to manage them. Do I have enough people? Do they have the right tools? How many people should I have? How do I professionally develop them? If you can’t look at the work of each contributor, you can’t audit it, help them grow, or correct them.

The third problem is that the security practice cannot properly communicate with the broader business. This is why CISOs often don’t end up at the executive table. Part of our research was trying to get the cybersecurity practice into something like general accounting principles: how many things are happening, how much is it costing, how much does a tool save us, what is the ROI?

When a CISO can go to the board and say, “If you give me X, I’ll give you 10X back and reduce risk,” and then prove it, that creates a sustainable conversation.

The fourth one is: security vendors lie. I’m a security vendor; I lie. It’s not always intentional, but vendors can get so deep into their own stuff that they think their thing can do more than it actually can in context.

The fifth one is that organizations are challenged in sharing information with each other efficiently.

The sixth one is that law enforcement and organizations don’t share information well.

And the seventh is based on that: law enforcement lacks the evidence to adequately prosecute criminals because it’s all inside our networks and not available to them in the right way.

Those are the seven that we study. We’ve been studying them over the last six years and doing thousands of experiments to learn what we’ve learned.

Manoj Tandon:
Are any of these companies doing them well, or are they generally lacking across the seven?

Charles Herring:
As a craft, we have systemic problems. It’s more art than craft right now. There should be a flowchart instead of chicken bones and great storytelling.

Manoj Tandon:
I can’t believe how quickly time has flown here. Out of ten questions, we’ve only gotten to three.

I do want to understand a little bit about Witfoo—how it came into existence and what exactly you guys are all about.

Charles Herring:
The product we sell is essentially a big data SIEM. Send all your data in, we process it, and we create units of work. There are SOAR playbooks and orchestration around those units of work.

Then we use those units of work to create reporting on things like whether there is a gap or overlap between my tools and how we’re detecting and responding. Should I tune a tool, or am I maximizing the value? Do I have false positives? Is the tool set to detect only or detect and block?

We also do automated compliance checking against NIST 800-53. How many controls are we detecting from your telemetry that are in place?

A lot of our research is around natural language processing and comprehending every message. If a message comes into our team that we’ve never seen before, that opens up a ticket. Our research team reaches out to vendors, understands the etymology of that message, and we build what we call a semantic frame.

Once we understand what’s in the message, why it was sent, what we’re supposed to do with it, and how it maps to frameworks like STIX or the MITRE ATT&CK framework, we’re able to stitch that together into these modus operandi investigations.

We also work a lot with MSSPs, so we have several federated, multi-tenant modes and work with sharing information. We’re also working on a “Report to Police” button so you can send an affidavit of attack in a way that makes sense to law enforcement.

It’s a way for us to make sense of all the data and try to stabilize those seven conversations. We like to say the heroes out in the field are doing the real work; we’re just giving them the equipment and gear to do it better.

Manoj Tandon:
How did the name come about?

Charles Herring:
“Wit” is keenness of mind—being witty, being smart with intelligence. That comes from what we’re doing: normalizing and collecting the things we already know.

There’s a big move toward AI and machine learning and all these trick plays, and we realized that most of the stuff we need to do, we already know. We just needed a way to collect it and operationalize it.

And “foo” is old hacker slang for skill. So Witfoo is about putting that knowledge to work.

Manoj Tandon:
Is there a charge for that?

Charles Herring:
Yeah, that’s how I eat.

If you go to witfoo.com, we’re pretty transparent. We only have one SKU, and it’s based on knowledge workers or employees in an organization. We also have managed security service providers that host it and provide services around it.

But the software includes all of the sharing and intelligence. We keep it turnkey, scalable, and simple.

A dear friend of mine, TK Keanini at Cisco, once said the first time he went to RSA it was like going to a transportation conference where all the attendees wanted to buy cars, but all the vendors were selling car parts. I really wanted to build something that had the windshield wipers and everything else, where you don’t have to bring your own carburetor and try to put it all together.

Manoj Tandon:
Are there any organizations, events, or things that you’d like to plug that our listeners should be aware of? Places you’ll be appearing, books coming out, papers you’ll be presenting?

Charles Herring:
A couple of things.

I’m launching a project called Log Fibber—L-O-G-F-I-B-B-E-R. It’s an open-source Apache 2 project. The repo will go public at Black Hat during the summer.

It’s essentially a kit for doing training or red team exercises where you write up all these lies that you want your tools—like Witfoo Precinct, Splunk, or whatever you’re using—to believe.

You configure fake devices and then generate the messages those devices would produce. Based on the natural language processing we’ve done with semantic frames, we have a library of what I call reframes that take generalized messages and turn them into proprietary messages.

We’re launching an education initiative later this month where this tool becomes part of it, so educators can say, “Here’s what Edward Snowden did in the Prism breach, and here’s what it would look like in our tools.”

I’m also speaking in real life for the first time—no Zoom—at B-Sides Northern Virginia on the first weekend of June. The talk is called Breaking in Bad, and it’s about breaking network behavioral anomaly detection, UEBA, and SIEMs—essentially how you poison data, trick blue teams into making bad decisions, and how blue teams can defend against those data poisoning tactics.

Manoj Tandon:
We’ll put links to both B-Sides and your new project in the show notes.

That’s fantastic. In parting, of all the places you’ve dove, for any budding divers out there, is there a place you would tell a beginning diver to go?

Charles Herring:
My most relaxing place to go is Cozumel, Mexico. It’s also a straight shot south of me here in Chicago. They have these wonderful, easy drift dives. You can go down to 30 or 60 feet and have currents gently taking you along beautiful parts of the reef, seeing all kinds of great sea life in warm water.

Cozumel is my favorite place to go just to relax. It’s not necessarily the most exciting diving, but the water is clear, the people are amazing, and it’s a very easy dive. I like to dive with Scuba Du. That’s the name of the dive shop, and I’m going to plug them because I love them so much. They’re a fantastic dive crew down there.

Manoj Tandon:
That’s great. And you can have a drink with an umbrella in it as soon as you’re out of the water.

Charles Herring:
I do room, dive, pier, bar. That’s basically how I vacation for as long as I can get away with it.

Manoj Tandon:
Well, Charles, thank you so much for joining us. This has been an enlightening conversation. I appreciate you being on the show, and we look forward to having you back sometime.

Charles Herring:
Thanks very much. It was a great pleasure for me. I really appreciate your time.