Security Confidential S19 E4 Andy Smith

Manoj Tandon: Hello everyone, this is your host Manoj Tandon. Welcome to another episode of Dark Rhiino Security, Security Confidential. Reminding you, please hit the like and subscribe button, because we have another great guest today. And if you want us to keep bringing you this kind of content, play with the algorithms, hit the damn button, do us a favor, appreciate it.

And then without further delay, I want to introduce you to Mr. Andy Smith. He’s kind enough to join us this afternoon.

Andy is the CEO and co-founder of Tracebit. We’re going to learn a little bit more about his organization. It’s a cybersecurity company that’s building deception technology and essentially very sophisticated honeypots, if you want to think of it that way, or security canaries across the cloud infrastructure. We’re going to have Andy explain to us a little bit more in depth about that.

But before he got to Tracebit, he was head of engineering at Tessian, where firsthand exposure to widespread supply chain incidents affecting thousands of companies sharpened his and Sam’s conviction in the assumed breach philosophy that underpins Tracebit today. And this is a very apropos topic. I mean, we’re getting into third-party risk. We’re getting into a lot of events that have made the news today.

So, Andy, thank you so much for coming on the show. I look forward to having this conversation.

Andy Smith: Thank you very much for having me. Really excited to dig into it with you today.

Manoj Tandon: So give us a little bit of your background. You know, other than the little blurb I had, who is Andy Smith, and how did you end up in this line of business?

Andy Smith: Yeah, I have been tinkering with computers and cybersecurity, I think, since I was around 10 years old. That typical hacker mentality of wanting to understand things by breaking them is something that’s been in me from an early age.

My dad was in computers, and I got access to one fairly early and was very quickly interested in the internet and topics around hacking and things like that. So it’s been a fascination of mine since I was a kid, basically.

Manoj Tandon: Okay. And did you practice hacking? Were you a gray hat ever in your life, I guess?

Andy Smith: To some extent, yes. I mean, what I would say is I think today, if you want to learn computer security, you have a lot of resources available to you. You can spin up all sorts of virtual machines and testing environments that make things a lot easier.

What I would say is it wasn’t quite as easy back then, but you could certainly do some of your own experimentation on your own machines.

Manoj Tandon: So since you started so early, I’d love to get a little bit of your insight on this one topic, which is the one thing that most defenders get wrong. In fact, I’ll say almost all defenders, because I have yet to find one that got this one right, is to actually get into the mind of the hacker. Having that mindset.

They seem to be very tool-centric, and I’m just going to go ahead and say that. People are tool-driven, tool-centric, and they do well what they know how to utilize. But with some rare exceptions, primarily in the defense agencies where they actually try to understand the mindset of the adversary and the motivations of the adversary, why is very little focus put on that topic in the training of a cybersecurity professional? It almost seems like an afterthought.

Andy Smith: Yeah, I think that’s super interesting. What I would say is it wasn’t long before I migrated from breaker to builder. So for the past 15 years, my career has actually been in software engineering rather than offensive security. So I would not necessarily place myself in some elite red teaming bucket or anything like that.

I do find it interesting, this story that attackers think in graphs, and that’s not necessarily how defenders are modeling their estates. It’s about chaining together those different exploits. It’s a trope, right, but you just need to make one mistake for someone to get in. I absolutely do think that way.

I was definitely a child of reading and consuming literature around the likes of Kevin Mitnick and social engineering and these sorts of folks. I think it’s very natural for defenders, and I’ve played the role of a defender, to get excited about exciting tooling and exciting attacks.

Often, it does just come back to the basics. My previous company, Tessian, where I led the engineering team, built the product, and led the software engineers there, we were phishing. I remember when I joined that company, even I was kind of like, “Isn’t this solved? Haven’t Microsoft and Google taken a good crack at this?”

There were folks sort of saying to me, “Are you sure you want to go after email security?” This was in my previous role. Tessian got bought by Proofpoint a few years ago, and now we see another wave of email security companies. I think there’s been since then arguably two waves of email security companies.

Often, it really is the basics that folks are getting through by. That’s definitely a lesson that I’ve taken to heart over the years.

Manoj Tandon: So what good does all this technology do then?

Andy Smith: Yeah, that’s a good question. The other lesson I’ve learned on my journey here is that you can kind of think of it as an immune system. You do want some technology placed together to stop pieces getting through.

Tracebit was founded under this assumption that you have to assume breach. No matter what you do, someone is going to get in. As long as those computers are switched on and accessible by humans and plugged into networks, you have to model under the assumption that someone is going to get in.

But I do think from a mitigation and defense point of view, it makes sense to invest in some reasonable array of technology to reduce the probability of those threats getting through.

Manoj Tandon: It seems that the threats have become much more aggressive in their implementation. I mean, look, for example, the Jaguar Land Rover attack, probably the biggest cyberattack in history because the bailout was so massive with that.

Andy Smith: It impacted the UK’s GDP.

Manoj Tandon: Yeah. And you think an organization like Jaguar Land Rover doesn’t have one CISO. They probably have multiple CISOs across various divisions. They probably have every tool known to mankind deployed in that organization. Yet they suffered one of the greatest catastrophic failures of cybersecurity in recent times.

Is there any hope of stopping something like that from happening?

Andy Smith: I think the stance I would take is I don’t think there is hope in guaranteeing that the threat actors will never get through. I think what’s actually really important is that we need to build systems that are resilient to the possibility that that can and will happen.

I think the risk can sometimes build in when we build systems that are brittle. We talk with customers all the time about building big walls and assuming that no one will ever get in. And that works great until someone is unfortunately able to hop over that wall in a way that you didn’t expect or anticipate.

The counsel we would always give there is, it’s really about investing also on the other side of that wall and ensuring you’re able to respond quickly, eject, and limit the blast radius. That’s just as much a part of it as the defense piece.

Manoj Tandon: Andy, I’m so glad you said that. That’s brilliant, because that’s the part of the equation that oftentimes gets neglected, I guess, for lack of a better word.

You’re absolutely right. You should have the proper cybersecurity hygiene in place, but you should also know that at some point a breach is going to happen. I think it would be foolish in today’s day and age to think that you’re not going to get hit.

The question is, have you built that resilient infrastructure in place where the outcome is operational continuity? Do you get to preserve that when things go wrong?

I’m glad you daylighted that because that is something that we think—we’ve been on that message for a long time, and we think if enough people say it, then maybe someone will start listening.

But we were always surprised. So you had firsthand experience with a third-party breach. What was that experience about?

Andy Smith: Not something I can go into a tremendous amount of detail about, but I was at a previous company. I was responsible for the security program until we brought in a security leader. So building the product and securing the product was part of my role. We were also keeping email data in our cloud environments for our customers that we needed to protect.

Without sharing too much detail, we got quite close to a supply chain attack that attacked a lot of folks in the industry and actually went undetected for quite some time.

So the origin story of Tracebit was that we’d explored deception technology, security canaries, as a control to implement this idea of assumed breach. And we’d basically come to the conclusion that for our modern cloud-native environment, the only piece that really made sense to us, the only way we’d implement this successfully, would be to build it in-house.

To be honest, we did not get as close to building that in-house as we’d wanted to. We actually kicked the can down the road on that project. Without sharing too much, in response to this incident, when we were digging into it, we realized that had we gone and built this out, it was something we would have actually caught months earlier than we otherwise did.

So it was honestly the spark that led to the founding of Tracebit. We’d have gotten value from this; others would have too.

Manoj Tandon: I’m sorry.

Andy Smith: It was honestly the spark that led to the founding of Tracebit. We’d have gotten value from this; others would have too.

Manoj Tandon: So tell us a little bit about the space Tracebit sits in, because deception technology has been around for a long, long time, right?

Andy Smith: One hundred percent.

Manoj Tandon: And we haven’t seen a huge focus on it like we have in other areas, especially anything with AI in front of it. That’s a separate topic. Tell us a little bit about the space.

Andy Smith: AI is part of it, honestly.

The history of deception technology is fascinating. There’s a famous book that I’m sure your listeners and yourself are aware of, Cliff Stoll’s The Cuckoo’s Egg, which is arguably one of the first documentations of a cybercrime taking place.

In that book, Cliff—or I think it was his partner, actually—they had this idea to basically build a honeypot to detect this Russian threat actor. So they basically laid a trap, something that appeared to be an attractive resource within the university network, that they then used to study and detect the threat actor in the environment.

I think that was in the early ’80s when the first ideas around deception technology in cybersecurity came around. Obviously, prior to cybersecurity, it goes a lot further than that. Decoys have been used in warfare for hundreds and hundreds of years.

As an industry space, there have been waves of excitement in the technology over the past 15 to 20 years. If you were at RSA eight or nine years ago, there was a time when the floor was covered with deception vendors. That’s not the case today. We are one of the few modern deception vendors that’s growing fast in this space.

We see things differently this time than they were in the past. Honestly, there are two parts to this.

One is cloud environments. If you speak to folks who deployed deception in the past, who bought these vendors maybe nine or 10 years ago, what you’ll hear over and over again is that these were very, very heavyweight to deploy.

Manoj Tandon: Okay.

Andy Smith: We’ve met people who’ve had teams of three, four, or five individual contributors just keeping these things online. They were often actually shipping racks out to you to go and install into your data center. This was very heavyweight.

If you think about it, if we want to go and spin up a large complicated network back then, and we wanted to have a chance of catching a threat actor in that network, we might have to spin up hundreds or even thousands of servers or virtual machines. That has a cost, right? There’s CPU, there’s memory, there’s disk.

That was a very non-trivial cost. Not just the creation, but also the maintenance and ongoing operation. It also introduces risks and complexity into the environment. You say to a CISO, “Hey, I want to go deploy 500 virtual machines into your network.” Not only are they thinking about the expense of running them, they’re also thinking about patch management, the third-party software that’s going to run all of these pieces that come into play.

Our approach, and honestly why we founded Tracebit and felt now is the time to spin up a company in this space, is very, very cloud-focused.

When you look at the cloud and you look at serverless technology, there’s a very different story when it comes to creating deceptive resources. As an example, if you take an AWS account, within an AWS account you have Secrets Manager secrets, you have S3 buckets, you have IAM roles.

Manoj Tandon: Okay.

Andy Smith: If you zoom into the Kubernetes clusters in that account, you’ll have service accounts, secrets, pods. You have a lot of resources that are actually the public cloud provider’s cost and responsibility in many senses.

An S3 bucket is basically free to create from a financial point of view. There’s no patching going on. There is, but Amazon Web Services are doing the patching under the hood, and that’s what you’re paying them for.

If you want to scale and deploy a lot of these resources in those environments, we’re at a point in the industry where most platform teams have aligned behind a small set of technologies to do that. For example, Terraform is pretty much the de facto, I would say, at this point for managing large, complicated cloud environments at scale.

These are some of the technology shifts that we got very excited about when we thought about building a deception company today.

Because if you take a large complicated AWS environment, as an example, you can actually deploy tens, hundreds, or thousands of S3 buckets, secrets, Dynamo tables, IAM roles, etc., in that environment without introducing a significant cost or maintenance burden.

Then the outcome that you have is many, many tripwires around that environment for some of your most critical resources for when a threat actor does compromise that environment.

Manoj Tandon: Delicious.

Andy Smith: The other piece before I pause is that we’ve actually been leveraging LLMs over the past few years for this.

When you speak to teams about the pain points of the past deception vendors, especially for large environments, configuration, deployment, and decision-making were a big part of it. Because you have to go and say, “Okay, I have my prod PCI-compliance environment over here, and this has a particular naming scheme, and it’s all in this region. We’ve got this project name called Project X.”

Obviously, for you to be likely to catch threat actors in that environment, you need to match all of your deceptive assets to those resource names. But security teams are notoriously under-resourced. The idea of them picking through every single environment in an organization and getting creative and coming up with great names and configurations that are going to tell a believable story for threat actors—I’m sure you can already imagine just how much work that would be.

If you take LLMs, it turns out they’re actually really effective at this, unsurprisingly. So what we’ll do is we’ll take metadata from a customer environment. We’ll understand how a resource is named, configured, and where it’s located. We’ll leverage an LLM to come up with canary resources, deceptive resources that tell a believable story from that environment, and deploy a range of different resources: some that are particularly attractive, some that just look like all the other resources.

Ultimately, this is something that you can actually effectively scale across very, very large enterprise environments. Those are the two key technological shifts for us that meant we thought it was worth building a company in this space today.

Manoj Tandon: I get why do it. Now, let me ask the counterpoint on that. With the advent of AI and its daily evolution, isn’t it going to be just like we had malware that could detect it was running in a sandbox environment, or it had been sandboxed, and it got very good at making sure it wasn’t leaving tracks that would sandbox it?

Isn’t that the same thing with canaries now, where it’s going to be like, “Okay, you know what, why can’t the AI detect the canary in the coal mine in the first place and say, I know that’s a canary and I’m not going to even go bother to tag it because it’s too obvious. Nobody puts anything out there that’s…” As an example, I’m making that up.

Andy Smith: Yeah. No, it’s a really good question. It’s a really, really good question.

I think there are two parts to the story. One, that is actually another big advantage of cloud environments. The old way of doing things was that you were emulating network services. Emulating network services is inherently fingerprintable.

There’s always going to be some little fraction in the TCP handshake or something on the banner or some signal that lets me go, “Aha, this is a Linux box pretending to be a printer,” or “This isn’t really an FTP server. It’s someone’s Python implementation of an FTP server.” There’s always going to be that detail you can dig into if you get deep enough.

I agree, LLMs are going to be really good at digging into that PCAP data, making those assumptions, spinning up their own test environments for the real stuff, and comparing. I completely agree. That’s one of the reasons why we haven’t gone that deep on the network-level pieces. I think there’s some value there, but I don’t think that’s where the sweet spot is.

If you compare that to cloud resources, an S3 bucket in an AWS environment is an S3 bucket in that AWS environment. There’s nothing emulated or fake about it. If you’re trying to fingerprint that, if you’re trying to say what’s the real one and what’s not the real one, there are actually only a handful of properties that exist when you create an S3 bucket.

There is the name. Is it using encryption? What type of encryption? Is it using versioning? What’s the IAM policy? All of these pieces, we can emulate. We can fairly easily mirror when we’re creating them.

Also, on top of that, if you need to go deep on that S3 bucket and inspect it further and look in the content, that’s the point where you’d actually go and trigger the canary.

So that’s part of it. That’s why I think cloud is a really, really suitable place for deception.

The other part that I think is even more interesting is when you think about the studies out there. The Tularosa study is a really interesting one, which compared a number of groups of expert red teamers and tested the psychological impact of deception upon them.

Long story short, and there were a number of results, there’s some indication that if a pen tester or red teamer believes there’s deception in the environment, they’re going to be slower and less effective at reaching their goal.

Some examples: they find a key, they find an opportunity to escalate privileges, and they avoid it because they think it’s a deceptive resource, even if it isn’t. They sometimes think, “That’s too good to be true. I’m not going to go for it.” Or they don’t do as much enumeration as they would normally do because they’re worried about setting off the canaries. There are all sorts of different reasons that this affects them.

We’re actually working on this research at the moment and asking the question, “What about LLMs? Do they mirror that behavior?”

Manoj Tandon: And that was going to be my next question, so thank you. Go ahead.

Andy Smith: Our early indications are that yes, they do. So you have situations where, because they are informed of the presence of deception in the environment, just like a human would be, there might be attack paths that they don’t explore because they’re concerned that they’re going to set off the deception.

They can take longer to explore the environment. They can be less aggressive on enumeration. They could also just skip real paths altogether just because they believe it’s deception.

The short answer is I think it’s pretty hard, regardless, for LLMs to do that fingerprinting. But then also what we’re seeing is that there is some psychological impact—or “psychological,” it’s obviously not actually psychologically impacting them—but it does seem to have a similar effect on LLMs in environments.

Manoj Tandon: But for that effect to take place, the attacker would have to know beforehand that there’s some, right? And most attackers, especially the ransomware crowd, that’s a lazy group of people anymore.

Andy Smith: Yeah, this is true.

Andy Smith: And so your point being that how would they know? Would they even research this?

Manoj Tandon: They won’t. They’re just doing ransomware as a service. They’re just hitting up people to see who falls. I mean, not all cyberattacks are like that. Don’t get me wrong. There are people who take their time and they case out the operation and they have a very set objective.

But the vast majority of attacks that take place don’t fall into that category. Most are, “Can I get a couple bucks out of these guys and move on to the next?”

Andy Smith: Smash and grab. Yeah, totally. Sorry, go on.

Manoj Tandon: Smash and get.

Go ahead, sorry.

Andy Smith: Yes, definitely. I would say though, if we’re going to assume that in the future, which to me completely checks out, a lot of these attacks are going to be fully automated through LLMs, I would also assume those LLMs are going to do their research ahead of the attack. It just makes sense.

If I was going to be plugging one of these things together, I’d at least say, “Let’s go look at the job ads and see what tech stack they have. Let’s go and read. Do they have a security policy? Let’s go and do these pieces,” because that’s going to make me quicker at getting to my goal. I’ll use fewer tools or at least know how to prioritize my toolkit.

So this is why we are actually massive proponents of being very public about your use of deception technology. This definitely splits opinions, but we’re one of the few deception vendors out there where, if you go check out our website, you will see customer case studies. You will see our customers talking about where they put deception and what value they got from it.

Obviously, it doesn’t go into a tremendous amount of detail that would allow someone to go and pick that deployment. A key part of what we’re doing is heavy, ongoing automation. So where deception was today in a customer environment, it may not be tomorrow. We’re moving these things around. It’s a moving target.

But yeah, this is exactly why we’re big proponents of talking about this publicly. My view there is that I think the LLMs will do some degree of exploration, and if they do that, they’ll come across that information.

Manoj Tandon: I could see why the split is there. On one hand, if you declare you’re using such technology, then it might act as another layer. Deception technology might act as another layer in a defense-in-depth architecture because you are slowing down movement of the adversary in a way.

I could also see why tell them anything at all. Let them see what they go and grab.

If I’m an organization that has a target on my back, and there are umpteen industries where that is certainly the case, I would, if I was their CISO, be operating under the premise that we have been breached. We just don’t know it yet.

Because in most of these organizations, the attack surface is so large, it’s almost impossible to defend the entire attack surface. And the biggest thing is you can’t alter human behavior. That’s the biggest gotcha.

If people actually started behaving properly and didn’t just randomly click on stuff and do things, we would have much less of a problem. But that’s not going to be the case. That’ll never be. People love pictures of cute kittens playing with yarn, and they’ll click on it, and ba-bang, things happen.

Can having deception technology, in an instance where we operate under the belief that the premise has been breached already, we just don’t know it, accelerate the detection? Can it accelerate the effectiveness of our red teams, if you will? Can we find it faster? And if so, how?

Andy Smith: Yeah, for sure. I would say that the majority of our customers are operating under that assumed breach philosophy.

I don’t think this necessarily needs to apply to just large enterprises. Especially in this day and age, with supply chain attacks that we’re seeing as of late, even small organizations that aren’t being targeted and have a relatively tight setup—the idea of a dodgy Python or Node package ending up on a developer’s laptop is just a fairly safe bet at this point in time.

In terms of whether deception can speed up that post-breach detection, I think I could share a really interesting anonymized customer story where we saw a lot of value.

We worked with a customer not too long ago to deploy across their cloud. They had all the tools you’d expect from a mature security team in that environment. We deployed over a couple of days across their public cloud environment some of the pieces I’ve mentioned: some infrastructure, some keys, the sort of pieces that a threat actor would go after.

I think it was two days after deployment, we got the first detection. We were not involved in the incident response. That was handled on the customer side. A few weeks later, we caught up with them, and long story short, it transpired that it was a service account that had been compromised. It had actually been compromised for a significant amount of time.

A lot of the anomaly detection that they had been using in that environment to detect unusual behavior wasn’t going off because the same threat actor was returning from the same set of IP addresses and performing the same set of behaviors. But in this case—

Manoj Tandon: Then somebody must have clicked it as being legit too.

Andy Smith: Back some time ago, possibly. I don’t know the root cause of that.

Manoj Tandon: Because that’s what happened in the Target breach. I don’t know if you remember that. Target department stores in the U.S. when they got—

Andy Smith: I remember the breach. I don’t remember that detail.

Manoj Tandon: The detail was, it actually happened here in Pittsburgh, but it was an HVAC contractor that plugged their laptop into their network. Their security operations center actually registered the breach, from what I have read. But when the alarms went off, analysts kept saying this is okay. That’s why I bring that up.

Andy Smith: Right, got you. I don’t know if that happened. I wouldn’t like to speculate.

But in this case, within two days, we were able to detect a latent compromise in that environment and kick that actor out.

Manoj Tandon: Awesome.

So you were a secondary check against the SOC then.

Andy Smith: We were feeding into the SOC, and we generally would be consumed by the SOC. But yeah, we would be that detection for when the other tools don’t quite do what you’re hoping they would do.

Manoj Tandon: See, that’s something I had not considered. That could be a very interesting layer to add.

You think about the amount of creativity that’s going into avoiding detection, especially the really good hacking outfits. They really engineer these attacks and they do a good job of it. They just do a good job of that.

A lot of times, especially if they know what SIEM you’re using and they are able to engineer ways around detection of that SIEM platform.

So potentially, I’m just thinking out of the box here, you have something like Tracebit that is an unbiased observer, if you will, that is the secondary warning that says, “Hey, wait a minute. Even though you don’t have any lights turning red on your screen, there’s something really wrong here.”

And something that overcomes the confirmation bias that’s inherent in—

Andy Smith: That is absolutely a use case that we see.

Manoj Tandon: SOC operations, if I said that right.

Andy Smith: For sure. I definitely think so.

We also hear from folks who work with us that when they read about an incident hitting some peers or in the news, or something that could be affecting them, the fact that we haven’t gone off is a really positive signal for them.

It’s not going to lead them not to run an investigation. It’s not going to lead them not to do that threat hunt. But when they’re frantically logging onto their laptop at three o’clock in the morning, the fact that they know the Tracebit canaries have not triggered is a big plus.

Manoj Tandon: Now, is Tracebit being run completely in the cloud? Is your service cloud native 100%? So there’s no local agent installation or anything of that sort?

Andy Smith: So, all the examples I’ve focused on so far have been public cloud. We go into a lot of other places.

Another popular deployment is workstations and user devices. We can push credentials and soon artifacts onto those devices. We’ve been mentioning the supply chain attacks; that’s obviously a really nice detection.

CI/CD environments are another really popular place for us to go. In the future, we want to head outside of public cloud into more typical SaaS environments. Think Google Drive, O365, SharePoint, these sorts of systems.

Today, we’re not in anything on-prem, but that is also something we’re feeling a little bit of pull for. I do still think there are those challenges I mentioned earlier around the weight of the deployment and the fingerprintability.

Manoj Tandon: Yes. Well, the fingerprintability is the thing that I’m thinking about. The thing with cloud-native solutions is they’re really cool. But if I can figure out what the IPs of the servers are, what infrastructure you use, and as a hacker, I want to see if there’s any communication back, I don’t even have to guess if there are any canaries out there.

What I want to see is: is anybody talking to the infrastructure? If they are, there’s obviously canaries in the environment.

Andy Smith: Yep. I think we do go to a reasonable amount of effort to limit the ability to detect Tracebit in the environment. Obviously, we still hold that position that folks knowing Tracebit is in the environment is probably a positive, not a negative.

Where we are focused is around them being able to easily enumerate which are and are not the Tracebit canaries. That would obviously be a negative if it was easy for them to go pull that list somehow.

We do have public customer stories where there have been red team engagements that have been aware that deception and Tracebit are in the environment. Part of their goal has been to make it from a developer laptop to a production database, while avoiding deception and Tracebit being in scope for that evasion. We’ve been able to detect them. We’ve been able to place enough tripwires in their way that basically detection has been unavoidable for those red teams.

Manoj Tandon: What are some things that people do that make the deployment of deception technology less effective?

Andy Smith: Great question. I think the big one for me is the initial deployment. Folks get quite focused on what good looks like today. Because I come from a software engineering background, maintenance is 80% of the work. So focusing on what good looks like in an ongoing deployment of deception is really important.

To give you a really simple, concrete example, and this is something we’ve seen from customers fairly often who’ve rolled something themselves or tried to start with something simple: you go and deploy some canary tokens to your end-user devices. So you put some AWS keys on your end-user devices. Maybe you write a script and you roll these out.

Something we’ve heard far too many times from customers is this: the beauty of deception is it doesn’t go off very often. It’s not going to be pinging your SOC every couple of weeks or every couple of months.

We’ve met folks who rolled out canary tokens across their estate, and then it was really 18 months before they saw something. So they got that trigger: “Hey, on this particular device, someone has hit a canary token.”

The question then comes, “Okay, when did that token leave the device?” And the token has an infinite expiry date, and it was placed on that device 18 months ago. Well, the answer is it could have left that device at any point in the last 18 months. It could have actually left that device the day you deployed it. Someone could have accidentally shipped that off to their home PC, and it could have sat there for that whole time. Then finally, their personal email address or whatever got compromised. We just don’t know the answer to that question.

Then the question comes up, “What do we do?”

Manoj Tandon: Right.

Andy Smith: Do we even have telemetry for 18 months? Do we even have that data? Are we even going to go and do that digging? Often, the answer is no.

So I think a mistake is not keeping that feedback loop and that deployment as tight as you possibly can.

We have some writing on our blog about this, and you don’t have to buy our product to do this. If you’re going to roll it yourself, just think about this upfront.

A simple solution is to automate the deployment of those canary tokens. So you push fresh ones down every 24 hours. If one of those triggers, and you know it was on that device, you have that really tight time window to go and run that detection.

This also has other nice benefits. In the 18-month example, that was deployed manually. Well, now someone needs to go and deploy that fix manually, right? You’ve just delayed some technical debt, and the security team now has to deal with it. Maybe if it was one trigger, it’s not so bad.

Manoj Tandon: Right.

Andy Smith: But what if a supply chain attack hit all of your developers, and it just lit up 500 laptops? Well, now you need to go and think about doing that manual rolling of keys.

So really, think about what good looks like in the long term. That’s what I would always counsel folks to think about quite deeply before they embark on such a project.

Manoj Tandon: Have you built a lot of automation into your platform so that, as in the example you gave, you can automate the rolling out of various canary types on a regular basis and on a randomized basis, if you will, not even a regular basis, without invoking an IT maintenance person to go and make sure it gets done?

Andy Smith: Yes.

Andy Smith: Yeah, exactly. Every time we go off for a new platform and implement a new module, that is one of the main questions we’re always thinking about. Sometimes, if we can’t automate the maintenance, we may not go after it because we don’t want to create that burden.

The big piece where we do this is in cloud environments because, as I mentioned earlier, we’re pulling back metadata to profile and understand these environments so we can create realistically named resources.

That’s an ongoing deployment. If you spin up Project Canary in your production environment, Tracebit is going to recognize that. We’re going to say, “Let’s spin up some other resources with that same name.”

Similarly, if you wind down a project, we’ll wind down the canaries for that name because they’re no longer relevant. If someone had intimate knowledge of that environment, they may detect it as a canary, or it may even cause some confusion.

Manoj Tandon: Now, do you have any examples of your technology being used to actively monitor bad actors? So where, as opposed to shutting them down, you’re actually tracing those canaries and where they’re going, or getting an ideation of the intention behind the compromise, or maybe feeding them intentionally bad data?

Andy Smith: Yeah, it’s a good question. This is something we’re exploring. It’s not something we have today.

Where we’d like to go is chaining multiple canaries together so that you can create—imagine someone downloads a file from an S3 bucket, then that file has a reference to a URL. You hit that URL, and it has a form that’s believable for the particular industry you’re in. You can then perform actions on that form.

Imagine all of those pieces being canaries and us being able to thread that together.

We don’t have that in the platform today. We’ve been really focused on high-fidelity detection. That is our focus today. Folks buy our product because they want to detect threat actors without a significant amount of noise.

Why we’re exploring the deeper piece there is because we’ve made those detections and then we’ve been asked the question, “Great, they downloaded a file. I’d love to know what happened next. That would be really helpful for my investigation.”

So that’s where we’re going with the platform. It’s not what we have today.

Manoj Tandon: Yes.

Andy Smith: Your question actually did trigger another example that I’ve not really touched on, but it is worth sharing.

As we’ve mentioned AI, the other really interesting way we’ve been used as a measure for customers is measuring AI agent use and misuse on user workstations. It may come as no surprise if you place a lot of AWS keys or SSH keys or other keys that allow lateral movement or use of tools on workstations. If someone is running an AI agent like Claude Code in quite an aggressive way on their device, they will very likely hit those keys.

So we’ve had customers that have been able to use us to basically measure and understand where these agents are being used. We had a customer recently say to us, “I had no idea the sales team was using Claude Code until I saw a canary detection on their workstations.”

That’s an example of the way we’re being used to measure unexpected behavior in environments that isn’t traditional threat actors but is still providing those high-fidelity detections.

Manoj Tandon: Interesting use case. On the traditional side, do you have any metrics by chance on reductions in dwell time of bad actors where your technology is in play, Andy?

Andy Smith: Yeah, that’s a good question. The honest answer is we’re actually working on some of those pieces at the moment with customers. I don’t have any concrete public data that I can share there.

Manoj Tandon: That would be very cool because those dwell times have gotten shorter and shorter and shorter. And if they eat the canary first, that’s relevant and material.

Andy Smith: Yeah. I mean, the example I shared earlier, there had been a multi-year dwell time that we were able to detect in two days. So there are case studies like that, but that bigger data set is something we’re still collecting.

Manoj Tandon: Is there a particular set of industries that have taken to deception technology and have really embraced it?

Andy Smith: I think if you look across the industry outside of our customer base, it includes manufacturing, finance, healthcare, and technology companies. Those are the ones that stand out.

By function of the tech stacks that we are built for, if you look at our website, you’ll see we’re working with a lot of tech-forward tech companies, just because they’re most likely to be big adopters of the big clouds and these sorts of things.

But yeah, we’re seeing that the industries that generally invest in security seem to be adopting deception.

Manoj Tandon: Yeah. I was thinking critical infrastructure, but a lot of that is still very traditional. It’s very heavy on-prem still, and they have a lot of private cloud deployments.

To me, one of the biggest gaps in security in a lot of environments is the critical infrastructure. I’m pretty sure that a lot of those environments are breached. It’s just that no one’s found them, and everyone who has breached it, I’m sure, is lying low and not doing anything until the time is right.

Andy Smith: Yeah. So here in the UK, the NCSC, which is kind of the UK’s CISA, I think it was 18 months ago now, started a call for UK government, national infrastructure, and UK organizations in general to adopt deception. They’re big fans of deception, and national infrastructure, critical national infrastructure in particular, is something that they’re pushing on.

Manoj Tandon: Are they able to get funding to push deception? I’m curious. And if so, how did they do it? Because maybe there’s a lesson we could learn.

Andy Smith: Well, I think often in the UK, the budgets are a little bit lower than the U.S. in general. They’ve shared a bunch of this publicly. They certainly have resources behind it. They have a team of people working on this. But I believe that’s something they’re trying to encourage to happen between those businesses and entities.

Manoj Tandon: Well, Andy, we’re pretty much at the hour. We wanted to give you a couple minutes here at the end to talk to our audience about anything you’d like to let them know, anything you want to plug. So it’s your floor, anything you’d like to discuss with them.

Andy Smith: Amazing. Thank you.

Thanks so much. I really appreciate the questions and conversations. I really enjoyed it.

The piece I’d love to call out is we launched back in December a community edition of our product. Hopefully I’ve been banging the drum enough here that we think this is something everyone should be doing. We think this is a really valuable technique.

I’d really encourage your listeners to go check out community.tracebit.com, and then they can sign up. They can sign up, and we have an install that can run and deploy canary tokens on your workstations.

We also just launched, in response to some of these supply chain attacks that are happening, a GitHub integration. So we can actually automatically inject canary tokens, SSH keys, things like this, into your GitHub Action workflows. So if one of those were to get compromised, you’d be able to go and detect it.

Hopefully, there will be even more features coming soon.

Manoj Tandon: Fantastic. Well, as things develop, Andy, stay in touch. If you’ve got something new to tell us, you’re always welcome back.

Andy Smith: Very well.

Thanks so much. I really appreciate it.

Manoj Tandon: This has been a fascinating conversation. Thanks for doing this, Andy. Really appreciate it.

Andy Smith: Thanks so much for having me, and thanks for the great questions. I really enjoyed the conversation.

Manoj Tandon: Take care.

Andy Smith: Thank you. Bye.

Manoj Tandon: Bye-bye.