Ethical Hacker Kevin Poulsen Demonstrates How Threat Actors Can Exploit Generative AI

Alex Kantrowitz:

All right, well, I'm definitely ready to learn more. I want to see how this can be used for bad, so let's get to the part that you're all here for, which is a live demo from a hacker himself. Let's turn it over to Kevin. Kevin, take it away.

Kevin Poulsen:

If we look at phishing attacks in particular, that's the place where generative AI has the most obvious application because large language models are experts in writing; that's basically what they are. They kind of solve the problem that you have in staging a widespread phishing attack—where you either have to choose between tailoring your content to a specific target by hand, handcraft the email that you're going to send your victim in order to try and fool them into clicking on a link or opening an attachment, or you can get lots of victims and just automate the whole thing with the same message or a slight variation on the same message.

The big change that we're going to see here, and that we're already starting to see, is that now you can automate the very well-crafted personalized attacks and scale them up, basically letting it run unattended and reach an unlimited number of victims, probably with a really high success rate. The creators and curators of these models know this, and there's been a lot of flack that they've gotten already over the prospect that their tools are going to be used for this purpose, so they've put in some safeguards to try and stop them from being used to generate phishing emails.

After playing around, I found out the best way around that is to have my own, to run a model locally, and the best model for doing that is Facebook's Llama 2 model—enormously powerful, 70 billion parameters, and you can run it on your own computer if you have the right kind of computer. You run it in kind of a slightly stripped-down form called quantized, so I'm doing that here. What this lets me do is I can tell it to generate a phishing email. As part of this process, I've automated the initial step of gathering a little bit of information on the target of my attack.

These chatbots can be useful for basically every phase of a social engineering attack, like a phishing attack, starting with gathering information on your target so that you can incorporate that information into a targeted spearfishing email. Bard is good for that. Unlike the other models, it's plugged into the live web basically with Microsoft search engine in the background, so it can give you current information. So, if I want to target... All right, I'll make myself a guinea pig.

Instead of just giving you a conventional search engine—a list of websites and little blurbs from the websites—and then you have to make sense of it and maybe click on some of the links, with Bard, we just get back a nice summary of who you are based on what the internet knows about you, and that gives you ways into crafting an email, attacking that person. You know a bit about them, you can try and impersonate somebody that they may have met or worked with in another job, stuff like that. What Bard won't do is put that into the form of a phishing email, like ChatGPT, it has safeguards against that.

So, it was thinking about it, maybe this time it's just going to cross over to the dark side. Nope, no, well beyond its capabilities. It's being modest there; it just doesn't want to do it. So, the same thing would happen with ChatGPT. It won't go so far as to actually write something for you. Now, people have found ways with ChatGPT to get it to ignore its own safeguards and to do things it's not supposed to do, and that's turned into kind of a cat-and-mouse game. Somebody finds a way to jailbreak ChatGPT, and then OpenAI comes in, and they fix it, and then someone develops a new way and it gets out, and OpenAI fixes that.

What I've found is that the best way to reliably produce criminal content from a large language model is to host it yourself. And there are a bunch of publicly available models that you can download and run on a computer with the right hardware. Of those, the most powerful currently is Facebook's Llama 2 model, which is enormous and is very smart. It also has some safeguards. If you ask it outright to write a phishing email, it'll give you a long lecture about why phishing is bad and you should do something more constructive with your life, but that's easy to get around just by changing the format that you use for writing the prompt. And once you've gotten around it, you don't have to worry about a cat-and-mouse game because it's on your computer and nobody's going to change it.

So, what I'm able to do with this is the first part, the part that I just used Bard for, gathering a little information about the person that I'm targeting. I wrapped that up and made that kind of automated now so that I could just put that right into my prompt the first go around. And what it's doing now is it's searching the web, gathering the top 20 or so results, and then sending the summary of those results to ChatGPT with a request to just summarize everything that I can tell about a person based on those search results.

So, ChatGPT should send back a brief summary like the one that we just saw from Bard, and then using that in my prompt, I'm going to ask Llama 2 to compose a targeted phishing email and do what Bard wouldn't do, so this is what a crime prompt looks like. So, we start with a general description of what it is that you're asking the AI to do, so here I'm just being completely upfront about it and telling him that I want him to craft a phishing email. To avoid a little bit of unpleasantry, I tell it that I'm a computer security person conducting a pen test or a red team attack on the target and not a phisher, but it knows what's going on. I give it some tips and I give it some examples, and then at the end, I get specific about my current victim, so my view in here, Ron, sorry, I picked you at random.

Ronnie Tokazowski:

That works. Yeah, most of the stuff is pretty accurate in there.

Kevin Poulsen:

Yeah, well-regarded security researcher. Wow, Gardner-Webb. So, with this information now in my prompt, I can send this to Llama 2, and it's going to have to think about it in a minute because it's an enormously smart AI running on my PC, but it shouldn't take too long.

So, it's decided, Ron, that it's going to fake an official email from your alma mater, and this is from Sarah Johnson, the Dean of Students at Gardner-Webb University. So the goal of this email, it's supposed to try and get you to click on a link.

Ronnie Tokazowski:

Yeah, it's really interesting with how it's coming through and describing based on my previous experiences of I'm going to pick the university that you're at, represent somebody who is at that university, and then based on that expertise, it's doing a really good job at crafting that email based on something that might be interesting to me or something that would make it more likely that I would be clicking on, so yeah, definitely pretty interesting on that.

Kevin Poulsen:

I can tell you're fighting the impulse to just reach out and click on that right now.

Ronnie Tokazowski:

Oh, I know. I just feel like reaching through my monitor and go click.

Kevin Poulsen:

So, you can see that didn't take very long and without me having to do any manual research, this could basically be a one-click process, which means I could just feed it a list of names and have it compose endless emails to people and just count on some portion of them clicking on it. So this takes what used to be kind of a craft, writing a compelling, credible phishing email, specifically targeting somebody based on information about that person, and it scales it up and makes it now a commodity.