Analysis

I Replaced Our Security Awareness Training With AI — Here's What Happened

Jason ·

Our security awareness program was a joke. Not literally — it was a perfectly standard program run through KnowBe4, with monthly phishing simulations and annual training videos. The problem was that nothing was improving. Our phishing click rate hovered around 12% for two years straight. People would watch the training, pass the quiz, and then click a phishing link the next week. The training was checking a compliance box, not changing behavior.

I got frustrated enough to try something different. For 90 days, I replaced our standard phishing simulation templates with AI-generated, highly personalized phishing campaigns. The results were dramatic enough that our CISO approved a permanent shift. Here's the full experiment.

The Problem With Generic Phishing Templates

Most phishing simulation platforms give you a library of templates. "Your package couldn't be delivered." "Your password expires in 24 hours." "HR needs you to update your benefits." These worked when phishing was unsophisticated. They don't work anymore because real phishing has evolved. The actual phishing emails hitting our users' inboxes were AI-generated, hyper-personalized, and referenced real projects, real colleagues, and real deadlines. Our training simulations were preparing people for attacks from 2019.

The disconnect was obvious: we were training people to recognize generic phishing while they were being targeted by personalized phishing. It's like teaching someone to identify a wolf by showing them pictures of dogs.

The AI-Generated Approach

I used Claude to generate phishing simulations tailored to each department. The prompt structure was:

"Generate a realistic spear phishing email targeting a [DEPARTMENT] employee at a mid-size company. The email should reference [CURRENT PROJECT/INITIATIVE] and appear to come from [PLAUSIBLE SENDER]. Include a plausible pretext for clicking a link. Make it sophisticated enough that a trained security professional would need to look carefully to identify it as phishing. Include subtle red flags that an educated user could spot."

For the finance department, the AI generated emails referencing quarterly close deadlines, vendor payment approvals, and audit requests. For engineering, it was code review requests, CI/CD pipeline alerts, and AWS billing notifications. For marketing, it was brand asset requests and social media analytics reports. Each email was contextually relevant to the recipient's actual work.

Week 1-4: The Baseline Shock

We sent the first batch of AI-generated phishing simulations without telling anyone we'd changed our approach. The click rate jumped from 12% to 31%. Nearly one in three employees clicked an AI-generated phishing link. In the finance department, it was 42%.

This was uncomfortable. It meant our previous 12% click rate wasn't measuring security awareness — it was measuring employees' ability to recognize the specific templates we'd been using for two years. They'd learned to spot KnowBe4-style phishing, not actual phishing.

The immediate reaction from leadership was concern. "Are we making the phishing too hard?" My response: "We're making it as hard as real attackers make it. Would you rather find out here or during an actual attack?"

Week 5-8: Targeted AI Coaching

Here's where the AI approach really differentiated itself. Instead of sending clickers to a generic "you fell for phishing, here's a video" landing page, I used AI to generate personalized feedback for each person who clicked.

The feedback included: which specific red flags were in the email they clicked, why the particular pretext was designed to target their role, and three things to check before clicking any link in a similar email. The feedback was 2-3 paragraphs, written in a conversational tone, and specific to the exact email they fell for.

I also generated role-specific "phishing spotting guides" using AI. The finance team got a guide focused on invoice fraud, payment redirect attacks, and fake audit requests. The engineering team got a guide about supply chain attacks, fake repository notifications, and credential harvesting through CI/CD tools. These guides were 2 pages each and referenced actual attack patterns relevant to their role.

Week 9-12: The Results

By week 12, the overall click rate on AI-generated phishing had dropped from 31% to 8%. On emails with difficulty comparable to the original templates, the click rate was 3%. That's a 75% improvement over our two-year plateau of 12%.

The department breakdown was telling:

  • Finance: 42% → 6% (biggest improvement, likely because they were the most targeted and got the most specific coaching)
  • Engineering: 22% → 5% (already had better baseline awareness, but the role-specific scenarios still caught people)
  • Marketing: 35% → 11% (slowest improvement, possibly because their role involves clicking links in emails more frequently)
  • HR: 28% → 7% (responded very well to the personalized coaching approach)
  • Executive team: 38% → 9% (the AI-generated board meeting and M&A themed phishing was devastatingly effective initially)

What Made the Difference

I attribute the improvement to three factors:

Relevance. People paid attention to the coaching because it was about their specific job. A finance person reading about how invoice fraud works cares more than a finance person watching a generic video about phishing. The AI-generated content was relevant to their daily work, which made it feel practical rather than performative.

Personalization. Getting a personalized "here's why this specific email fooled you" message is more impactful than a generic "you clicked a phishing link" landing page. People remembered the specific red flags because they were tied to a specific experience.

Escalating difficulty. I gradually increased the sophistication of the AI-generated phishing over the 12 weeks. Early simulations had 3-4 obvious red flags. Later ones had 1-2 subtle ones. This progressive difficulty trained people's detection skills incrementally rather than presenting the same difficulty level every month for two years.

The Cost and Effort

Let's be honest about the work involved. Generating personalized phishing simulations and coaching content took me about 6 hours per campaign (we ran biweekly campaigns). Over 12 weeks, that's about 36 hours of my time. AI API costs were negligible — under $50 for the entire experiment.

Compare that to our KnowBe4 license cost of $18,000/year that was producing stagnant results. The AI approach cost me $50 in API fees and 36 hours of time, and produced a 75% improvement in phishing resilience. The ROI isn't even close.

Going forward, we're keeping our KnowBe4 platform for the phishing simulation infrastructure (sending emails, tracking clicks, reporting) but replacing all their templates with AI-generated content. Best of both worlds: proven delivery infrastructure with dynamic, personalized content.

Lessons for Your Organization

  • Your phishing metrics might be lying to you. If your click rate has been stable for more than a year, you're probably measuring template recognition, not security awareness.
  • Personalization beats volume. One targeted, role-relevant phishing simulation per month teaches more than four generic ones.
  • Invest in the coaching, not just the test. The phishing simulation is the assessment. The coaching afterward is the training. Most programs over-invest in simulations and under-invest in coaching.
  • Start uncomfortable. Your initial AI-generated phishing results will be embarrassing. That's the point. You're seeing reality for the first time.

If your click rate has flatlined for a year, you don't have a training problem. You have a content problem. Generate five role-specific phishing scenarios with AI this week and send one. The number that comes back will tell you whether your awareness program is working or just running.