I Audited Our AI Tools for Data Leakage — The Results Were Ugly

Six months ago, our CISO asked me a question I couldn't answer: "Do we know what data our AI tools are collecting?" I said I'd look into it. I wish I hadn't. What started as a quick audit turned into three weeks of increasingly unpleasant discoveries that made me question every AI tool we'd adopted.

I'm going to walk through what I found, what we did about it, and what you should check in your own environment. Some of this will be uncomfortable. Good. It should be.

The Audit Scope

We had 14 AI-powered tools in production use across the organization. That number surprised me — I'd expected maybe 8. The extras were tools that individual teams had adopted without going through procurement or security review. Shadow AI is real, and it's everywhere.

For each tool, I looked at four things: what data do employees put into it, what does the vendor's data retention policy actually say (not what sales told us — what the ToS says), what logs and telemetry does the tool collect, and is our data used for model training.

Finding #1: Customer PII in Prompt Logs

Our support team was using an AI assistant to help draft customer responses. Reasonable use case. But they were pasting entire customer tickets into the AI — including full names, email addresses, account numbers, and in several cases, the last four digits of credit cards. The AI tool's logging captured every prompt. That means customer PII was sitting in a third-party vendor's log storage, with a retention policy of "indefinitely unless you request deletion."

How much PII? I pulled the usage logs (the vendor provides admin access to see prompt history) and found approximately 2,300 prompts containing identifiable customer data over four months. That's 2,300 instances of customer PII stored on a server we don't control, subject to a privacy policy our customers never agreed to. Our legal team was not happy about this.

Finding #2: API Keys and Credentials in Conversations

Developers were using AI coding assistants to debug integration issues. Completely normal. Except when you're debugging an API integration, what do you paste into the AI? The API call. With the headers. Which contain the API key. Or the bearer token. Or the basic auth credentials encoded in base64 that anyone can decode in two seconds.

I found 47 instances of API keys or tokens in developer AI conversations over a three-month period. Eleven of those were production credentials. Three were for payment processing APIs. One was an AWS root account access key — I nearly fell out of my chair when I found that one.

We rotated every exposed credential immediately. The scary part was estimating the exposure window. Some of those API keys had been sitting in AI conversation logs for months. Did the vendor's employees have access to those logs? Could another customer's queries somehow surface our data? The vendor's answers to these questions were... unsatisfyingly vague.

Finding #3: Sensitive Architecture in a Training Dataset

This one was the worst. A member of our infrastructure team had been using a free-tier AI tool (not our approved one) to help design network configurations. He'd pasted firewall rules, VLAN layouts, internal IP ranges, and server roles into this tool over a period of two months. The free-tier tool's terms of service explicitly stated that inputs could be used for model training.

I can't prove that our specific network architecture ended up in a training dataset. But the ToS gave them the right to use it. And unlike PII — which at least might get filtered during training data processing — network configurations don't trigger privacy filters. There's no automated system that says "hey, this looks like someone's internal firewall rules, maybe we shouldn't train on it." It's just text.

We contacted the vendor about data deletion. Their response, paraphrased: once data enters the training pipeline, it can't be selectively removed. Lovely.

Finding #4: Meeting Transcription Data Going Places

We'd adopted an AI meeting transcription tool. Great productivity feature. But I reviewed the data handling agreement and found that meeting transcripts were being stored in the vendor's cloud for 90 days, processed for "service improvement" (which could mean training), and accessible to the vendor's support team "for troubleshooting purposes."

Our meetings include discussions about security vulnerabilities, incident details, employee performance, legal matters, and M&A activity. All of that was being transcribed, stored, and potentially accessible to vendor employees. Nobody had thought about this when we approved the tool, because the evaluation focused on features and price, not data handling.

Finding #5: Browser Extension Telemetry

Several employees had installed AI browser extensions for writing assistance, summarization, and research. These extensions — by design — read the content of every web page you visit so they can offer relevant suggestions. That includes internal wiki pages, HR portals, financial dashboards, and email. The extensions were sending page content back to their servers for processing.

One extension's privacy policy acknowledged collecting "page content for feature delivery" with a 30-day retention period. Thirty days of every internal page visited by employees using that extension, stored on a server operated by a startup with 15 employees. I have nothing against small companies, but that's a lot of sensitive data for a small attack surface to protect.

What We Did About It

First, we created an approved AI tool list. Only four tools survived the review. Each one had a data processing agreement, a clear data retention policy, an opt-out mechanism for training, and SOC 2 Type II certification (or equivalent). Everything else got blocked.

Second, we built a data classification guide specifically for AI usage. Green: public information, documentation, generic code samples. Yellow: internal procedures, non-sensitive configs, anonymized data. Red: PII, credentials, architecture details, incident data, anything related to legal or HR. Red category data cannot go into any AI tool, period. Yellow requires using our approved, enterprise-licensed tools only. Green is fair game.

Third, we deployed DLP monitoring on AI tool endpoints. Our web proxy now flags when someone pastes content matching PII patterns, credential patterns, or internal IP formats into any AI-hosted domain. It doesn't block (yet) — it alerts. In the first month, we caught 67 instances of employees pasting sensitive data into AI tools. Training is ongoing.

Fourth — and this was the hardest conversation — we disclosed the PII exposure to our legal team and, ultimately, to affected customers. Not a fun letter to write. But it was the right thing to do, and our legal team agreed that the exposure likely triggered notification requirements under two of the privacy regulations we're subject to.

Lessons for Your Audit

Don't assume your employees are being careful with what they put into AI tools. They're not. They're busy, they're trying to get their work done, and they're not thinking about data classification when they paste a customer record into a chatbot to help write a response. That's a training and tooling problem, not a people problem.

Read the actual terms of service and data processing agreements. Not the marketing page. The legal text. Pay special attention to: data retention periods, training data usage, employee access to customer data, and data deletion capabilities. If the vendor can't clearly answer "where does my data go and who can see it," that's your answer.

And do the audit soon. Every day you don't know what data is flowing into AI tools is a day that data might be going somewhere you don't want it. I put it off for months. I wish I hadn't. The mess would have been smaller if we'd looked sooner.