Security

Privacy

December 28, 2025 · 8 min read

AI Coding Assistants: A Security and Privacy Reality Check

AI coding assistants are wildly useful. They're also a new kind of risk surface. This post is a reality check. No doom, no hype, just how these systems actually behave, where data can flow, and how to use them responsibly.

TL;DRAI coding assistants are useful but carry real security and privacy risks. They're language models that can suggest insecure code, don't separate instructions from data (prompt injection), and may retain or train on your data depending on product tier. Use enterprise modes, minimize context, lock down permissions, and treat all output as untrusted until reviewed.

AI coding assistants are wildly useful. They're also a new kind of risk surface: part autocomplete, part chat, part "agent" that can read context, call tools and occasionally do something profoundly unhinged with your repo.

This post is a reality check. No doom, no hype, just how these systems actually behave, where data can flow and how to use them without quietly donating your company's secrets to the void.

I'm writing it from a particular seat. In the regulated Belgian work I do, tooling questions get settled by lawyers and security officers well before they reach an engineer, and the answer is often no. That is a frustrating place to sit until you notice what it does to you: when you have to justify a tool rather than just install it, you end up learning how the thing actually works, and most of what follows came out of having to explain it to someone who was going to sign for it.

What the thing actually is

An LLM is best understood as a text-to-text prediction machine: given input tokens, it predicts what tokens should come next. That is the core mechanism.

This matters because:

It doesn't "know" what's true. It produces plausible continuations, which can include made-up APIs, invented config flags, or confident-but-wrong security claims. I've seen it invent entire npm packages that don't exist.
It doesn't reliably separate instructions from data. If malicious instructions get into its context, whether that's a file, an issue, a PR description or a web page it was told to go read, the model can be tricked into following them. The UK NCSC calls this a "confused deputy" problem and argues prompt injection may never be fully solved.

Treat it like a power tool, then. Genuinely useful, and completely indifferent to what you point it at.

Where does your data go?

When you use an assistant, you are typically sending some combination of:

Your prompt ("please refactor this")
Code context (current file, surrounding lines, sometimes more)
Tool outputs (terminal logs, build errors, stack traces)
Metadata/telemetry (feature usage, clicks, accept/reject signals)

What happens next depends heavily on product tier + contract.

The vendor specifics below were checked against public documentation when this post was written, in December 2025. Data policies are some of the fastest-moving text on the internet. Treat what follows as the shape of the problem, not as current fact, and re-read the source before you take any of it into a security review.

OpenAI API vs consumer chat products: OpenAI's API policy has long emphasized that API data is not used to train models by default (unless you explicitly opt in). For business offerings, OpenAI also highlights retention controls and enterprise privacy commitments.

GitHub Copilot: GitHub's Copilot Business/Enterprise messaging is quite specific: in IDE chat/completions, prompts and suggestions are "not retained" (while user engagement data can be retained longer). GitHub also states it maintains a zero data retention agreement with some model providers in its stack.

Claude (consumer): Anthropic updated consumer terms to allow training on user chats/coding sessions if the user allows it and extended retention up to five years in that opt-in mode.

The pattern underneath all of it is simple enough: consumer tools drift toward "data helps improve the product", enterprise tools sell you a contractually enforced "no training, controlled retention" posture, and the gap between the two is exactly the gap a security reviewer will put their finger on. Read the fine print.

"We don't train on your data" and "we don't retain your data" are different claims. Watch how often they get conflated in a procurement meeting. In a regulated context the retention claim is usually the one that decides the outcome, because retention determines where the data physically sits, for how long, under whose jurisdiction and who can be compelled to produce it, and a training opt-out changes none of that.

Popular code isn't always good code

Coding assistants learn from oceans of public code. That means they're biased toward what's common, not what's correct.

Security is the perfect trap here. Insecure patterns are extremely common in the wild, they have been copied into tutorials and forum answers and internal wikis for as long as those things have existed, and a model optimized to produce the most plausible continuation will happily reproduce them unless you explicitly constrain it.

Classic example: string concatenation SQL

Vulnerable pattern (don't do this):

var sql = "SELECT * FROM Users WHERE Email = '" + email + "'";

This is the kind of thing assistants suggest all the time because it "looks normal." It's not. It's injection-prone: attacker-controlled input becomes executable query logic.

Safer pattern (parameterized):

var sql = "SELECT * FROM Users WHERE Email = @Email";
command.Parameters.Add("@Email", SqlDbType.VarChar, 256).Value = email;

Note the explicit type and length rather than AddWithValue. AddWithValue infers the type from the runtime value, so a .NET string arrives as nvarchar; against a varchar column that forces an implicit conversion and can stop the optimiser using the index. The query stays correct and quietly gets slow. It is also a neat example of this post's own point: AddWithValue is what assistants suggest because it is what the internet is full of.

Even here, you still need good habits (least privilege DB accounts, validation where appropriate, avoiding dynamic SQL in stored procedures), but parameterization is the foundational move.

The deeper point: the assistant is not a security oracle. It's an autocomplete engine with vibes, and the vibes were trained on whatever was most common in public code.

The work moves from writing to reviewing

AI assistants don't remove engineering work; they move it:

From typing code to designing constraints
From implementation to review, verification and testing
From "how do I write this" to "how do I know this is safe and correct"

That suits people who have already been burned often enough to smell nonsense at a glance. It's much harder on everyone else, and juniors carry most of that cost, because what the assistant produces is clean-looking wrongness: wrong edge cases, insecure defaults, subtle concurrency bugs, tests that pass without asserting anything meaningful. It reads like something a competent person wrote. It compiles. It's still wrong.

So the real skill becomes judgment:

Is this correct in our system?
Is this safe under our threat model?
What did it assume that isn't true here?

The assistant writes the code and you still own the outcome, which in a regulated sector means owning it in front of an auditor who will ask how the code got there and will not find "the tool suggested it" a satisfying answer.

Two fears people keep mixing up

Training data extraction, or model memorization

Carlini and colleagues demonstrated this at USENIX Security in 2021: targeted querying of GPT-2 pulled back verbatim training examples, personal data among them, out of a model nobody had asked to memorize anything in particular. The EDPB's report on AI privacy risks and mitigations for large language models puts memorization among the risks it expects to see actively mitigated rather than filed under academic curiosity, which matters a great deal on the day you have to defend a tool choice to a DPO.

For coding assistants, that raises questions like: Could proprietary code end up in training data? Could it later be extracted in fragments? Enterprise "no training" terms help, but you still care about retention, logging and who can access transcripts.

Prompt injection plus tool access

As assistants become more agentic (reading repos, browsing issues, running tools), attackers can try to trick the assistant into leaking data it already has access to. This is no longer theoretical.

Unit 42 at Palo Alto Networks has written up indirect prompt injection against code assistants specifically, where the hostile instructions ride in on the external context the assistant was told to go and read. Docker's writeup of the GitHub MCP prompt injection is the version I'd send to a skeptical tech lead, because nothing in the chain is a vulnerability in the classic sense: a malicious issue sitting in a public repository, an assistant holding a token broad enough to reach private ones, and content quietly crossing a boundary it was never supposed to cross.

The NCSC's position, in the post arguing that prompt injection is not SQL injection, is that you shouldn't sit waiting for the patch that makes this go away. Design for impact reduction instead.

The biggest near-term risk isn't "the model becomes Skynet." It's "the model gets socially engineered while holding your GitHub token." That's the one that keeps me up.

What I actually do about it

Classification first, and the default is no secrets. Treat every prompt as potentially logged, indefinitely, by people you will never meet. No credentials, no private keys, no tokens, no customer data or PII, no proprietary algorithms. If it's sensitive, summarize or redact it before it goes anywhere near a text box.

Then the tier. If your organization is serious about this, use products where "no training" is a contract term and retention is configurable, and read the processing agreement rather than the marketing page. For personal data and regulated content it frequently still isn't enough, and the tool has to run somewhere you control. I've written separately about where each class of data is allowed to live.

Then minimize what it can see. Don't hand it the whole repo by default. Scope it to the single file, the minimal folder, the smallest reproduction. The less context it ingests, the less there is to leak.

Then lock down what it can do. If the assistant can act, through agents, MCP tooling or direct repo access, give it least-privilege tokens and repo-scoped access, and keep org-wide credentials well away from it. Broad tokens plus prompt injection is the combination that produces the incidents.

After that it's ordinary engineering discipline. Output is untrusted until proven otherwise, so it clears the same bar as any other PR: tests on the happy and unhappy paths, security review for anything touching auth, data access or SQL, and a sanity check on every dependency it suggested, in case one of them turns out not to exist. And automate the boring paranoia where you can, with secret scanning in pre-commit and CI, SAST rules for injection patterns, redaction in the logging pipeline, and public code matching filters where the product offers them.

That is simply the posture you would take toward any third party you handed a key to, which is exactly what an AI assistant with tool access is.