Timothy De Bock | Freelance .NET & Azure Software Engineer

Security

Privacy

December 28, 2025 · 8 min read

AI Coding Assistants: A Security and Privacy Reality Check

TL;DR: AI coding assistants are useful but carry real security and privacy risks. They're language models that can suggest insecure code, don't separate instructions from data (prompt injection), and may retain or train on your data depending on product tier. Use enterprise modes, minimize context, lock down permissions, and treat all output as untrusted until reviewed.

AI coding assistants are wildly useful. They're also a new kind of risk surface: part autocomplete, part chat, part "agent" that can read context, call tools, and occasionally do something profoundly unhinged with your repo.

This post is a reality check. No doom, no hype, just how these systems actualy behave, where data can flow, and how to use them without quietly donating your company's secrets to the void.

1. It's a Language Model, Not "Intelligence"

An LLM is best understood as a text-to-text prediction machine: given input tokens, it predicts what tokens should come next. That's not an insult, it's the core mechanism.

This matters because:

It doesn't "know" what's true. It produces plausible continuations, which can include made-up APIs, invented config flags, or confident-but-wrong security claims. I've seen it invent entire npm packages that don't exist.
It doesn't reliably separate instructions from data. If malicious instructions get into its context (a file, an issue, a PR description, a web page), the model can be tricked into following them. The UK NCSC calls this a "confused deputy" problem and argues prompt injection may never be fully solved.

So: treat it like a power tool. A nail gun doesn't "understand carpentry," but it sure can remodel your foot.

2. Where Does Your Data Go?

When you use an assistant, you are typically sending some combination of:

Your prompt ("please refactor this")
Code context (current file, surrounding lines, sometimes more)
Tool outputs (terminal logs, build errors, stack traces)
Metadata/telemetry (feature usage, clicks, accept/reject signals)

What happens next depends heavily on product tier + contract.

OpenAI API vs consumer chat products: OpenAI's API policy has long emphasized that API data is not used to train models by default (unless you explicitly opt in). For business offerings, OpenAI also highlights retention controls and enterprise privacy commitments.

GitHub Copilot: GitHub's Copilot Business/Enterprise messaging is quite specific: in IDE chat/completions, prompts and suggestions are "not retained" (while user engagement data can be retained longer). GitHub also states it maintains a zero data retention agreement with some model providers in its stack.

Claude (consumer): Anthropic updated consumer terms to allow training on user chats/coding sessions if the user allows it, and extended retention up to five years in that opt-in mode.

This is the broader pattern: consumer tools drift toward "data helps improve the product"; enterprise tools sell you a contractually-enforced "no training / controlled retention" posture. Read the fine print.

"We don't train on your data" and "we don't retain your data" are different claims. I've seen people confuse these. Treat them as separate checkboxes in your threat model.

3. Popular Code Isn't Always Good Code

Coding assistants learn from oceans of public code. That means they're biased toward what's common, not what's correct.

Security is the perfect trap here, because insecure patterns are extremely comon in the wild. And the model may happily reproduce them unless you explicitly constrain it.

Classic example: string concatenation SQL

Vulnerable pattern (don't do this):

var sql = "SELECT * FROM Users WHERE Email = '" + email + "'";

This is the kind of thing assistants suggest all the time because it "looks normal." It's not. It's injection-prone: attacker-controlled input becomes executable query logic.

Safer pattern (parameterized):

var sql = "SELECT * FROM Users WHERE Email = @Email";
command.Parameters.AddWithValue("@Email", email);

Even here, you still need good habits (least privilege DB accounts, validation where appropriate, avoiding dynamic SQL in stored procedures), but parameterization is the foundational move.

The deeper point: the assistant is not a security oracle. It's an autocomplete engine with vibes. Sometimes dangerous vibes.

4. Experience Still Matters: Writing to Reviewing

AI assistants don't remove engineering work; they move it:

From typing code to designing constraints
From implementation to review, verification, and testing
From "how do I write this" to "how do I know this is safe/correct"

That's great for senior devs who can smell nonsense. Less great for everyone else.

Juniors are at real risk here. The assistant produces clean-looking wrongness: wrong edge cases, insecure defaults, subtle concurrency bugs, tests that "pass" but don't assert anything meaningful. It looks professional. It compiles. It's still wrong.

So the real skill becomes judgment:

Is this correct in our system?
Is this safe under our threat model?
What did it assume that isn't true here?

In other words: the assistant can write code, but it can't own outcomes. You still do.

5. The Future Risk: Targeted Extraction

There are two overlapping fears people mix together:

A) Training data extraction (model memorization)

Research has shown that large language models can sometimes be induced to output memorized training examples via targeted querying. Regulators and privacy bodies explicitly call out memorization as a privacy risk area that needs mitigation.

For coding assistants, that raises questions like: Could proprietary code end up in training data? Could it later be extracted in fragments? Enterprise "no training" terms help, but you still care about retention, logging, and who can access transcripts.

B) Prompt injection + tool access

As assistants become more "agentic" (reading repos, browsing issues, running tools), attackers can try to trick the assistant into leaking data it has access to. This is no longer theoretical.

Security researchers describe indirect prompt injection risks in code assistants when external context is ingested. Real-world writeups show how overly-broad tokens + AI workflows can enable cross-repo or sensitive-data exfiltration when the assistant is manipulated through seemingly innocent content.

The NCSC's position is basically: don't expect a magic patch, design for impact reduction.

Put simply: the biggest near-term risk isn't "the model becomes Skynet." It's "the model gets socially engineered while holding your GitHub token." That's the one that keeps me up.

6. Practical Guidelines: Six Concrete Rules

Rule 1: Classify what you feed it (and default to "no secrets")

Treat prompts as potentially logged. Never paste credentials, private keys, tokens, customer data/PII, or proprietary algorithms. If it's sensitive, summarize or redact first.

Rule 2: Prefer enterprise/API modes with explicit controls

If your org is serious about this, use products where "no training" is contractual and retention is configurable.

Rule 3: Minimize what the assistant can see

Don't give it the whole repo by default. Scope to the single file, the minimal folder, the smallest reproduction. The less context it ingests, the less it can leak.

Rule 4: Lock down tool permissions

If the assistant can act (agents, MCP/tooling, repo access): use least-privilege tokens, repo-scoped access where possible, avoid "org-wide" credentials. Prompt injection + broad tokens is a nasty combo.

Rule 5: Assume output is untrusted until proven otherwise

Require the same bar as any PR: tests (happy + unhappy paths), security review for auth/data/SQL changes, dependency sanity checks (avoid hallucinated packages).

Rule 6: Build guardrails into your pipeline

Automate the boring paranoia: secret scanning (pre-commit + CI), SAST rules for injection patterns, logging redaction, allow/block public code matching features where available.