Lecture VI: Working Safely with AI Code

AI-Assisted Programming for PhD Researchers

Author
Affiliation

Dr. Tobias Vlćek

Helmut Schmidt University, August 2026

The Numbers

How Bad Is It?

Independent audits keep landing in the same place.

. . .

  • About 45% of AI-generated samples carry OWASP-class vulnerabilities (80 tasks across 100+ models) (Veracode 2025)
  • AI-authored pull requests: ~1.7× more issues overall, up to 2.74× for security issues (correlational, n=470 PRs) (Loker 2025)

And It Replicates

  • This is not a new-model glitch: ~40% of Copilot programs were already vulnerable in the 2022 academic baseline (Pearce et al. 2022)
  • Three years, different tools: the rate barely moves

Why?

  • Trained on public code, including the insecure half
  • Optimized for plausible, not for safe
  • No adversarial mindset: it does not ask “who attacks this?”
  • Your context rarely says “this will face the internet”

And You’ll Feel Safer Anyway

  • Stanford ran a randomized trial with and without an AI assistant

. . .

  • Participants with the assistant wrote less secure code, and rated it more secure than the control group did (Perry et al. 2023)
  • Overconfidence is the failure mode; the review habits below are the counter

Security Failure Modes

Injection

Plausible, passes tests, ships a hole. Spot the problem:

def get_measurements(conn, station):
    query = f"""
        SELECT * FROM readings
        WHERE station = '{station}'
    """
    return conn.execute(query).fetchall()

. . .

The station value is pasted straight into SQL. Pass it as a parameter (?), never an f-string.

Secrets in Code

  • Hardcoded API keys, tokens, and passwords in tracked files
  • Agents echo the patterns they saw in training, including bad ones
  • AI-assisted commits leak secrets at 3.2% vs a 1.5% baseline (Nabiullina and Winqwist 2026)

The Secrets Rule

Secrets live in environment variables or your keychain, never in files the agent writes, and never pasted into a prompt.

Hallucinated Dependencies

  • The agent imports a package that does not exist
  • Squatters register those invented names: “slopsquatting”
  • Check every new dependency it adds: real? maintained? license?

The Quiet Ones

  • Path traversal: unsanitized filenames escaping a directory
  • Unsafe eval or pickle on data you did not create
  • Missing input validation on the boring, trusting path

You do not memorize these. You build review habits and checks.

Prompt Injection

  • Untrusted content the agent reads (a webpage, a tool result, a file) can carry instructions it then obeys
  • The term was coined in 2022 (Willison 2022)
  • The rule: never combine the lethal trifecta: private data + untrusted content + external communication (Willison 2025)

Hallucination Risks

Confidently Wrong

  • Invented APIs, wrong parameter names, outdated idioms
  • Delivered in the same confident tone as correct code
  • Training cutoffs mean the model’s world is months old

Countermeasures

  • Verify against real docs. You get a way to check claims against real sources this afternoon
  • Run the code; a claim is not a result
  • Prefer boring, well-documented libraries over the clever new one

Licensing and IP

Who Owns AI Output?

  • Legally unsettled, and it depends on your jurisdiction
  • Courts and institutions are still deciding
  • Practical stance: treat it as your code and your responsibility

Three Practical Rules

  • Check the licenses of any dependencies it pulls in
  • Do not paste licensed code in as context and ask for a “rewrite”
  • Know your institution’s policy. It may already bind you

Reviewing AI Code

Read Every Diff

  • Non-negotiable: every line, every time
  • git add -p turns review into a ritual, hunk by hunk
  • If the diff is too big to review, the step was too big: redo it smaller

Writer and Reviewer

  • A fresh agent session reviewing the diff has no attachment to the code
  • Review against the spec, not your taste. You arbitrate
  • A checklist you are handed does not transfer; it has to be practiced (Bouvier et al. 2025)
  • That practice is exactly what Lab 4 is

Review vs Tests: What Each Catches

Tests catch Review catches
Behavior regressions Design smells
Broken edge cases Security holes
Wrong outputs Silent data handling

You need both: neither substitutes for the other.

Your Data

What Leaves Your Machine

  • Every prompt you type
  • Every file the agent reads
  • Every command output it sees

All of it goes to the model provider’s API. Plain fact, not a scare.

Where It Goes Matters

  • Jurisdiction, retention, and whether it trains future models
  • Our stack: Mistral, EU-hosted, training opt-out. You set that in setup, check it now
  • Zed-hosted models run in the US: know which you are calling

Unpublished Research Data

Before you send it, ask:

  • Personal data under GDPR?
  • Under embargo or an NDA?
  • Owned by a third party?

Any “yes” → use a synthetic sample or a local model. Your data-management plan may already answer this.

Rules of Thumb

Lab data is synthetic, so send it freely. Your own data: check first, sample small, anonymize. Secrets and credentials: never.

Academic Integrity

Disclosure Is the Norm Now

The Rules Agree Everywhere

Could Anyone Tell?

  • Suppose you skip disclosure, could anyone prove a model wrote your text?
  • Since August 2026: probably, yes
  • The idea behind this is not very complicated, so let’s see how it works

How a Watermark Hides in Plain Text

  • Before each word, the model secretly splits its vocabulary into a green and a red half which is reshuffled at every word
  • It then softly prefers green words; the text still reads normally
  • By chance, you write ~half green words, but the model writes far more of these

Who Can Actually Detect It

  • Detection is simply counting: one paragraph can push chance below \(10^{-13}\) (Kirchenbauer et al. 2023)
  • But only if you know the secret key that decided which half was green, so in practice only the provider can run it

When Did the Watermark Appear?

  • EU AI Act, Article 50: since 2 August 2026 providers must mark AI content machine-readably (European Commission 2026)
  • Claude models released since then watermark all text output; detection tooling is still forthcoming from Anthropic (Anthropic 2026)
  • Google has watermarked Gemini’s text output with SynthID-Text since 2024 (Dathathri et al. 2024)

What a Mark Proves (and Does Not)

  • Proves: a model processed this text, but not who did the work and not whether a human verified it
  • Heavy editing or paraphrasing can wash it out; short snippets may not carry it, hence an absence proves nothing
  • Presence without disclosure, though, looks exactly like what it is

How to Disclose

  • In methods or acknowledgments: which tools, for what, and that a human verified the output
  • Keep your prompts and session logs: they are your lab notebook

In This Course

  • When you show your project on Day 3, say which tool you used and for what
  • It is part of the work, not a confession

Lab 4

Your Mission

In Lab 4 the agent works fast and unsupervised on a branch:

  • It builds three features without asking you questions
  • You find what is wrong with them before they reach main
  • Then you earn the merge through review, not vibes

Lab 4: Review and Explain

Continue Your Journey

Next Up

References

ALLEA - All European Academies. 2023. The European Code of Conduct for Research Integrity. ALLEA - All European Academies. https://doi.org/10.26356/ECoC.
Anthropic. 2026. “How Claude Marks AI-Generated Content.” In Claude Help Center. https://support.claude.com/en/articles/16266773-how-claude-marks-ai-generated-content.
Bouvier, Dennis J., Bruno Pereira Cipriano, Richard Glassey, et al. 2025. “The Rest of the Robots: Generative AI in Post-Introductory Computing Education.” Proceedings of the 2025 Working Group Reports on Innovation and Technology in Computer Science Education (Nijmegen Netherlands), June, 61–107. https://doi.org/10.1145/3760545.3783970.
Committee on Publication Ethics. 2023. Authorship and AI Tools. Committee on Publication Ethics (COPE). https://publicationethics.org/guidance/cope-position/authorship-and-ai-tools.
Dathathri, Sumanth, Abigail See, Sumedh Ghaisas, et al. 2024. “Scalable Watermarking for Identifying Large Language Model Outputs.” Nature 634: 818–23. https://doi.org/10.1038/s41586-024-08025-4.
Deutsche Forschungsgemeinschaft. 2023. Statement of the DFG Executive Committee on the Influence of Generative Models for Text and Image Creation on Science and the Humanities and on the DFG’s Funding Activity. Deutsche Forschungsgemeinschaft (DFG). https://www.dfg.de/download/pdf/dfg_im_profil/geschaeftsstelle/publikationen/stellungnahmen_papiere/2023/230921_statement_executive_committee_ki_ai.pdf.
European Commission. 2024. Living Guidelines on the Responsible Use of Generative AI in Research. European Commission, Directorate-General for Research; Innovation. https://research-and-innovation.ec.europa.eu/document/download/2b6cf7e5-36ac-41cb-aab5-0d32050143dc_en?filename=ec_rtd_ai-guidelines.pdf.
European Commission. 2026. Guidelines on Transparency Obligations for Providers and Deployers of Certain AI Systems. https://digital-strategy.ec.europa.eu/en/policies/guidelines-transparency-ai-generated-content.
International Committee of Medical Journal Editors. 2025. Recommendations for the Conduct, Reporting, Editing, and Publication of Scholarly Work in Medical Journals: Defining the Role of Authors and Contributors. International Committee of Medical Journal Editors (ICMJE). https://www.icmje.org/recommendations/browse/roles-and-responsibilities/defining-the-role-of-authors-and-contributors.html.
Kirchenbauer, John, Jonas Geiping, Yuxin Wen, Jonathan Katz, Ian Miers, and Tom Goldstein. 2023. “A Watermark for Large Language Models.” Proceedings of the 40th International Conference on Machine Learning, 17061–84. https://proceedings.mlr.press/v202/kirchenbauer23a.html.
Loker, David. 2025. “AI Vs Human Code Gen Report: AI Code Creates 1.7x More Issues.” In CodeRabbit. https://coderabbit.ai/blog/state-of-ai-vs-human-code-generation-report.
Nabiullina, Anna, and Carole Winqwist. 2026. “The State of Secrets Sprawl 2026: AI-Service Leaks Surge 81% and 29M Secrets Hit Public GitHub.” In GitGuardian Blog - Take Control of Your Secrets Security. https://blog.gitguardian.com/the-state-of-secrets-sprawl-2026/.
Pearce, Hammond, Baleegh Ahmad, Benjamin Tan, Brendan Dolan-Gavitt, and Ramesh Karri. 2022. “Asleep at the Keyboard? Assessing the Security of GitHub Copilot’s Code Contributions.” 2022 IEEE Symposium on Security and Privacy (SP) (San Francisco, CA, USA), May, 754–68. https://doi.org/10.1109/SP46214.2022.9833571.
Perry, Neil, Megha Srivastava, Deepak Kumar, and Dan Boneh. 2023. “Do Users Write More Insecure Code with AI Assistants?” Proceedings of the 2023 ACM SIGSAC Conference on Computer and Communications Security (Copenhagen Denmark), November, 2785–99. https://doi.org/10.1145/3576915.3623157.
Thorp, H. Holden. 2023. “ChatGPT Is Fun, but Not an Author.” Science 379 (6630): 313–13. https://doi.org/10.1126/science.adg7879.
“Tools Such as ChatGPT Threaten Transparent Science; Here Are Our Ground Rules for Their Use.” 2023. Nature 613 (7945): 612–12. https://doi.org/10.1038/d41586-023-00191-1.
Veracode. 2025. 2025 GenAI Code Security Report. Veracode. https://www.veracode.com/wp-content/uploads/2025_GenAI_Code_Security_Report_Final.pdf.
Willison, Simon. 2022. “Prompt Injection Attacks Against GPT-3.” In Simonwillison.net. https://simonwillison.net/2022/Sep/12/prompt-injection/.
Willison, Simon. 2025. “The Lethal Trifecta for AI Agents: Private Data, Untrusted Content, and External Communication.” In Simonwillison.net. https://simonwillison.net/2025/Jun/16/the-lethal-trifecta/.