Lecture VIII — Your Research Project
AI-Assisted Programming for PhD Researchers
Research Code Is Different
The Artifact Is the Claim
In research, code correctness is result validity.
. . .
- A bug here is not a crash but a wrong Table 1
- Remember Lab 3: one silent
fillnashifted every summary statistic - Nobody sees a stack trace; they see a number that looks fine and is not
- The code and the result stand or fall together
Reproducibility Checklist
Four things make a result someone else can rerun:
- Pinned environment —
uv lock,renv, orProject.toml - Fixed seeds — every random draw is deterministic
- Data versioned or archived — findable and persistent (Wilkinson et al. 2016)
- One command from raw data to every figure
Notebooks: Handle with Care
- Hidden state and out-of-order execution quietly wreck reproducibility
- Of ~1.16M public notebooks Pimentel et al. mined, only ~4% reproduced their own saved output (Pimentel et al. 2019)
- 36% ran cells out of order — a contributing cause, not the failure rate itself (Pimentel et al. 2019)
- Fine for exploration; the record should be a pipeline — like the one you built
Agents and Reproducibility
Agents are good at exactly this cleanup work:
- “Make this run end-to-end from scratch in a clean environment”
- “Pin my dependencies and generate a lock file”
- “Find the hidden state in this notebook and flag out-of-order cells”
AI Beyond Code
Literature Research with Agents
The deep-research pattern: search → read → synthesize with citations.
. . .
- Point an agent at a question; it fans out, reads, and summarizes with sources
- Genuinely useful for mapping an unfamiliar field fast
- A strong starting map — not a finished bibliography
The Catch: Invented Citations
Models produce references that are confident, well-formatted, and fake:
- 55% of GPT-3.5’s citations were fabricated; GPT-4 still 18% (Walters and Wilder 2023)
- Even the real ones carried errors — 43% (GPT-3.5) / 24% (GPT-4) (Walters and Wilder 2023)
- Newer is better, not safe: GPT-4o still ~20% fabricated, worst on niche topics (Linardon et al. 2025)
Spot the Fake
Same format, same confidence — one exists, one does not:
Fabricated: Smith, J. (2019). "Deep learning for
epidemic forecasting." Nature Methods, 16(4), 331–339.
Real: Pimentel et al. (2019). "A large-scale study
about quality and reproducibility of Jupyter notebooks."
MSR 2019, 507–517.You cannot tell from the text — so every citation gets verified.
The Verification Discipline
For every AI-suggested source, three checks:
- DOI resolves — the link actually goes somewhere
- Paper exists — real venue, real authors, real year
- It says what was claimed — read the abstract, minimum
. . .
AI does not remove the work; it moves it — from finding to verifying (Lee et al. 2025). Yesterday’s Lab 5 rule, now for your thesis.
Drafting Text with Skills
A clunky sentence through conciseness + academic-grammar:
Before: It should be noted that, in order to obtain the
results, we made use of a large number of samples.
After: We used a large sample.The skills suggest; you decide what stays.
Cross-Check Everything
AI drafts are drafts: verify claims sentence by sentence, numbers against sources, citations against papers.
. . .
The deeper risk is producing more while understanding less — “illusions of understanding” (Messeri and Crockett 2024). You sign it, you own it.
Institutions Are Responding
- AI agents in research are now a policy topic — journals, funders, and institutions are writing the rules (“AI Scientists Are Changing Research — Institutions, Funders and Publishers Must Respond” 2026)
- The disclosure and verification habits from Lecture VI keep you ahead of them
- Do the honest thing now; the requirements are only catching up
Your Capstone
The Assignment
Tomorrow you build. Today you scope.
. . .
- Deliverable: a working demo plus a presentation, Friday afternoon
- Judged on the four points: demo works, process reflection, honest challenges, transferable insights
- The same four you have seen all week — no surprises
What Makes a Good Capstone
- Comes from your own research, a real problem you have
- Core is finishable in ~5 hours, with stretch goals beyond it
- Has a verifiable output — a figure, a table, a passing test suite
- Uses the full workflow: spec → plan → implement → verify
No Project? Option A: Literature Manager
A BibTeX/DOI tool built from the Lab 5 pieces:
- Parse a messy
.bibor a list of DOIs - Fetch metadata via the MCP server
- Deduplicate, then emit a clean reading list
- Full brief on the capstone page
No Project? Option B: Parameter Study
A small simulation you sweep and plot:
- Pick a model — a queue, an epidemic, your choice
- Sweep two parameters over a grid
- Produce result plots and a findings table
- Full brief on the capstone page
Scope It Now
- Open the capstone page and write your spec with the interview pattern
- I circulate while you work
- By 16:30: spec + plan reviewed by a fresh agent session or a neighbor
Continue Your Journey
Next Up
- You now know why research code carries a higher bar, and how to keep AI honest in it
- Tomorrow, 09:00: build day — bring coffee
- The Capstone Project — scope yours this afternoon
- Lecture IX — Judgment and Presenting — tomorrow morning
- Course literature and references
References
Lee, Hao-Ping (Hank), Advait Sarkar, Lev Tankelevitch, et al. 2025. “The Impact of Generative AI on Critical Thinking: Self-Reported Reductions in Cognitive Effort and Confidence Effects From a Survey of Knowledge Workers.” Proceedings of the 2025 CHI Conference on Human Factors in Computing Systems (Yokohama Japan), April, 1–22. https://doi.org/10.1145/3706598.3713778.
Linardon, Jake, Hannah K Jarman, Zoe McClure, Cleo Anderson, Claudia Liu, and Mariel Messer. 2025. “Influence of Topic Familiarity and Prompt Specificity on Citation Fabrication in Mental Health Research Using Large Language Models: Experimental Study.” JMIR Mental Health 12 (November): e80371. https://doi.org/10.2196/80371.
Messeri, Lisa, and M. J. Crockett. 2024. “Artificial Intelligence and Illusions of Understanding in Scientific Research.” Nature 627 (8002): 49–58. https://doi.org/10.1038/s41586-024-07146-0.
Pimentel, Joao Felipe, Leonardo Murta, Vanessa Braganholo, and Juliana Freire. 2019. “A Large-Scale Study About Quality and Reproducibility of Jupyter Notebooks.” 2019 IEEE/ACM 16th International Conference on Mining Software Repositories (MSR) (Montreal, QC, Canada), May, 507–17. https://doi.org/10.1109/MSR.2019.00077.
Walters, William H., and Esther Isabelle Wilder. 2023. “Fabrication and Errors in the Bibliographic Citations Generated by ChatGPT.” Scientific Reports 13 (1): 14045. https://doi.org/10.1038/s41598-023-41032-5.
Wilkinson, Mark D., Michel Dumontier, IJsbrand Jan Aalbersberg, et al. 2016. “The FAIR Guiding Principles for Scientific Data Management and Stewardship.” Scientific Data 3 (1): 160018. https://doi.org/10.1038/sdata.2016.18.