CORE Seminar - Sept. 28 2026 - Dr. Dan Poenaru

Value of the Information & Strength of the Argument

The talk offers substantial practical value for clinical researchers navigating LLM adoption. Its core argument is that the utility of LLMs is governed by the cost of verifying their output relative to the task’s value, a framing that is both intuitive and actionable. The speaker supports this with concrete examples: low-friction tasks like reformatting a CV or finding literature yield clear time savings, while high-friction tasks like data interpretation or manuscript writing require careful human review. He grounds his claims in published studies, such as a 2026 Journal of Clinical Epidemiology review of 63 studies on AI-assisted systematic reviews and a Journal of Evidence-Based Medicine review of 18 studies, reporting high agreement (0.98–0.99) and dramatic time reductions (500 to 5–25 hours). He also cites a Lancet 2026 analysis of 2.5 million papers and 125 million references, finding a 0.57% fabrication rate, and notes that fabrication is higher for obscure topics. The argumentation is largely coherent and self-critical: he acknowledges his own use of AI in writing, admits uncertainty about future risks, and distinguishes between his expertise and areas beyond it. However, some claims are anecdotal (e.g., his son’s interview process, agent swarms invading Australia) and lack citations, and the evidence on deskilling is referenced only generally. The overall argument is persuasive but relies on a mix of robust literature and personal observation.

Scientific Rigor, Source Quality, Title Accuracy

The speaker demonstrates scientific rigor by citing specific journals and studies, including the Journal of Clinical Epidemiology, Journal of Evidence-Based Medicine, Lancet, and JMIR. He is careful to note limitations, such as the lack of a ground truth in systematic reviews and the difficulty of attributing citation fabrication to AI versus human error. He also acknowledges that his quantitative figures are approximate and that he checked the journals he cites. However, the presentation lacks a formal reference list, and some sources are mentioned only in passing (e.g., the Lancet gastroenterology study on colonoscopy deskilling). The title, ‘From Spell Checker to Co-investigator: Working with large language models in clinical research,’ accurately reflects the content, which spans basic to advanced LLM applications in research. The talk does not overpromise; it delivers a balanced view of capabilities and risks. The absence of defined chapters means the audience must rely on the speaker’s structure, which is clear but informal.

The title accurately reflects the content: the talk moves from basic LLM use (spell-checker level) to advanced co-investigator workflows in clinical research, matching the announced topic.

7/10

The talk is delivered by a credentialed clinician-researcher (Professor of Surgery and Pediatrics, McGill; senior scientist at RI-MUHC; associate academic member at Mila) and is grounded in his own lab's experience and in named journal literature. However, several quantitative claims are delivered from memory or paraphrased without precise citations, some dates and figures appear imprecise, and the presentation is a seminar-style expert opinion rather than a systematic review. The reliance on a small number of cited studies and on personal anecdote limits verifiability, though the speaker is transparent about uncertainty and about the limits of his own knowledge.

Markers derived by PSI from the transcript: the creator did not define chapters.

• 00:00Introduction by Ben Smith, outlining Dr. Poenaru's credentials and research focus.
• 03:00Audience poll on LLM usage, revealing widespread adoption and experiences with confident errors.
• 10:00Contrast between software engineering (free verification via compiler) and medicine (costly verification).
• 15:00Evolution of LLMs to agentic systems, with agents planning, executing, and self-checking tasks.
• 20:00Discussion of risks: agent swarms, automation bias, deskilling, and biosecurity.
• 30:00Evidence on LLM performance in systematic reviews: high agreement for screening, weaker for risk of bias.
• 40:00Citation fabrication rates from a Lancet 2026 study, with higher rates for obscure topics.
• 50:00Multi-agent 'virtual lab' systems that generate research with minimal human writing.
• 1:00:00Lab framework based on 'friction': low for ideation/literature, high for analysis/writing, adversarial review for grants.

• Journal of Clinical Epidemiology 2026 study on AI in systematic reviews — Review of 63 studies using AI for systematic reviews, reporting high agreement for screening and data extraction.
• Journal of Evidence-Based Medicine study on AI in systematic reviews — Review of 18 studies showing sensitivity/specificity of 0.98–0.99 and time savings from 500 to 5–25 hours.
• Lancet 2026 study on citation fabrication — Analysis of 2.5 million papers and 125 million references, finding a 0.57% fabrication rate.
• JMIR study on citation fabrication in mental health — Comparison of common vs. rare topics, showing higher fabrication for obscure topics.
• Nature 2026 multi-agent systems publications — Three separate multi-agent systems (e.g., virtual lab, co-scientist) that automate research tasks.

• Journal of Clinical Epidemiology 2026 study — Supports the claim that AI performs well in systematic review screening and data extraction.
• Journal of Evidence-Based Medicine study — Corroborates high sensitivity/specificity and time savings in systematic reviews.
• Lancet 2026 citation fabrication study — Provides quantitative evidence on the low but non-zero rate of fabricated citations.

• Anecdotal reports of high citation fabrication in grant reviews — The speaker's colleague reported a case where all references in a grant were fabricated, contrasting with the low 0.57% rate from the Lancet study; this suggests variability by model, prompt, or context.