Long-running agents: the long and short of it -- Daniel Khashabi

Value of the Information & Strength of the Argument

The talk provides valuable insights into the limitations of context scaling, backed by a controlled experiment that isolates the effect of training context length. The ‘information abundance paradox’ is a novel and thought-provoking finding, with clear implications for model training. The argumentation is logical, moving from observed trends to a hypothesis, then to experimental validation. The second part, self-compaction, offers a practical and simple solution that outperforms baselines, with a detailed explanation of its design and computational efficiency. The speaker acknowledges the simplicity of the approach and highlights potential for improvement. The overall argument is convincing and well-supported by preliminary results, though the sample of benchmarks is limited.

Scientific Rigor, Source Quality, Title Accuracy

The scientific rigor is high: the speaker describes a controlled experiment with fixed data and compute, varying only the training context window. He also compares multiple baselines and analyzes the effect of problem difficulty. However, no external sources are cited explicitly; the talk relies on the speaker’s own research and mentions benchmarks like MIMO and Browse without providing references. The title is perfectly aligned with the content, and the talk stays on topic throughout. The absence of citations does not undermine the internal validity of the presented work, but external verification of claims would strengthen confidence.

The title accurately reflects the content: it discusses long-running agents, focusing on two main strategies — scaling context ('long') and managing bounded context via compaction ('short'). The subtitle 'the long and short of it' is a fitting play on words.

8/10

The talk is presented by a researcher at JHU, based on original experiments conducted by his team. The methodology is transparent and includes controlled experiments on context scaling and a new compaction algorithm. However, the results are not yet peer-reviewed in this format, and some claims rely on specific benchmarks that may not be representative. Overall, the scientific rigor is high but with expected limitations of a conference-style presentation.

Markers derived by PSI from the transcript: the creator did not define chapters.

• 00:00Introduction to current state of agents and the problem of isolated conversations.
• 03:00First bet: scaling context window; discussion of cost and data barriers.
• 07:00Thought experiment on context scaling and the information abundance paradox.
• 10:00Experimental results showing degradation beyond optimal context length.
• 14:00Second bet: bounded context management; definition of compaction and its flaws.
• 18:00Introduction of self-compaction with rubric prompts and cost analysis.
• 23:00Evaluation results on MIMO v2 comparing no compaction, fixed interval, and self-compaction.
• 29:00Conclusion: advocating for context management and metacognitive abilities.

• Project Gutenberg — Source of long documents used in the context scaling experiment.
• Browse benchmark — Mentioned as a benchmark for evaluating long-running agents.
• MIMO v2 — Benchmark used for evaluating self-compaction performance.