Resource library

Nicholas Roberts presents Train-to-Test (T²) scaling laws, which jointly optimize model size, training tokens, and test-time samples—and show why reasoning models should be overtrained well beyond Chinchilla-optimal ratios.

Text-to-optimization requires two separable capabilities: modeling — choosing the right optimization structure — and binding — grounding every coefficient, index, and parameter in the concrete problem data. We study this via Text2Opt-Bench, a scalable benchmark of solver-verified optimization problems spanning 12 categories, from textbook linear programs to stochastic and multi-objective formulations with up to thousands of variables. Across 10+ models,…

Modern LLMs scale at test-time, e.g. via repeated sampling, where inference cost grows with model size and the number of samples. This creates a trade-off that pretraining scaling laws, such as Chinchilla, do not address. We present Train-to-Test (T2) scaling laws that jointly optimize model size, training tokens, and number of inference samples under fixed end-to-end budgets. T2 modernizes pretraining scaling laws…

Long-horizon agents operate across many dependent states and transitions, often spanning multiple tools, environments, and periods of external feedback. The difficulty comes from preserving coherent progress as earlier decisions constrain later actions. A single workflow may involve researching evidence, changing files or records, waiting for external responses, revising plans, validating intermediate results, and returning to earlier systems with new information….

Most agent benchmarks still evaluate a thin slice of the job. The agent receives a task, produces an answer, gets scored, and the episode ends. Enterprise workflows work differently. An underwriting agent may need to read policy documents, inspect customer records, call internal tools, ask a simulated user for missing information, update state, and follow approval rules. A correct final…

Anthropic’s Claude Opus 5 recently debuted as the second model overall on the current Senior SWE-bench leaderboard, behind Fable 5. It also achieves the highest score of any evaluated model on the benchmark’s Bug & Performance Investigation category, reinforcing the rapid progress frontier coding models continue to make on increasingly realistic software engineering tasks. Just as notable, Opus 5 reaches…

At our latest Snorkel AI Reading Group, Henry Ehrenberg presented Senior SWE-Bench, an open-source, Harbor-compatible benchmark for evaluating coding agents on realistic, senior-level software engineering work. Its 100 tasks, with 50 public and 50 kept private to mitigate contamination, are sourced from real pull requests across 12 production repositories and cover complex features, migrations, bugs, and performance issues. Senior SWE-Bench…

We’ve evaluated Grok 4.5 on Snorkel’s GDPval+ dataset, Snorkel’s expert-created dataset of professional workplace reasoning tasks from across the economy. To compare performance against other frontier models, we ran the evaluation alongside GPT 5.5 and Claude Opus 4.8. Overall, Grok 4.5 demonstrated the strongest overall performance. Dataset GDPval+ is part of the Snorkel Data Series (SDS), Snorkel’s portfolio of expert-curated…

A top 10 US bank manages CLO portfolios totaling billions in assets, each governed by contracts up to 500 pages.