Fable 5.2 vs Fable 5.1: Coding Benchmark Results

In a previous evaluation of Fable 5.1, the model was fast and token-efficient on the runs it completed, but it lost many attempts to terminal failures such as fragile large-file writes, malformed outputs, and failed tool calls, often with the work close to correct when the run died.

To measure how Fable 5.2 compares, it was run through the same expert-created frontier task set, with 30 tasks, five attempts per task, and 150 trials per model. To do a fair comparison, the two models were evaluated on a matched set of tasks, where both models had at least three clean rollouts. This ensured that every task in the comparison carried enough valid attempts from each model to support a side-by-side comparison.

That requirement was met for 26 of the 30 tasks. Across those 26 tasks, the comparison covered 106 trials from Fable 5.1 and 121 from Fable 5.2. The values reported within this analysis are computed from those matched trials.

Results on the matched set

Fable 5.2 leads on every measure in the analysis. It solved 71.4% of tasks on a single attempt, 18 points above Fable 5.1, and held a smaller but clear lead when each model was given three attempts per task. The task-level results showed the source of that gap. Both models solved 16 of the 26 tasks at least once, Fable 5.2 alone solved another 5, Fable 5.1 alone solved 2, and 3 went unsolved by both. The five tasks Fable 5.2 newly cleared were not near-misses for its predecessor. Fable 5.1 passed zero attempts on all five, and Fable 5.2 passed two of them on every attempt.

The pass rates also understate one of the clearest changes, which was reliability. Fable 5.1 passed every trial on only 6 of the 26 matched tasks. Even on tasks it usually solved, it tended to lose at least one attempt to a mid-run failure. Fable 5.2 passed every trial on 15 of the 26. A model that passes a task on some rollouts and loses others forces the user to run it several times and hope for a better outcome. When Fable 5.2 was able to solve a task, it usually did so on a single attempt.

Where Fable 5.2 performed well

Beyond the pass rates, every trial was also reviewed for the skills it demonstrated, with each skill rated on quality. Verification stood out as Fable 5.2’s most frequent and highest-rated skill, with 302 recorded instances and 71% of them rated as strong. This shows that the model checked its own work without being prompted. Across the run, it built adversarial test fixtures with malformed inputs and corrupted archives, hand-counted expected results before comparing them against its output, and reran finished steps to confirm the results were identical. This is a clear change from Fable 5.1, where the analysis showed the model claiming verification steps it had never actually run.

Fable 5.2 also recovered well from its own mistakes. The reviews recorded 90 recovery episodes across the run, and the model recovered successfully in 77 of them (86%). Most recoveries were quick. The median recovery took a single step, and 62 of the 77 were resolved within two steps. A failed command often started a run-ending cascade for Fable 5.1. For Fable 5.2, a failed command was usually corrected on the next step.

Execution quality held up even in failing trials. Across those trials, 96% of the 295 skill ratings were adequate or strong, and only 10 were rated weak. The failing runs rarely stumbled on execution. More often they carried out a flawed plan competently, which matches the failure profile seen throughout the evaluation of the model.

Two failure signatures

Fable 5.1’s failures were mechanical. Premature termination was the most frequent root cause. In the typical case, the model had written most of a correct solution, attempted a large file write through the terminal, corrupted the session, and could not recover the state it had built. The wrong-format and wrong-argument failures had the same character. The solution existed, and the step that delivered it failed.

Little of that profile was seen in the Fable 5.2 run. Across the set of trials there were no premature terminations, no format failures, and only a single wrong-arguments trace. The failures that remained were in planning and reasoning, led by wrong strategy (35% of its root-caused trials), faulty inference (28%), and misreading the task (23%).

These Fable 5.2 challenges were repeatable rather than random. The faulty-inference failures concentrated on two tasks, where every attempt independently reached the same wrong conclusion about the expected behavior. This suggested a stable misreading rather than sampling variance. These failures indicate that the model succeeds in executing its plan but that it may sometimes choose the wrong plan.

Where Fable 5.2 gave ground

The gains were not uniform. Two tasks that Fable 5.1 passed reliably now fail on every attempt, and a third fell from five of five to two of five. The traces show strategic causes rather than mechanical ones. In one task, every failing attempt commits to the same faulty inference about the expected behavior and then verifies its work against that incorrect reading, so its own checks pass while the verifier’s fail. In another, the model misreads what the task asks and plans against that reading from the first step. These regressions match the failure profile that reveal the model’s ceiling. A stable misreading fails five of five attempts the same way the previous terminal crashes did, and extra sampling does not recover it.

The takeaway

Fable 5.1’s weak spots were mechanical and last-mile. Work was substantially complete and then lost to the terminal, to an unparsed output, or to a check that was asserted rather than run. The Fable 5.2 run shows that gap largely closed. Terminal-interaction failures are near zero, timeouts preserve work, and 15 of the 26 matched tasks pass on every clean trial.

The remaining failures sit in strategy and task understanding, the same class that defined Opus 5 on this task set. That class is harder to fix with scaffolding, because the failing runs execute cleanly and verify against their own misreading. Whether the pattern holds across future generations remains untested. On this task set, the mechanical failures receded and the reasoning failures remained.