Fable 5.1 on Frontier Coding Tasks: Efficient Successes, Distinct Failure Modes

We evaluated Fable 5.1 on a series of frontier coding tasks from our proprietary Terminal-Bench+ dataset and compared the results against Opus 5. Fable remained competitive across most categories and was materially more efficient on successful runs, while its gap was concentrated in a small set of terminal-heavy and build/dependency tasks. Because category sizes are small and uneven, we treat the category results as descriptive of this task set.

Fable 5.1 gives you speed with robustness. On the same coding tasks it is dramatically cheaper per successful run than Opus 5 and competitive across most categories, but it loses more attempts outright. On a curated set of tasks undertaken by both models, Fable 5.1 and Opus 5 both solved 18, Opus alone solved 5, Fable alone solved 2, and 2 defeated both. Across successful runs on that same task set, Fable 5.1 solved the tasks using 58% fewer output tokens and finishes 36% faster. The result is a model with a clear shape rather than a clear rank; it is more token-efficient and faster on the work it completes but may be less reliable in getting there.

Where it leads: 87% on Debugging, 88% on Games

Overall, Fable 5.1 posts 61.5% on pass@1, 74.1% on pass@5, with 62.7% as the mean reward. While these tell us the overall performance of the model, they don’t really give insight into where the model performs well.

In looking at the actual tasks where Fable 5.1 excels, the results indicate that the model performs best when the task environment can contradict it. Its best categories are the ones where tests, simulators, or other task assets return concrete feedback capable of contradicting the model’s assumptions.

The strongest categories were debugging at 87% across four tasks and games 88% across four tasks, both of which provide mechanisms that can tell the model when it is wrong. A failing test is a hypothesis the model can run against, while a simulator returns a score, both of which supply a signal back to the model. Furthering this takeaway is that Software engineering, the largest category, lands at 60%.

Ten tasks were clean sweeps, passing every valid run at 100% of tests. Our tasks span a wide spectrum, from REST API debugging to languages such as OCaml, JavaScript, and time-series-based processing.

The weak spot sits at the opposite pole. Build-and-dependency management (18%) was the low point, where correctness is more a matter of convention than something where the model can check the environment to get a signal to guide it.

Fable 5.1 performance in context

that can be matched acrossOn the tasks common to both evaluations, the models split their performance. Opus leads by 6.2 points on pass@1 and solves three more tasks at least once than Fable.

By category, the two were close in several areas. Fable leads on Games and Debugging and ties on Software Engineering. Opus’s advantage is concentrated in Build and Dependency Management, where Fable posts 18% against Opus’s 67%, and secondarily in Data Processing.

The two models also lose runs differently, and the contrast is quite interesting:

• Fable’s main failure occurs when valid attempts are lost mid-trajectory rather than finishing incorrectly. Its scored timeout failures overwhelmingly follow fragile large-file terminal writes and ineffective recovery, but the work is usually close to right when the terminal state is lost.
• Opus’s main failure is budget exhaustion with the work being preserved. It hits the timeout more often but it usually can survive with most of the timed-out runs still earning full reward. That resilience is a large part of its higher success rate. It is also a large part of why it costs more.

The takeaway is that Fable is not a strict upgrade in this evaluation. It is faster and more token-efficient, with clear wins on specific tasks, but Opus remains more robust across the full matched set.

Efficiency is the clearest advantage

On the matched comparison, the Fable 5.1 median successful run uses 58% fewer output tokens and had a 36% lower wall-clock duration than Opus 5. When Fable solves a task, it seems to solve it with less effort and time.

These are descriptive measures though and are not a controlled benchmark, but the direction is consistent enough to be a meaningful finding in the comparison.

Investigating the failures

The completed failures show a consistent last-mile pattern. It passed a median 94% of verifier tests, meaning most of the measured requirements were satisfied even though one or more still failed. Taken on its surface it appears that the work is nearly done, but fails because something goes wrong, even if it is only a narrow failure.

In this, four main patterns appear:

1. It asserts verification it did not actually perform. In one dependency-resolution task, the model’s final turn claims it re-verified the output against the spec and names five specific areas, but nothing was actually re-verified. And, this was seen across every attempt on this task.
2. Effort ends up going to the cheaper, and sometimes easier, half of the problem. Rather than checking the semantically harder things, the model only does a spot check here, and on more surface properties that are easy to check, the model checks them thoroughly and correctly.
3. Narrow requirements in the tasks are missed despite actual sustained effort. In several cases the model completed most of the work correctly, but still missed a narrow behavioral detail even after having spent real time iterating on it rather than just stopping early.
4. The lowest-performing areas in this sample showed a similar pattern. Build and dependency management reached 18%, while System administration contained only one scorable task. Both stress correctness-by-convention and details where nothing in the environment would signal a problem until final verification. The model doesn’t see this and fails on these tasks.

Taken together, the traces show many failures of execution and last-mile verification, rather than wholesale failures to make progress. The model reaches a substantially correct solution and then loses it at the last step, whether to a lost terminal, to a check that was asserted rather than run, or to a convention that was never surfaced.

The takeaway

In this evaluation, Fable 5.1 was strongest where the environment could push back, efficient when it succeeded, and vulnerable where correctness depended on convention or where the terminal itself became the obstacle.

That combination is actionable. Two of the three weak spots, asserted-but-unperformed verification and destructive terminal loss, are last-mile failures where the work was substantially complete. They are not a knowledge ceiling and sit closer to reaching a pass rather than what just the raw pass rate suggests.

Interested in learning more? Connect with the Snorkel AI team.