Agents' Last Exam

Agents’ Last Exam, or ALE, evaluates whether AI agents can complete long-horizon professional work and produce verifiable deliverables. The benchmark covers computer-based work across 55 sub-industries, using tasks sourced and validated by industry experts. ALE-V1 contains 147 public reference tasks selected from a larger corpus of more than 1,500 tasks.

ALE includes work in finance, engineering, healthcare, life sciences, law, media production, architecture, manufacturing, and computing. Tasks include reconstructing financial statements, running mold-flow simulations, annotating clinical variants, diagnosing Kubernetes failures, producing video storyboards, and calculating legal fees under jurisdiction-specific rules.

Many tasks require professional software such as Adobe After Effects, Siemens NX, Unreal Engine, Moldex3D, Rhino 3D, and FSLeyes. The agent must use the relevant files, data, and tools to create the requested output.

Each task runs inside a sandboxed Windows or Linux environment containing the required software and input data. The agent receives a task description and works independently using its own action loop, tools, memory, and sub-agents. ALE supports workflows that combine terminal commands with graphical computer interaction.

After the run ends, ALE introduces the hidden reference, grades the files or other artifacts produced by the agent, and records the trajectory, logs, screenshots, tool calls, and evaluation result.

Agents’ Last Exam uses task-specific deterministic graders to compare the agent’s output with a hidden reference. Each task produces a score between zero and one based on the requirements satisfied by the completed artifact. ALE then aggregates those task results into Pass Rate and Score metrics for the leaderboard.

Pass Rate is the percentage of tasks the agent completes fully. A task contributes to Pass Rate only when it satisfies the complete success criteria.
Score averages the graded result across all tasks and includes partial credit. An agent may therefore earn a meaningful Score by completing substantial portions of several workflows even when those tasks do not count as full passes. The two metrics separate useful progress from dependable end-to-end completion.

The hidden reference is added to the environment only after the agent finishes, preventing the expected answer or artifact from leaking into the task. The grader can then inspect measurable properties such as extracted figures, simulation outputs, file structure, dimensional accuracy, identified root causes, or agreement with a reference dataset.

This approach keeps evaluation tied to the completed work. It also avoids using an LLM as the final judge, which could introduce grader variability or preferences unrelated to the task requirements.

ALE-V1 is the current public release, containing 147 reference tasks across all 55 targeted sub-industries. It represents a subset of the broader task corpus. Some workflows remain private because they rely on licensed software, restricted data, or held-out evaluation material.

Agents’ Last Exam uses rolling evaluation. Approximately every six months, fresh task instances enter the public set, private tasks rotate into public release, and retired tasks leave the active evaluation. Held-out tasks are used for official scoring, reducing the value of training directly against a fixed public test set.

ALE evaluates the complete agent configuration, including the foundation model, harness, tools, memory, context management, action loop, and graphical or terminal interfaces. The same model can perform differently when paired with another harness or reasoning configuration.

Leaderboard comparisons should therefore treat each row as a model-and-harness system. Pass Rate and Score should also be considered alongside estimated cost, runtime, and token consumption when comparing practical efficiency.

Humanity’s Last Exam evaluates models on difficult, closed-ended academic questions across subjects such as mathematics, science, and the humanities. Agents’ Last Exam evaluates whether an agent can execute professional workflows inside real software environments and produce completed work products.

A system can possess strong academic knowledge while still struggling to manage files, operate specialized tools, recover from execution errors, and satisfy all the requirements of a long professional assignment.

The ALE repository provides an open evaluation framework, public tasks, sandbox provisioning, grading tools, and integrations for multiple agent harnesses. Supported execution options include cloud virtual machines, local containers for a lighter Ubuntu subset, QEMU/KVM, and existing computer-use sandboxes. Researchers can also add a custom agent by implementing an ALE deployer.

Domain experts can propose workflows that are complex, representative of real industry practice, and objectively verifiable. Researchers and engineers can help convert those workflows into reproducible environments and graders.