Continual Learning Bench

Continual Learning Bench evaluates whether an AI system improves from experience gathered during an evaluation. Instead of treating each problem as an independent attempt, the benchmark presents related instances in sequence. A system can retain information from earlier interactions and apply it to later decisions.

Continual Learning Bench 1.0 contains six expert-validated task sequences: Codebase adaptation asks agents to resolve successive issues in a shared repository. Blind spectrum monitoring requires them to identify radio emitters from incomplete sensor data. Cohort studies involve estimating survival outcomes across datasets with inconsistent schemas. Database exploration tests question answering over an unfamiliar SQLite database. Exploitable poker rewards learning an opponent’s strategy. Sales prediction requires agents to discover store-specific demand patterns from sequential datasets.

Each benchmark task contains multiple related instances presented in a fixed sequence. The instances share useful underlying structure, such as repository organization, database conventions, emitter behaviour, or an opponent’s betting strategy.

The system receives a reward after each instance. Information retained from earlier interactions can help it complete later instances more accurately or efficiently. The benchmark harness manages the sequence, preserves the permitted system state, and records the resulting trajectories and rewards.

A stateful system carries information from one instance to the next. That state may consist of conversation history, notes, retrieved memories, learned procedures, or another representation maintained by the agent architecture.

The stateless baseline uses the same model and system configuration but resets between instances. Comparing the two conditions shows whether retained experience improved performance, rather than simply measuring how capable the underlying model was before the sequence began.

Gain is the reward achieved by the continual, stateful system minus the reward achieved by its stateless baseline: Gain = stateful reward − stateless reward

Positive gain means the system performed better when it could use prior experience. Zero gain means retaining state provided no measurable benefit. Negative gain means the accumulated context or memory reduces performance.

Reward measures how well the system performs the task. Gain measures how much of that performance comes from learning across the sequence.

A strong model can earn high reward while showing little gain if it already solves individual instances without relying on earlier experience. A less capable model could earn lower total reward but higher gain if its performance improves substantially as the sequence progresses. Both metrics are needed to separate baseline capability from continual learning.

The six tasks use different reward scales, so their raw scores cannot be averaged directly. Continual Learning Bench normalizes each task’s reward against a reference ceiling and fixed stateless baseline before calculating aggregate reward.

Aggregate gain applies corresponding normalization to the difference between the stateful and stateless runs. These aggregate metrics allow performance across all six domains to contribute to the overall leaderboard without tasks with larger numerical reward ranges dominating the result.

Retained information is only useful when the system stores the right evidence and applies it in the right context. An agent may overfit to a recent observation, preserve an incorrect assumption, retrieve irrelevant memories, or continue using knowledge after the environment has changed.

Negative gain indicates that the stateful system performed worse than the same system running from a fresh reset. Continual Learning Bench therefore measures whether memory is beneficial, rather than assuming that a longer context or dedicated memory layer automatically improves an agent.

The benchmark can compare systems that retain full interaction histories, maintain external notes, retrieve selected memories, or use dedicated memory-management components. Its published evaluations include basic in-context learning, persistent notepads, standalone memory systems, and established coding-agent harnesses.

Each leaderboard row represents the full system configuration. Model choice, memory architecture, context management, prompts, tools, and agent logic can all affect reward and gain.

Average reward reflects overall task performance, while average gain reflects improvement attributable to retained experience. Cost records the mean model spending associated with the evaluated task rollouts.

A system with high reward is generally capable across the task suite. A system with high gain makes effective use of sequential experience. Practical comparisons should consider all three metrics because additional context, retrieval, or memory operations may improve learning while also increasing execution cost.

Continual Learning Bench is open source. The repository includes the task environments, evaluated systems, execution harness, schedules, result viewers, and commands for running individual tasks or the complete suite. Local execution requires Python 3.13 or later, and Docker is required for containerized tasks and systems.

Researchers can add custom systems to test new memory, retrieval, prompting, or tool-use strategies. They can also contribute new sequential tasks with learnable structure, repeatable environments, and reward functions that measure improvement across episodes.