prime-rl gets an Algorithms layer

Today we're introducing a first-class algorithms layer in prime-rl. Six algorithms ship built-in — GRPO, MaxRL, On-Policy Distillation, Self-Distillation, SFT distillation and ECHO — each powered by the same abstraction, and each selectable per environment, so a single run can train GRPO on one env and ECHO on another. Bringing your own algorithm means subclassing that same abstraction, without having to touch the trainer. Our world-modeling post ended by promising flexible, performant ECHO support in prime-rl; this is that support, and the layer it lives in.

Before: an RL codebase with other algorithms bolted on

prime-rl was primarily designed to be a highly performant framework for large-scale async RL. Other algorithms were added over time — most notably OPD and SFT distillation — but they were not accounted for in the original design. Each new algorithm arrived as a thread through the entire system rather than as a module. Ask where "the algorithm" lived and you'd get a different answer depending on which algorithm you meant: a top-level training_mode switch (rl / opd / sft) branched the orchestrator; a compute_teacher_logprobs step in the orchestrator's batch loop produced teacher logprobs; a shared [orchestrator.teacher] slot declared a reference model that only some modes read; loss selection lived in the trainer as a per-batch switch on training_mode — one loss function per batch, so samples from different modes could never share one.

The algorithms layer

The goal of introducing an algorithms layer was to centralize everything algorithm-specific in one place and give it a real abstraction — a space researchers can hack on without giving up on performance or having to touch trainer internals.

The centralization is quite literal: every algorithm is one module under orchestrator/algo/, and reading a module top to bottom reads the algorithm. The abstraction underneath is thin. An algorithm fixes three things. Which model generates its rollouts — the live policy or a frozen endpoint — which is the distribution the loss is taken over, and the reason on-policy distillation gets to call itself on-policy. How credit is assigned: every token receives an advantage; a scalar broadcasts uniformly over the completion. And which of three fixed losses consumes that credit: policy gradient, cross-entropy, or reverse KL. Where a loss needs more than a weight, the algorithm ships that too — OPD attaches the teacher logprobs its reverse KL is computed against. That's the whole mathematical definition. In practice an algorithm is more of a runtime than a function — OPD manages its teacher client, SFT owns the frozen model that samples for it — so the class also carries a setup() lifecycle.

The narrowness is deliberate. The trainer knows exactly three loss components — rl, ce, ref_kl — and everything an algorithm decides reaches it as per-token weight streams: data, not code. Samples from different algorithms pack into the same micro-batch, because to the trainer they are indistinguishable. Since each component is normalized by its own global token count, an ECHO env that trains on far more tokens never dilutes the gradients of a GRPO env packed beside it.

In code, an algorithm is a named class with two scoring hooks — score_rollout, called per rollout as it arrives, and score_group, called once an example's group completes — plus a setup() for connecting endpoints:

class Algorithm:
    action_loss_type: ClassVar[ActionLossType] = "rl"    # rl | ce | ref_kl

    async def setup(self) -> None: ...  # connect frozen endpoints

    async def score_rollout(self, rollout: Rollout) -> None: ...    # one rollout, on arrival
    async def score_group(self, group: list[Rollout]) -> None: ...  # the entire group

The right algorithm depends on the environment

Algorithms resolve per environment, not per run, because the right training signal is a property of the environment. ECHO is the clearest case. In a terminal environment, tool outputs are the environment dynamics: the observation is the consequence of the agent's own action, and training the policy to predict it teaches it how the environment works. In a search environment, the observation is retrieved web text — spending cross-entropy learning to predict other people's documents would not improve the model at search, but would lead to memorization. ECHO makes sense for terminal, much less for search. Forcing one algorithm across a mixed-env run means picking the wrong signal for some envs.

Per-env resolution also gives you things we didn't have to explicitly build. Multi-teacher OPD falls out for free: each env declares its own algorithm, so each env can distill from its own teacher — a code-specialized teacher on your coding env, a math-specialized one on your math env, in a single run against a single student. To our knowledge, no other open-source RL framework lets you mix training algorithms per environment like this.

Concretely, GRPO for Math and Search, ECHO for Terminal tasks:

[orchestrator.algo]
type = "grpo"

[[orchestrator.train.env]]
id = "math-env"    # inherits grpo

[[orchestrator.train.env]]
id = "search-env"  # inherits grpo

[[orchestrator.train.env]]
id = "terminal-env"
algo = { type = "echo" }

Or multi-teacher OPD, with one teacher per domain:

[[orchestrator.train.env]]
id = "code-env"

[orchestrator.train.env.algo]
type = "opd"

[orchestrator.train.env.algo.teacher]
name = "CodingExpertModel"
base_url = ["http://localhost:8001/v1"]

[[orchestrator.train.env]]
id = "math-env"

[orchestrator.train.env.algo]
type = "opd"

[orchestrator.train.env.algo.teacher]
name = "MathExpertModel"
base_url = ["http://localhost:8002/v1"]

prime-rl only ever sets up and serves the trainable policy; any teacher or reference model is an external OpenAI-compatible endpoint you point it at.

A word on loss functions

The algorithm layer is mostly concerned with credit assignment, going from rollouts with rewards to rollouts with advantages. While we do allow registering a loss function for the RL loss (e.g. using CISPO or IcePop instead of our handpicked DPPO-style loss), we see this less as an algorithms question and more of a systems question, as the loss function is mostly about correcting for off-policyness through lag and trainer-inference mismatch. So while we do support custom losses, it is not part of the algorithm layer itself.

Only the beginning

The algorithms layer currently only answers one question — what per-token signal does a finalized rollout produce? — and it's the first of several axes along which we want to make prime-rl and verifiers more expressive. Others include:

• Sampling behavior. The Sampler sits between the orchestrator and each environment, and today answers only which model generates rollouts. Soon, it will be in charge of deciding which examples get sampled at all: online difficulty filtering, difficulty pools (examples that come back too easy or too hard get re-binned and resampled on their own schedule), curricula and replay buffers are all example strategies for future samplers.
• Agentic judging. Currently, judges in verifiers are single-turn and not embodied in a harness. A judge that is itself an agent — with tools, multiple turns, and access to the full trace — can verify claims instead of pattern-matching them.
• Agentic credit assignment. Similar to agentic judging, but on the advantage level: a model that reads the trace and can weigh different parts of the same rollout differently.
• Beyond the group. Today the group is a privileged concept: score_group fires when an example's group_size rollouts complete, and that is the only barrier the pipeline knows. But a group is just one dependency pattern. The general object is a dependent set: the rollouts an algorithm declares it must wait for before it can assign credit. We want the barrier itself to be algorithm-declared — the algorithm states its dependencies, the pipeline just waits for them — so score_group becomes finalize over whatever structure the signal needs.
• Multi-agent environments. Environments in verifiers are currently single-agent by design. This will soon change, as we want to introduce ways of expressing interesting multi-agent setups, such as proposer-solver or self-play environments.

The algorithms layer is on main today:

git clone https://github.com/PrimeIntellect-ai/prime-rl

Run one of the six built-in algorithms, or bring your own — and if you'd rather skip dealing with the infrastructure, start post-training your own models on Prime Intellect Lab.

Citation

Please cite this work as:

Konstantin Dunas and Prime Intellect Team, "prime-rl gets an Algorithms layer," Prime Intellect Blog, July 2026.

Or use the BibTeX citation:

@article{primeintellect2026algorithmslayer,
author = {Konstantin Dunas and Prime Intellect Team},
title = {prime-rl gets an Algorithms layer},
journal = {Prime Intellect Blog},
year = {2026},
month = {July},
note = {https://www.primeintellect.ai/blog/algorithms-layer}
}