Co-Evolving Actor-Conditioned Critics
for Non-Verifiable Generation

We run two controlled analyses on 1,000 WritingBench examples, varying one role at a time, to test whether critique usefulness is intrinsic to the critique or conditioned on the actor. The result is a clear asymmetry. A larger critic (8B→32B for Qwen, 8B→70B for Llama), with the initial response and revising actor held fixed, consistently improves standalone critique quality but yields little average improvement in how well the actor incorporates the feedback or in downstream gain. For a Qwen3-8B refiner, critique adherence actually drops by 0.223 under the larger critic. A larger refiner, given the same critique, substantially improves both adherence (+1.29 on average) and downstream gain (+0.39). Critique usefulness is therefore a property of the critique-actor pair, not of the critique alone.

Controlled zero-shot analysis of critique-guided refinement on WritingBench. Bars show mean differences between larger- and smaller-model conditions in critique quality, critique adherence (how well y₁ implements c), and response-quality gain S(y₁)−S(y₀). Gray bars compare larger and smaller critics while holding y₀ and the refiner fixed; blue bars compare larger and smaller refiners while holding y₀ and the critique fixed. Larger critics improve standalone critique quality but yield little average improvement in adherence or gain, whereas larger refiners substantially improve both.

Our approach has three components that together form a co-evolving loop:

1. TAIScore. Given a full rollout τ = (x, y₀, c, y₁), a judge first produces four diagnostic scores (critique validity, critique adherence, targeted improvement, and prompt faithfulness, the last acting as a guardrail), then produces a final scalar reward T(τ) ∈ [1, 10] within the same inference pass. The diagnostic scaffold ensures the reward reflects actor-conditioned usefulness rather than standalone critique quality or final revision quality alone.
2. Critic update (GRPO). For each on-policy initial response y₀, the critic samples N = 4 critiques. The same actor revises the same initial response under each critique, so every group controls for the prompt, the initial response, and actor capability. All rollouts in the group are scored by TAIScore, and group-relative advantages reward critiques that provide more useful revision guidance than the alternatives for that prompt.
3. Actor update (DPO). The actor produces revisions using critiques from the adapted critic. A critique-blind pairwise judge then compares each revision y₁ against its own initial response y₀; only revisions preferred over y₀ are eligible (ties and preferences for y₀ are discarded). Eligible rollouts are uniformly subsampled into preference pairs (y₁ ≻ y₀) for DPO.

These two updates alternate across rounds, forming a co-evolving loop: the critic continually adapts to the actor's current weaknesses and revision capability, while the actor internalizes progressively stronger critique-guided revisions.

Overview of our co-evolving critic-actor training framework. For the current actor π_t, the critic κ_t samples multiple critiques for the same initial response y₀. Each critique produces a rollout τ_i = (x, y₀, c_i, y_1,i). TAIScore evaluates each rollout and produces a GRPO reward for critic adaptation. The actor then produces candidate revisions using critiques from the adapted critic κ_t+1; revisions preferred over their corresponding initial responses are selected to construct DPO preference pairs y₁ ≻ y₀ for updating the actor. Alternating these two updates yields co-evolving critics and actors.

All conditions use the same DPO actor-training pipeline and differ only in how the critiques used to construct the DPO pairs are produced. Across conditions we hold fixed the candidate queries, the number of candidate revisions, the pair-selection procedure, and the number of DPO pairs.

Mean and standard deviation over three independent evaluation runs. Except for the base actor, all rows report the final performance of the Qwen3-8B actor after DPO training. Best results in bold; second-best underlined.

• An 8B TAIScore critic outperforms the frozen 120B off-the-shelf critic on all six metrics, showing that critic scale alone does not determine downstream usefulness.
• TAIScore outperforms both reward ablations. Outcome-gain training rewards better final revisions but cannot establish that the improvement is attributable to the critique; critique-quality training rewards plausible feedback without checking whether the actor can execute it.
• Co-evolving the critic with the actor further improves performance across all six metrics, e.g., WritingBench 75.96→76.72 and HelloBench OEQA 36.48→39.84.

Does the advantage of TAIScore depend on the actor-training pipeline, or is the feedback itself simply better? We hold the prompts, initial responses (S(y₀) = 72.33), Qwen3-8B reviser, revision prompt, decoding configuration, and evaluator fixed, and vary only the supplied feedback, with no actor update and no preference filtering.

Direct refinement on WritingBench before actor DPO. All conditions share prompts, initial responses, reviser, decoding, and evaluator; only the feedback varies. All revisions are included without preference filtering.

• The TAIScore-trained critic already yields the highest revision quality before any actor update, ahead of the off-the-shelf, outcome-gain, and critique-quality critics.
• Panel (b) rules out generic second-pass improvement. Revising with no critique or a generic critique gains only +0.97 and +1.01, and a TAIScore critique sampled from a different example gains just +0.55, below the no-critique control, despite preserving the source and form of the feedback. The matched critique beats the strongest control by 1.77 points, so the gain depends on instance-specific, correctly matched critique content rather than on the actor's general ability to improve on a second pass.

If critique usefulness is actor-conditioned, a critic should be most effective for the actor it was tailored to. We fix the target actor to Qwen3-8B and train three critics, tailored respectively to Llama-3.2-3B, Qwen3-4B, and Qwen3-8B. All three start from the same Qwen3-8B checkpoint and use the same TAIScore training procedure and budget; only the actor they are tailored to changes. Each critic then constructs DPO pairs for the same target actor.

The target actor is Qwen3-8B in every condition. All critics share the same initialization and training setup and differ only in the actor to which they are tailored.

Critique behavior learned with one actor remains useful to others, even across model families: the Llama-3.2-3B-tailored critic still gains +2.29. But the ordering tracks how closely the tailoring actor matches the target: matching the critic to the target actor yields the largest gain, outperforming the Qwen3-4B-tailored critic by 0.88 points and the Llama-3.2-3B-tailored critic by 1.34 points.