---
title: "LoRA Adapter for Qwen3.8-27B: Terminal-Bench 2.1 Rises From 39.5% to 55.8% With In-Harness RL"
url: https://aibenchmark.gr/en/news/lora-adapter-qwen3-8-27b-terminal-bench-2-1-rl-inside-harness/
language: en
published: 2026-11-20T22:21:43.026Z
updated: 2026-11-20T23:19:17.403Z
site: "AI Benchmark"
section: "Research"
---

# LoRA Adapter for Qwen3.8-27B: Terminal-Bench 2.1 Rises From 39.5% to 55.8% With In-Harness RL

Hugging Face trained adapters for coding agents with reinforcement learning. In pi, the Terminal-Bench 2.1 success rate rises from 39.5% to 55.8%, with roughly half the tokens per task.

> Written with AI from the sources listed below. Editorial responsibility: Konstantinos Perros. How we write: https://aibenchmark.gr/en/how-we-write/

## Key facts

- On Terminal-Bench 2.1 with the pi harness, Qwen3.8-27B reaches 39% pass@1 and, with a LoRA adapter trained with RL, 55%, according to Hugging Face's post. [1]
- Average consumption drops from 2.9K to 1.5K tokens per task, according to Hugging Face. [1]
- MarkTechPost reports that the Qwen3.8-27B-pi-adapted adapter scores 64.7% on SWE-bench Pro hard-51 (up from 39.2%) and 55.8% on Terminal-Bench 2.1 (up from 39.5%). [2]
- One adapter was trained per harness (pi, Hermes Agent, OpenCode), and the three were merged into a single “super adapter”. [1]
- Of 1,127 candidate tasks from the Hub, 858 passed the openenv validate check, while the 253 tasks from the target benchmarks were removed. [1]

## What was measured and in what setup

Hugging Face presented a method for adapting a model to its harness, meaning the environment that organizes tool calls and the agent loop. The company's key observation is that models aim for general performance and are not optimized for a specific harness or task distribution. As a result, the model-harness combination leaves room for gains in accuracy and cost.

For Qwen3.8-27B with the pi harness, the post reports 39% on Terminal-Bench 2.1 and an average usage of 2.9K tokens per task. With the weights frozen and a LoRA adapter trained with RL added, the result becomes 55% pass@1 (the share of tasks solved on the first attempt) with 1.5K tokens on average. The company attributes the gain mainly to the model reasoning less, calling fewer tools and getting them right more often on the first try.

MarkTechPost, which refers to the Qwen3.8-27B-pi-adapted adapter under the Apache-2.0 license, gives more precise figures: 39.5% to 55.8% on Terminal-Bench 2.1 and 39.2% to 64.7% on the SWE-bench Pro hard-51 subset. The post's figures of 39% and 55% differ in precision from MarkTechPost's, and the sources do not explain the difference.

*Base model and model with LoRA adapter, as reported by the sources (pass@1)*

| Metric | Base Qwen3.8-27B | With adapter | Source |
| --- | --- | --- | --- |
| Terminal-Bench 2.1, pi harness | 39% | 55% | Hugging Face |
| Terminal-Bench 2.1 | 39.5% | 55.8% | MarkTechPost |
| SWE-bench Pro hard-51 | 39.2% | 64.7% | MarkTechPost |
| Average tokens per task (Terminal-Bench 2.1, pi) | 2.9K | 1.5K | Hugging Face |

## How the training was built

Training took place inside the harnesses themselves, using Pi 1.5.2, OpenCode 2.2.7 and Hermes Agent v0.22.6. According to Hugging Face, the harness runs as distributed and the trainer learns from the exact tokens the model produced, recorded by a proxy between the harness and vLLM. This differs from RL frameworks that train on a reimplementation of the agent loop. The data, training and agent runs all took place on the Hub with Jobs, using TRL's AsyncGRPOTrainer, OpenEnv and harbor.

The task set came from three sources. SWE-smith injects bugs into real Python and Go repositories. SWE-Gym, packaged as Harbor tasks, converts real GitHub issues and their fixes. Tasksmith generates tasks from merged pull requests: from 30 pull requests came 24 tasks across 10 Python repositories that passed its checks, 17 of them also on a second run.

The initial search on the Hub found about 59,000 Harbor tasks, including duplicates, evaluation sets and tasks unsuitable for training. Filtering kept 1,127 candidates, and 858 passed the openenv validate check. Checks with known wrong outputs uncovered flaws, among them a Terminal-Bench 2.1 verifier (regex-log) that accepted wrong answers. Most failures were infrastructure problems, such as images that would not start. A difficulty check with the base model followed, because in GRPO a task that all attempts solve or all fail provides no learning signal.

## Limits and next steps

The figures come from Hugging Face itself and concern its own adapter, while MarkTechPost relays them. The sources contain no independent evaluation. The 253 tasks from the target benchmarks, and every task from a repository those benchmarks use, were removed from the training tasks to limit overlap with the measurement sets.

Hugging Face states that each adapter outperforms the base model in its own harness and on both benchmarks, in both performance and efficiency. The three LoRAs were merged into a single shared adapter that runs unchanged in pi, Hermes Agent and OpenCode. The company also publishes the artifacts (adapters, GGUFs, RL tasks, the safeagents plugin) in a collection and presents the work as a reproducible recipe for other combinations of model, harness and tasks. The post's content also includes the open question of what the model actually learned.

## Questions and answers

**How much did Qwen3.8-27B improve on Terminal-Bench 2.1?**

Hugging Face reports 39% without and 55% pass@1 with the adapter in the pi harness. MarkTechPost gives 39.5% and 55.8%.

**Did token consumption also fall?**

Yes, according to Hugging Face, average usage per task on Terminal-Bench 2.1 with pi fell from 2.9K to 1.5K tokens.

**Which harnesses are covered?**

Adapters were trained for pi, Hermes Agent and OpenCode, plus a merged adapter that runs on all three.

## Sources

1. Hugging Face: [Introducing harness adapters: adapt your favorite model to your harness to improve performance and cost](https://huggingface.co/blog/harness-adapters-at-scale)
2. MarkTechPost: [New LoRA Adapter Lifts Qwen3.8 Coding Success From 39% to 65%](https://www.marktechpost.com/2026/11/18/hugging-face-qwen3-8-27b-pi-adapted-lora)
