What was measured and in what setup
Hugging Face presented a method for adapting a model to its harness, meaning the environment that organizes tool calls and the agent loop. The company's key observation is that models aim for general performance and are not optimized for a specific harness or task distribution. As a result, the model-harness combination leaves room for gains in accuracy and cost.
For Qwen3.8-27B with the pi harness, the post reports 39% on Terminal-Bench 2.1 and an average usage of 2.9K tokens per task. With the weights frozen and a LoRA adapter trained with RL added, the result becomes 55% pass@1 (the share of tasks solved on the first attempt) with 1.5K tokens on average. The company attributes the gain mainly to the model reasoning less, calling fewer tools and getting them right more often on the first try.
MarkTechPost, which refers to the Qwen3.8-27B-pi-adapted adapter under the Apache-2.0 license, gives more precise figures: 39.5% to 55.8% on Terminal-Bench 2.1 and 39.2% to 64.7% on the SWE-bench Pro hard-51 subset. The post's figures of 39% and 55% differ in precision from MarkTechPost's, and the sources do not explain the difference.
Base model and model with LoRA adapter, as reported by the sources (pass@1)
| Metric | Base Qwen3.8-27B | With adapter | Source |
|---|---|---|---|
| Terminal-Bench 2.1, pi harness | 39% | 55% | Hugging Face |
| Terminal-Bench 2.1 | 39.5% | 55.8% | MarkTechPost |
| SWE-bench Pro hard-51 | 39.2% | 64.7% | MarkTechPost |
| Average tokens per task (Terminal-Bench 2.1, pi) | 2.9K | 1.5K | Hugging Face |
How the training was built
Training took place inside the harnesses themselves, using Pi 1.5.2, OpenCode 2.2.7 and Hermes Agent v0.22.6. According to Hugging Face, the harness runs as distributed and the trainer learns from the exact tokens the model produced, recorded by a proxy between the harness and vLLM. This differs from RL frameworks that train on a reimplementation of the agent loop. The data, training and agent runs all took place on the Hub with Jobs, using TRL's AsyncGRPOTrainer, OpenEnv and harbor.
The task set came from three sources. SWE-smith injects bugs into real Python and Go repositories. SWE-Gym, packaged as Harbor tasks, converts real GitHub issues and their fixes. Tasksmith generates tasks from merged pull requests: from 30 pull requests came 24 tasks across 10 Python repositories that passed its checks, 17 of them also on a second run.
The initial search on the Hub found about 59,000 Harbor tasks, including duplicates, evaluation sets and tasks unsuitable for training. Filtering kept 1,127 candidates, and 858 passed the openenv validate check. Checks with known wrong outputs uncovered flaws, among them a Terminal-Bench 2.1 verifier (regex-log) that accepted wrong answers. Most failures were infrastructure problems, such as images that would not start. A difficulty check with the base model followed, because in GRPO a task that all attempts solve or all fail provides no learning signal.
Limits and next steps
The figures come from Hugging Face itself and concern its own adapter, while MarkTechPost relays them. The sources contain no independent evaluation. The 253 tasks from the target benchmarks, and every task from a repository those benchmarks use, were removed from the training tasks to limit overlap with the measurement sets.
Hugging Face states that each adapter outperforms the base model in its own harness and on both benchmarks, in both performance and efficiency. The three LoRAs were merged into a single shared adapter that runs unchanged in pi, Hermes Agent and OpenCode. The company also publishes the artifacts (adapters, GGUFs, RL tasks, the safeagents plugin) in a collection and presents the work as a reproducible recipe for other combinations of model, harness and tasks. The post's content also includes the open question of what the model actually learned.