All articles
    agents
    agent-harness
    context-runtime
    benchmarks
    context-management
    evidence

    Stress-Testing Agent Harnesses Where Models Cannot Hide Their Weaknesses

    October 31, 2026redevops.io
    Stress-Testing Agent Harnesses Where Models Cannot Hide Their Weaknesses

    We deliberately test agent infrastructure with smaller, less expensive models and tighter context budgets. The objective is not to achieve the highest possible benchmark score. It is to expose differences in execution orchestration, context retention and recovery that a more capable model with a larger context window might compensate for. That is our test-design hypothesis, not a finding from a controlled comparison across model sizes.

    The absolute scores are not product-readiness scores. A 0.33 JobBench group score is poor if interpreted as autonomous task reliability. Scores of 0.206 and 0.194 are worse still. But this experiment was deliberately constrained, and the three harnesses were not run under fully equivalent time limits, sandboxes and tool configurations. The results should not be read as the normal capabilities—or production suitability—of OpenCode, Harness² or ReDevOps.

    Our original goal was to compare ReDevOps Agent Harness + Context Runtime with Google Research's Harness² and stock OpenCode. The work also revealed a gap in our own harness: much of its earlier development had taken place alongside redevops-rag, where retrieval controls the information passed to the agent. We improved the path where an application instead sends a large prompt or document set directly, without retrieval, then tested the revised stack.

    Our question was narrower than “which agent is best?” We wanted to learn which failures the harness could prevent when the model itself had less room to recover, and whether Context Runtime still helped when large inputs arrived directly instead of through retrieval.

    The harness comparison

    We used 11 JobBench tasks, three replicates, a locally hosted Qwen3.8-27B FP8 model with a 65,536-token context window, and JobBench's group-level scoring with Kimi judging. A run without a deliverable scored zero. The choice was intentional: constrain model capability and available context to make orchestration and context-handling failures more visible. We did not run a stronger-model control, so we cannot quantify how much a larger model would improve or conceal these results.

    ConfigurationJobBench group scoreDeliverables
    ReDevOps Agent Harness R3 + Context Runtime0.36329/33
    Stock OpenCode0.20624/33
    Harness²0.19418/33

    These are low absolute scores from a constrained diagnostic setup, not estimates of how these projects perform with stronger models or more generous resources. None justifies unattended deployment for high-stakes tasks. Their value here is in investigating failure modes under pressure—not in dismissing the external projects.

    In this setup, our harness scored higher and delivered more often. This is not a controlled proof that one harness architecture is superior. The OpenCode-based runs had a 2,400-second rollout limit versus 3,600 seconds for our harness; endpoint contention, sandboxes, tools and orchestration also differed. In fact, 151 of 254 OpenCode-based rollout attempts hit their time limit. These results describe the tested configurations, not an intrinsic ranking of the projects.

    We also tried connecting Harness² and OpenCode to Context Runtime through a gateway. A subsequent audit uncovered two gateway defects: archived-content placeholders could leak into tool calls, and separate rollouts could share a context session. We fixed both and validated the changes offline, but did not rerun the model experiments. Consequently, those gateway scores cannot establish whether Context Runtime helps or hurts either external harness.

    Does Context Runtime explain the result?

    To isolate context handling under the same constrained-model conditions, we kept the same ReDevOps harness, model, tools and execution settings and changed only its context manager. Across 11 tasks and three replicates (99 judged outputs):

    Context strategyJobBench group scoreDeliverables
    Full Context Runtime (archive + compaction + retrieval)0.33528/33
    No proactive context management; emergency clipping only0.33127/33
    Retrieval-only, without proactive compaction0.21523/33

    The difference between full Context Runtime and no proactive management was +0.004, with a 95% bootstrap interval of −0.105 to +0.116. That is no demonstrated overall improvement on this task set. Full Context Runtime outscored the retrieval-only variant by +0.120 (95% interval +0.020 to +0.234), but the cause of that gap is not established.

    One revealing detail: the agent invoked archive search only 12 times in 33 full-runtime runs, and never in the retrieval-only runs. This was primarily a large direct-context test, not a demonstration of RAG retrieval quality. redevops-rag was not used in any arm.

    The important failure mode: compaction isn't always helpful

    On three tasks—bookkeeping, producers and social-science research—the unmanaged-context arm scored higher than full Context Runtime by approximately 0.23, 0.19 and 0.18, respectively, averaged over replicates. The direction persisted in the two later replicates for bookkeeping and social science; the producer difference became small.

    All three tasks produced deliverables in both arms, so this was a quality difference, not a missing-file artifact. One plausible explanation is that early compaction removes details needed for document-heavy work. But we have not isolated compaction as the cause: the task selection was post-hoc, the sample was small, and rerun variability was substantial.

    The practical lesson is to avoid assuming that a shorter context is a better context. For direct-document workflows, retaining more of the original material may outperform aggressive compression. Context strategy should depend on the task and available window, not a universal threshold.

    What changes next

    This investigation reinforced three engineering priorities for ReDevOps:

    1. Treat direct context as a first-class input path, not an edge case behind retrieval.
    2. Make compaction conservative and measurable, with explicit safeguards against unusable archive placeholders and cross-execution state leakage.
    3. Evaluate context policies independently of the harness, so an improvement in orchestration is not mistakenly attributed to retrieval or compression.

    We have not yet tested a later-compacting or task-adaptive policy. That remains an experiment to run, not a result to advertise.

    Bottom line: A harness should earn its value when model capability and context are limited—not only when a powerful model can compensate for its mistakes. In our deliberately constrained test, ReDevOps Agent Harness scored higher than the tested alternatives, but the low absolute scores are diagnostic, not production-quality results or a general harness ranking. The ablation suggests the harness's advantage over the tested OpenCode-based configurations cannot be credited to Context Runtime's retrieval or compaction. The more useful discovery is where our context management needs to become less intrusive—especially when an agent receives a large body of information directly.

    Methodology note: This reported experiment used one smaller local model, not a measured sweep across model sizes. The claim that stronger models can mask harness weaknesses is a test-design hypothesis, not an experimental finding here. One model, 11 tasks, three replicates, one judge, and non-randomized run order. Reported confidence intervals use a task-cluster bootstrap. Results are exploratory and should not be generalized to other models, workloads or context windows.

    Want ReDevOps in your Google Top Stories & AI results?Follow ReDevOps on Google