Same Model, Different Harness: CUGA, Hermes, and Deep Agents

Same Model, Different Harness: CUGA, Hermes, and Deep Agents — When to Pick CUGA?

EnterpriseArticlePublished
November 3, 2026
Upvote 12
Sami Marreedsamimarreedibm-research
Anupama Murthianupamamurthiibm-research
Vinod Muthusamyvinodmutibm-research
Ankita Naikankita-naikibm-research
Segev Shlomovsegevshlomovibm-research
Jayaram Radhakrishnanjayaramkribm-research
Aminat Aaadebiyiibm-research
Ayhan Sebinayhansebinibm-research
Hamid Adebayoharmedoxibm-research
Erez Bilgoryerezbiibm-research
Asaf AdiAsafAdiibm-research
Jim Laredolaredojibm-research
Merve Unuvarmrvnvribm-research

There are nearly 200 agent harnesses to choose from. We gave three of them the same open-weight model and 90 AppWorld tasks, and changing the harness moved task completion from 30% to 81%. For enterprise use, CUGA adds what this benchmark does not test: rules that the harness enforces and deployment inside your own boundary.

⭐ Star CUGA on GitHub🛡️ Try the Policy Playground →

Community lists such as awesome-agent-harness and best-of-Agent-Harnesses track nearly 200 of them, and they make one point clearly: the harness matters, not only the model. In one analysis that best-of-Agent-Harnesses cites, the same GLM-5.2 model went from 23% to 52% on SWE-bench Pro when only the harness changed.

So the right question is not “which harness is best?” but “which harness fits my workload, with my model?” Hermes from Nous Research is built for always-on personal assistants, and Deep Agents from LangChain for long, multi-step tasks such as research. Both are excellent at those jobs.

Enterprise work is a different job. Agents act on business systems, such as CRM and payments, and they must follow rules that hold whatever the model decides. CUGA (ConfigUrable Generalist Agent) is IBM’s open-s, inside your own environment, including sovereign clouds and air-gapped deployments. To measure what the harness adds, we kept the model the same and changed only the harness

A controlled experiment: same model, different harness

We ran CUGA, Hermes, and Deep Agents on AppWorld, a benchmark of multi-step tasks across 9 apps and 457 APIs, such as email, payments, and file storage. The apps are consumer apps, so AppWorld is a proxy: it tests a subset of the capabilities that enterprise automation also needs, such as logging in, reading API docs, paging through results, and changing state across several apps. Every harness used the same open-weight model, GLM-5.3-flash, on the same 90 tasks in 30 scenarios, with one attempt per task. All runs used the Harbor evaluation framework, and our setup is public in harbor-eval, our Harbor fork.

The results

Figure 1. AppWorld, 90 tasks in 30 scenarios, GLM-5.3-flash for every harness. TGC: tasks passed. SGC: scenarios where all three variants passed.

How to read this. This is not a verdict on Hermes or Deep Agents. Neither was built for multi-app API work like AppWorld, and that is the point: the harness has to fit the job.

The most useful number is SGC. A scenario passes only when all three of its variants pass, so SGC rewards consistency, not a lucky run. Here the gap is larger: 20 scenarios for CUGA, 8 for Hermes, and 2 for Deep Agents.

Three limits apply. This is a 90-task sample on one benchmark, so the results may not generalize to other tasks. The setups were not identical (see the setup table above). And the runs did not use policies, so they do not measure policy enforcement.

Where the points come from

In the 90-task comparison, the model was the same in every run, so the differences come from the harness. Every CUGA agent runs the same compact loop: prepare, call the model, execute, and repeat. The same loop scales from one CugaAgent with its own tools to a CugaSupervisor coordinating several agents, local or remote over A2A. These are the CUGA features that ran in the AppWorld setup, grouped by what they do:

🧭  Plan the work
Task decomposition: splits a task into steps, one app at a time
Focused tools: each step sees the tools of its app and a few shared ones, not all 469
Tool shortlister: find_tools searches for any other tool a step needs
🐍  Act in code
CodeAct: the model writes Python that calls the app APIs
Sandbox: the code runs in a restricted environment, with a time limit
🔍  Check before acting
Verification before execution: a check reviews each code block that changes data before it runs
🔁  Check after acting
Automatic reflection: reviews the results and lets the agent correct its plan
Pagination audit: flags a list call that returned a full page when the next page was never requested

With these features, the model makes fewer decisions, and the harness catches more mistakes before they reach the result.

Every step the harness owns is an enforcement point

Each step that the harness controls, like those in the list above, is also a place where it can check a rule.

Most teams write business rules, such as payment approval or keeping customer PII inside the organization, into the system prompt. A prompt is an instruction, not enforcement. The model can misread it, a user can talk the agent out of it, and an injected message can change it in the middle of a task.

CUGA policies move these rules out of the prompt and into the harness. We describe this policy layer in our ACM CAIS 2026 paper, Governance by Construction for Generalist Agents (arXiv). The policies act at four points in every run:

Guidance steers the model, but approvals and ToolGuards are enforced by the runtime. Put rules such as payment approval and PII handling in the second group, where they hold whatever the model decides.

You can see the difference yourself in the Policy Playground, which sends the same message to two live agents, one without policies and one with them:

Figure 2. For “Pay everything I owe.”, the agent without policies paid $90, including a denied claim in collections. The agent with policies held the payment for approval, then paid only the $10 that was due.

Why pick CUGA

Choose a harness as carefully as the model. In our comparison, the same model completed between 30% and 81% of AppWorld tasks, depending on the harness. For enterprise work, CUGA gives you:

• Strong results, now with open-weight models. With hosted models, CUGA held #1 on WebArena (February to September 2025) and on AppWorld (July 2025 to February 2026).
• Rules that the runtime enforces. Approvals and tool guards stop a call in the harness, whatever the model decides.
• Deployment inside your sovereign boundary. CUGA is Apache 2.0, works with any OpenAI-compatible endpoint, and runs locally, on Kubernetes in your private or sovereign cloud, or even in an air-gapped sovereign cluster, with OIDC sign-in and separate chat and management roles. IBM Sovereign Corebuilds on CUGA to run agents fully inside the customer’s boundary.
• Measurable cost. Budgets limit spending, and run receipts show tokens, tool calls, and time for every run.
• Configuration, not code. Set tools, budgets, and policies in the UI or SDK, test a draft, and publish a version.

When to pick something else

CUGA is not the right harness for every job:

• For a terminal coding agent, use opencode, Codex, or Claude Code.
• For a personal, always-on assistant that lives in your chat apps, use OpenClaw or Hermes.
• For a blank-canvas graph framework, use LangGraph directly. CUGA is built on it.

CUGA is for automating business work across APIs, apps, and web UIs, under rules you can enforce.

Try CUGA

• 🛡️ Policy Playground · policies guide
• 🧪 Reproduce the comparison
• ⭐ Star CUGA on GitHub
• 🚀 Try CUGA: quick install

The CUGA team ran this comparison, and CUGA used an AppWorld configuration that the other harnesses did not. If you can configure Hermes or Deep Agents to do better, send a configuration to harbor-eval. We will rerun it on the same tasks and publish the results, whichever harness wins.