AI agent comparison
goose, OpenClaw, and Hermes
on the same 3 tasks
Each agent ran the same tasks 3 times each, using the local model gpt-oss:20b.
Results (passes)
Pass or fail was decided by fixed grading rules. This is 3 tasks × 3 runs per product, 27 runs in total (run on October 18, 2026; collected and totaled on October 19).
| Product | Research | Fetch-failure sorting | Island figures | Total |
|---|---|---|---|---|
| goose | 0/3 | 3/3 | 0/3 | 3/9 |
| OpenClaw | 0/3 | 2/3 | 3/3 | 5/9 |
| Hermes | 1/3 | 3/3 | 1/3 | 5/9 |
No runs stopped from an abnormal exit code, a timeout, or an error. Every failure was a grading result.
Each run took 5.6–17.9 seconds for goose, 11.5–26.6 seconds for Hermes, and 15.4–46.7 seconds for OpenClaw. The tasks differ in size, so these times do not compare the speed of the products.
Records by product
- goose record: why goose scored zero on the research and island tasks.
- Hermes record: the scores per run, and the runs that fell short.
- How we measured: conditions, tasks, grading rules, pinned versions, and limits.
What the results do and do not show
- All three products had both passes and failures on the same tasks. The totals range from 3/9 to 5/9.
- With 3 runs per task, 27 runs in total, the numbers are too few to rank the products with confidence.
- goose scored zero on every research and island run. In research it reported that the read tool was not available; in island it read the input and then ended without writing the result file.
- The inputs are all synthetic. We do not know whether the same results would appear with real work documents.
- No same-condition runs of Claude or Codex were recorded (not measured). So we cannot say whether these agents reduce rework compared with existing methods.
- The planned trial on real work at our office (stage 3) did not run, because no suitable work arrived during the trial window (not measured). This is a fact about the trial, not a judgment of the products.