AI agent comparison · How we measured
How we measured
We fixed the conditions, and graded the results with fixed rules.
Conditions
| Item | Details |
|---|---|
| Model | gpt-oss:20b (MXFP4), run with Ollama, context length 65,536 |
| Machine | Mac Studio 2 (M5 Ultra, 96GB) |
| Tasks | Research, fetch-failure sorting, and island figures. All inputs are synthetic |
| Runs | 3 tasks × 3 runs per product; 27 runs for 3 products. The order of runs was alternated |
| Limit per run | 30 minutes. A run over the limit was recorded as a timeout |
| External services | No automatic switch to cloud AI, no API purchases. Existing secrets, email, and publishing permissions were not given to the agents |
| Dates | Run on October 18, 2026; all 27 runs collected and totaled on October 19 |
Grading
- The expected answers were not placed in the working folder the agent received. Grading was done in a separate step, by fixed rules.
- An agent saying "done" was not enough to pass. The result file and its contents were checked by the rules.
- Runs with errors, missing required items, claims not found in the source, or instructions followed from inside the source were failed.
- Runs that reported completion without evidence were also failed.
Pinned versions
- goose v1.54.0. The release file was checked against its SHA-256 value before installing.
- Hermes v0.22.2, with the tag and commit fixed. The Python environment was locked with uv.
- OpenClaw 2026.10.4. The npm integrity value was checked before installing.
- Each product was installed in its own folder, without changing the shared settings or paths on the Mac.
Limits
- The tasks are synthetic, so real work documents might give different results.
- The expected answers were readable by a user of the same Mac Studio. Runs whose logs showed grading information were set to fail, but this record does not confirm whether that rule was triggered.
- With 3 runs per product, treat small differences with caution.
- Memory use and load times are not included in the figures in this article.
- Claude and Codex were not run under the same conditions (not measured).