AI agent comparison · How we measured

How we measured

We fixed the conditions, and graded the results with fixed rules.

Conditions

ItemDetails
Modelgpt-oss:20b (MXFP4), run with Ollama, context length 65,536
MachineMac Studio 2 (M5 Ultra, 96GB)
TasksResearch, fetch-failure sorting, and island figures. All inputs are synthetic
Runs3 tasks × 3 runs per product; 27 runs for 3 products. The order of runs was alternated
Limit per run30 minutes. A run over the limit was recorded as a timeout
External servicesNo automatic switch to cloud AI, no API purchases. Existing secrets, email, and publishing permissions were not given to the agents
DatesRun on October 18, 2026; all 27 runs collected and totaled on October 19

Grading

  • The expected answers were not placed in the working folder the agent received. Grading was done in a separate step, by fixed rules.
  • An agent saying "done" was not enough to pass. The result file and its contents were checked by the rules.
  • Runs with errors, missing required items, claims not found in the source, or instructions followed from inside the source were failed.
  • Runs that reported completion without evidence were also failed.

Pinned versions

  • goose v1.54.0. The release file was checked against its SHA-256 value before installing.
  • Hermes v0.22.2, with the tag and commit fixed. The Python environment was locked with uv.
  • OpenClaw 2026.10.4. The npm integrity value was checked before installing.
  • Each product was installed in its own folder, without changing the shared settings or paths on the Mac.

Limits

  • The tasks are synthetic, so real work documents might give different results.
  • The expected answers were readable by a user of the same Mac Studio. Runs whose logs showed grading information were set to fail, but this record does not confirm whether that rule was triggered.
  • With 3 runs per product, treat small differences with caution.
  • Memory use and load times are not included in the figures in this article.
  • Claude and Codex were not run under the same conditions (not measured).