AI agent comparison · Hermes

Hermes record
5 of 9 passed

Version v0.22.2. Local model gpt-oss:20b, run on Mac Studio 2 (M5 Ultra, 96GB).

Results by task

TaskPassedScores (runs 1, 2, 3)
Research1/31.0, 0.778, 0.778
Fetch-failure sorting3/31.0, 1.0, 1.0
Island figures1/30.824, 0.0, 1.0

Scores show the share of the fixed grading items that were correct (1.0 is full marks). Only runs scoring 1.0 passed.

The research runs 2 and 3 scored 0.778, island run 1 scored 0.824, and island run 2 scored 0.0. This record does not identify why those scores were low.

What this shows

  • There were no abnormal exit codes, timeouts, or errors. Every failure was a grading result.
  • In research and island, only 1 of the 3 runs passed. With so few runs, we cannot tell whether passing is reliable.
  • The inputs are synthetic. We do not know whether real work documents would give the same results.