AI agent comparison · Hermes
Hermes record
5 of 9 passed
Version v0.22.2. Local model gpt-oss:20b, run on Mac Studio 2 (M5 Ultra, 96GB).
Results by task
| Task | Passed | Scores (runs 1, 2, 3) |
|---|---|---|
| Research | 1/3 | 1.0, 0.778, 0.778 |
| Fetch-failure sorting | 3/3 | 1.0, 1.0, 1.0 |
| Island figures | 1/3 | 0.824, 0.0, 1.0 |
Scores show the share of the fixed grading items that were correct (1.0 is full marks). Only runs scoring 1.0 passed.
The research runs 2 and 3 scored 0.778, island run 1 scored 0.824, and island run 2 scored 0.0. This record does not identify why those scores were low.
What this shows
- There were no abnormal exit codes, timeouts, or errors. Every failure was a grading result.
- In research and island, only 1 of the 3 runs passed. With so few runs, we cannot tell whether passing is reliable.
- The inputs are synthetic. We do not know whether real work documents would give the same results.