AI agent comparison · goose

goose record
3 of 9 passed

Version v1.54.0. Local model gpt-oss:20b, run on Mac Studio 2 (M5 Ultra, 96GB).

Results by task

TaskPassedScores (runs 1, 2, 3)Main cause
Research0/30.0, 0.0, 0.0The read tool was not available
Fetch-failure sorting3/31.0, 1.0, 1.0—
Island figures0/30.0, 0.0, 0.0Read the input, then ended without writing the result

In the research task, all 3 runs failed to read the input files, so no result file was produced. The log repeatedly shows "Tool 'read' was not advertised for this model turn."

In the island task, all 3 runs read the input with a shell command, then ended without calling any tool to write the file. In run 3, the JSON was printed to the output, but no result file was created.

All three fetch-failure runs passed. The results differ sharply by task, which is the main pattern in this record.

What this shows

  • The exit code was 0 in all runs. These were not connection or timeout failures.
  • The research and island failures were not re-measured under changed conditions, so a different setup might give different results.
  • The inputs are synthetic. We do not know whether real work documents would give the same results.