Practical tasks for local language models
Results from 30
practical AI agent tasks
Two Mac Studios tried tasks in calculation, formatting, extraction, classification, and checking.
Results
On October 29, 2026, all 30 initial tasks completed both creation and review. The models passed 27 of 30 creation tasks on the first attempt; after retries, all 30 passed.
| Mac and model | Creation | Review | Tokens used |
|---|---|---|---|
| Studio 1 · M5 Max 36GB qwen3.8:27b | 14/14 passed | 16/16 correct | 330,898 |
| Studio 2 · M5 Ultra 96GB qwen3.6:35b-a3b | 16/19 passed (3 passed after retry) | 14/16 correct (2 corrected after retry) | 424,652 |
The 30 tasks covered five types: calculation, table formatting, text extraction, inquiry classification, and invoice checking. Each type included two ordinary tasks, two traps, and two tasks with insufficient input.
What the review stage measured (30 creation tasks + 20 error-detection tasks)
In the initial 30 tasks, reviewers saw only answers that had already matched the expected result. Across 32 review attempts, 30 judgments were correct. Both incorrect judgments marked a correct answer for revision. The server rejected those judgments, and retries were correct.
To measure whether reviewers can catch wrong answers, a separate review-only set of 20 tasks (15 seeded incorrect candidates with deliberate rule violations and 5 correct candidates) was also completed on October 29, 2026.
| Metric (20 review-only tasks) | Result |
|---|---|
| Correct judgment on first attempt | 16/20 (80%) |
| No valid output on first attempt | 4/20 (20%: 3 incomplete processes, 1 expired lease) |
| Completed with correct judgment after retry | 20/20 |
| False acceptance of wrong candidates (15) | 0 (0 across all 14 valid first-attempt judgments) |
| False rejection of right candidates (5) | 0 (0 across both valid first-attempt judgments) |
| Studio 1 (qwen3.8:27b) | 9/9 valid judgments correct (8/8 on first attempt) |
| Studio 2 (qwen3.6:35b-a3b) | 11/11 valid judgments correct, 4 without valid output (8 correct and 4 without output on first attempt) |
Total token usage recorded in the ledger for the 20 review tasks was 576,440 tokens (including one full-budget charge for the attempt whose lease expired without a saved result). While all 20 valid judgments had 0 false acceptances and 0 false rejections, the 4 first-attempt runs without valid output are not excluded to claim "100% on the first attempt."
What these results do and do not show
- In the initial 30 tasks, 27 of 30 creation tasks were correct on the first attempt. All three misses were trap tasks and passed after retries.
- All 10 tasks with insufficient input were correctly marked as indeterminable on the first attempt.
- In the 20 review-only tasks, false acceptance across 15 seeded wrong answers was 0, and false rejection across 5 right answers was 0 (16/20 completed on the first attempt; 4/20 produced no valid output and completed on retry).
- The 15 wrong candidates contained deliberate rule violations and do not represent the natural error distribution of models.
- This was one model per Mac across 30 + 20 tasks that can be recomputed from instructions and inputs. It does not rank models in general or establish accuracy on other work.
Setup and records
The local models ran on Mac Studio 1 (M5 Max, 36GB, qwen3.8:27b) and Mac Studio 2 (M5 Ultra, 96GB, qwen3.6:35b-a3b), with a 65,536-token context. No external AI was used. In the initial 30 tasks, the server compared each creation answer with the expected JSON and sent only exact matches to review. In the 20 review-only tasks, seeded candidates (15 wrong, 5 right) were sent directly to review.
The workflow records were read from D1 on October 29, 2026 (08:11 JST for the 30 tasks and 12:11 JST for the 20 review tasks). The site operations page also describes the results and caveats.