Practical tasks for local language models

Results from 30
practical AI agent tasks

Two Mac Studios tried tasks in calculation, formatting, extraction, classification, and checking.

Results

On October 29, 2026, all 30 initial tasks completed both creation and review. The models passed 27 of 30 creation tasks on the first attempt; after retries, all 30 passed.

Mac and modelCreationReviewTokens used
Studio 1 · M5 Max 36GB
qwen3.8:27b
14/14 passed16/16 correct330,898
Studio 2 · M5 Ultra 96GB
qwen3.6:35b-a3b
16/19 passed
(3 passed after retry)
14/16 correct
(2 corrected after retry)
424,652

The 30 tasks covered five types: calculation, table formatting, text extraction, inquiry classification, and invoice checking. Each type included two ordinary tasks, two traps, and two tasks with insufficient input.

What the review stage measured (30 creation tasks + 20 error-detection tasks)

In the initial 30 tasks, reviewers saw only answers that had already matched the expected result. Across 32 review attempts, 30 judgments were correct. Both incorrect judgments marked a correct answer for revision. The server rejected those judgments, and retries were correct.

To measure whether reviewers can catch wrong answers, a separate review-only set of 20 tasks (15 seeded incorrect candidates with deliberate rule violations and 5 correct candidates) was also completed on October 29, 2026.

Metric (20 review-only tasks)Result
Correct judgment on first attempt16/20 (80%)
No valid output on first attempt4/20 (20%: 3 incomplete processes, 1 expired lease)
Completed with correct judgment after retry20/20
False acceptance of wrong candidates (15)0 (0 across all 14 valid first-attempt judgments)
False rejection of right candidates (5)0 (0 across both valid first-attempt judgments)
Studio 1 (qwen3.8:27b)9/9 valid judgments correct (8/8 on first attempt)
Studio 2 (qwen3.6:35b-a3b)11/11 valid judgments correct, 4 without valid output (8 correct and 4 without output on first attempt)

Total token usage recorded in the ledger for the 20 review tasks was 576,440 tokens (including one full-budget charge for the attempt whose lease expired without a saved result). While all 20 valid judgments had 0 false acceptances and 0 false rejections, the 4 first-attempt runs without valid output are not excluded to claim "100% on the first attempt."

What these results do and do not show

  • In the initial 30 tasks, 27 of 30 creation tasks were correct on the first attempt. All three misses were trap tasks and passed after retries.
  • All 10 tasks with insufficient input were correctly marked as indeterminable on the first attempt.
  • In the 20 review-only tasks, false acceptance across 15 seeded wrong answers was 0, and false rejection across 5 right answers was 0 (16/20 completed on the first attempt; 4/20 produced no valid output and completed on retry).
  • The 15 wrong candidates contained deliberate rule violations and do not represent the natural error distribution of models.
  • This was one model per Mac across 30 + 20 tasks that can be recomputed from instructions and inputs. It does not rank models in general or establish accuracy on other work.

Setup and records

The local models ran on Mac Studio 1 (M5 Max, 36GB, qwen3.8:27b) and Mac Studio 2 (M5 Ultra, 96GB, qwen3.6:35b-a3b), with a 65,536-token context. No external AI was used. In the initial 30 tasks, the server compared each creation answer with the expected JSON and sent only exact matches to review. In the 20 review-only tasks, seeded candidates (15 wrong, 5 right) were sent directly to review.

The workflow records were read from D1 on October 29, 2026 (08:11 JST for the 30 tasks and 12:11 JST for the 20 review tasks). The site operations page also describes the results and caveats.