↓Skip to main content
  1. Posts/

A Personalized LLM Benchmark

·2 mins
AI AI Benchmarks
Table of Contents

Intro #

So I’ve got a personal AI box at home now that I use for more and more tasks. I do some agentic stuff, made a radio station with actual talk show hosts, do RAG searches and all sorts of stuff. Anyways, I’ve been trying to figure out what model is best, and if the bigger dense models are even better despite the speed. My inference box is a strix-halo machine with 128 GB of RAM (Framework Desktop) so while it can run big models, it’s not the fastest.

What I built #

To help figure this out I decided that I’d do a custom benchmark, based on actual tasks I do. So I built a benchmark with 26 tests across seven categories;

  • Vision & OCR
  • Coding
  • Technical Reasoning
  • Analysis and Writing
  • Tool Use
  • Long Context Retrieval
  • Creative Generation

Everything is scored 0-10, about half of it is objective, code runs and produces the right answer or it doesn’t, and half graded by a separate judge model (from a different family than those tested).

Note, I’m not going to share the specifics of the benchmark, except for one task that I find kind of fun. I’m not sharing the specifics because of a few reasons, one privacy. The tasks are built from my own photos, records, and private documents. Another is contamination, while this blog is just flotsam in the ocean, it will eventually get scraped and those tests may get learned. I will share the aggregate scores though, and will probably revisit whenever a new model that fits my criteria comes out. I just won’t publish the task information.

Results #

Benchmark Results

Findings #

Scale definitely wins, but the ceiling is close. Small models do more than I thought, and fail hard. The easy categories don’t do much, and speed is more important than I thought. Gemma 4 31B was a clear winner, except for the speed.

What Changed #

I actually stopped using the Qwen MOE after this. The Gemma MOE was actually better overall and slightly faster. I also shifted some of my vision tasks to the Gemma-4 2B model because it was perfectly acceptable and way faster.

The Public Piece #

There is one piece I want to show off, The Tarot Card Generation task. I’ve tasked each model with generating a SVG of “The Fool” from tarot. This site is located at https://thefoolbench.mattwmarshall.com and will be updated as I add more models to the benchmark. It’s cute.