0:00  The best model you can run at home naturally depends on the hardware that you have.
0:02  In this video, we're going to be talking about the 8
0:05  gigabyte VRAM class of GPUs with
0:08  alongside it, about 12 gigabytes of
0:11  system memory available for inference.
0:14  So first, the configuration I recommend.
0:16  It outperformed a dozen others on my benchmark.
0:19  Get yourself the Qwen3.6 35B-A3B.
0:23  We're looking at the IQ3_XXS Model Quant.
0:27  Download the mmproj Vision Tower if you want,
0:30  and that'll be resident in main memory.
0:33  Use the Q4, KV cache at 128K context,
0:37  and we'll be using the sampling parameters from the model card.
0:41  This model's performance degrades gracefully as you
0:44  offload weights to system memory.
0:46  The vast majority of the weights in the 35B
0:49  model are experts that are only loaded at about a 3% rate.
0:53  So they can be offloaded to system memory with a pretty small
0:56  but noticeable loss in performance.
0:59  In llama.cpp, --n-cpu-moe
1:03  is the flag that allows us to run a model that's 30 gigabytes
1:06  or more in full precision on only an 8 gigabyte card.
1:10  So the configuration is on screen.
1:12  It's also in the YouTube video description.
1:14  I've got it on GitHub.
1:15  I would encourage you to try this out.
1:17  If you start downloading the model now, you can probably use it
1:20  later in the video, if you're on a fast internet connection.
1:23  It's fast enough to be interactive.
1:25  It's a great coding assistant.
1:27  So far, it's the best that I've found under these hardware
1:30  budget constraints of 8 gigabytes of VRAM and 12 gigabytes of system RAM.
1:35  So, what can you expect in this video?
1:37  We're going to talk about the hardware.
1:39  This stuff is incredibly cheap.
1:40  You can run this on anything.
1:41  It's basically the potato camera equivalent of GPU
1:45  LLM setups.
1:47  75 bucks to $200 on eBay
1:50  shipped within the US.
1:52  What can you expect in terms of speed and intelligence?
1:55  We're going to look at some of the difficult problems that this model
1:58  succeeds at, and then over the cusp, what
2:01  it fails at, and we're going to compare it in aggregate
2:04  against a bunch of other models I've tested on similar local hardware.
2:10  Third, testing methodology.
2:13  Disconfirming evidence, how I evaluate a
2:16  model and all the configurations that we've looked at.
2:19  And then finally, we'll wrap up with some future work.
2:22  What I want to improve about the testing framework that I have so
2:25  far, and what I don't know yet about the performance
2:28  of models like this.
2:30  Okay, a little bit more about hardware configurations.
2:32  We're talking about a server or a PC with
2:36  16 gigabytes minimum of system RAM.
2:39  You can be running Linux or Windows if you're a savage.
2:42  The GPUs that we're targeting today, like I said, are very cheap.
2:45  The 1080, I saw one for about 80 bucks on eBay.
2:49  The 2080 goes for a little bit more, but not too much more.
2:52  At about $100.
2:54  And then there's the 3050, uh, which is coming
2:57  in under $200, but be careful because some of the 3050s
3:00  are 6 gigabyte cards.
3:01  So for a lot of people, I think you already have one of these sitting around at home.
3:05  And if we look at the steam hardware survey data,
3:09  basically anybody that's buying and playing games on steam.
3:13  And responding to their survey, already has hardware,
3:17  like what we're talking about today.
3:18  This is a very accessible tier.
3:20  So I'm lucky enough to have better hardware at home.
3:23  I have a 4090 and a 3090 that I program on, and
3:26  luckily the prices of those have gone up since I bought them,
3:29  so I could probably sell them for a profit now.
3:31  But I had not tried the
3:34  configuration that I am testing today, and I wanted to correct that.
3:38  So last week on stream live on YouTube,
3:41  we pulled out the 3090 from my infant server.
3:46  We stuck a 1070 Ti in there, literally $70 card.
3:51  And um, the the card had been sitting around
3:55  at my house for like 8 years.
3:58  It's not modern stuff, and we fired it up to do some programming.
4:01  So we were getting about 400 tokens per 2nd
4:04  prompts processing, which does feel a little bit slow, and
4:07  we were using about 10 gigabytes of system RAM.
4:10  And on stream.
4:12  We were getting 25 tokens per second, which for
4:15  me, that's pretty close to the limit where I'm like, okay,
4:18  this is usable and interactive.
4:21  And it doesn't feel as fast as if you're using Claude for sure.
4:25  And it doesn't, you know, I'm used to a much faster tokens per 2nd output.
4:30  So it's not optimal, but we're talking about budget hardware here.
4:34  So we have to keep our expectations in check.
4:38  The main issue, I think, that we had with this setup on
4:41  stream was where the model was doing work.
4:44  testing its own work, with taking screenshots
4:47  and then reading them and inspecting them.
4:49  And this took a really long time to read each image.
4:52  It's important to me if I'm going to make a recommendation to actually try
4:55  it, and so I made sure that we did that live to kind of experience
4:58  what this thing is like to actually use.
5:02  And I'm not going to say I was impressed with the results, but
5:05  I will say that I expected worse.
5:08  Obviously, it's better to have a $1000
5:10  GPU than a $70 GPU.
5:13  This isn't rocket science.
5:15  Okay, so what about the top end of the intelligence?
5:17  What can you expect from this?
5:19  One of the hardest task ladders in my private task set,
5:22  Bob Bench, involves letting an agent loose on a docker image
5:25  where there's a working, running, ComfyUI service.
5:30  The agents are instructed to do various tasks that involve querying
5:34  the running system, to see what servers are running, reading the
5:37  source code to determine what functionality exists.
5:40  You have to probe the API and make sure that you
5:43  can automate your request to the API, and then finally...
5:47  You know, agents have to verify their work through the file system.
5:51  So this is a very agentic kind of situation.
5:54  For this particular task that I'm going to go over, the agent is instructed
5:57  to run a sweep over a bunch of different parameters, hitting
6:01  the ComfyUI, and it's not given.
6:03  A lot of guardrails or instructions about how to do it.
6:05  This is more of a, please accomplish this for me, and then you're
6:08  free to figure out how it works, rather than a, here
6:12  is a spec, and I want the agent and the model to go off and
6:15  implement some.
6:16  Some thing against the spec.
6:18  So on this particular task, there's a 900-second timeout,
6:21  and the 35B A3B is running slowly, so time is a concern.
6:25  It finished in about 750 seconds.
6:28  It made about 70 tool calls.
6:30  And over the course of 64 turns, it
6:34  did a great job on this task.
6:35  Now, it did fuck some shit up along the way.
6:38  And you know what?
6:39  That's kind of expected.
6:40  It detected that there was a problem
6:43  and it adjusted its course,
6:46  all excellent behavior that
6:49  you want in an agent to be able to determine that there was an
6:53  issue and that it has to undo something, roll back a change.
6:57  And you see this with much more intelligent agents like Claude Opus, too.
7:00  So difficult task, lots of ambiguity,
7:04  completed in under 900 seconds, can't ask for more
7:07  in terms of a budget configuration.
7:09  So I'll play a little bit of the...
7:10  the footage from the live stream, from the unbiased
7:13  Bob channel, where we installed the 1070 Ti, and we did programming with this thing.
7:18  We created the interactive solar system visualization with
7:22  a prompt directly YOLO from a subscriber watching on
7:25  in the chat.
7:26  Yeah, and I also want to stress the insightful and intelligent
7:29  commentary that I gave.
7:31  As we were doing this live.
7:35  We need sunspots.
7:37  At sunspots.
7:40  And I'm pretty sure that this
7:43  is nowhere near to scale.
7:48  There's the sun spots.
7:50  This reminds me of those egg bites at Starbucks.
7:53  It really has that vibe.
7:55  Okay, so that's specific tasks, and a couple,
7:58  you know, basically cherry picked examples of where the model does a good job.
8:02  But in the aggregate, let's talk a little bit
8:05  more about Bob Bench Wide, which is my 86
8:08  task, held out private, not in the training
8:11  data task set, and we'll talk
8:14  about how this
8:17  model fits in broadly.
8:19  So each point on Bob Bench wide is a
8:22  given model and a given configuration and its score versus
8:25  how long it took to get there.
8:28  So models like Qwen3.8 27B,
8:32  where they change the reasoning and the model thinks forever, they
8:35  can get a lower score, uh, in terms
8:39  of time, but then a very high score in terms of
8:42  total accuracy if you let them cook forever.
8:45  My benchmark set penalize
8:48  his models for thinking for a long time.
8:50  And the tasks have timeouts of
8:53  usually 15 or 30 minutes per task, depending on the difficulty.
8:57  Each point on the bob bench wide chart is a benchmark
9:00  run on a given model and a given configuration.
9:03  The top left quadrant are fast and smart configurations,
9:06  Qwen 3.6 completes the entire benchmark, in
9:10  less than 5 hours and gets a 90%.
9:13  But that's a higher model quant with more memory and with MTP enabled.
9:17  So the red lines are your trade-offs with
9:20  24 gigabytes of memory available to you and a fast graphics card.
9:24  It's not exhaustive, but this is.
9:26  a parado curve for 24 gigaway cards.
9:30  So, the X axis here measures how it retarded the model is,
9:33  if you're way out here to the right, say, like IBM
9:36  granite, it means you took forever to execute, and
9:40  the model was super slow.
9:41  And as we go down, that's the other retarded axis
9:44  where the models get a lot dumber, like if you're IBM
9:47  granite, way down here at the bottom, you got a very low score.
9:51  So that's a little bit about exploring the curves on the BBW,
9:55  something I'm a really big fan of.
9:56  Methodology.
9:57  Now, I'm not a scientist, but I apply techniques from science to this problem.
10:01  So, first, get yourself that falsifiable hypothesis,
10:05  something specific and actionable and something that you
10:08  can disprove, okay?
10:09  And then for small models that fit on consumer hardware,
10:13  nothing beats Qwen so far.
10:15  So Qwen's going to be our baseline.
10:17  So we're going to say, look, Qwen, 3.6,
10:20  35B-A3B, is the best model
10:23  We're gonna do a lot of offloading the CPU on the 8 gigabyte card.
10:28  And look, this is a very defensible
10:31  baseline position.
10:34  And then we're going to go off and we're going to look for disconfirming evidence, okay?
10:38  I don't think scientists do that today.
10:40  I think they're more into publishing papers that they wrote with chat GPT,
10:43  but what you do is you go and you
10:46  say, well, if this wasn't true,
10:50  you know, how could I, how could I prove it?
10:52  And so we can get creative by looking at different model
10:55  families, different quantizations,
10:58  either extreme high or extreme low end of quantizations, we have both of those here.
11:03  And so this is how I go about trying to figure
11:07  out the best configuration.
11:09  In my experience, in the workplace and at
11:12  home, decisions are rarely made with a
11:15  framework like this.
11:17  Now with open weights models and open source inference,
11:20  we have the freedom to methodically
11:23  and methodologically approach this.
11:26  Is methodologically a word?
11:28  don't actually know, but we can approach it diligently and methodically.
11:32  and methodologically.
11:35  So, I tested over six model families and a dozen configurations,
11:38  all the while knowing that Qwen 3.6,
11:41  35B-A3B was probably going to be the best, and it was.
11:46  So each test, each one of these blue dots was
11:49  run on the same hardware, the same harness, the same version
11:52  of llama.cpp, the same task timeouts,
11:55  the same system prompts, you get the picture.
11:57  We hold everything constant that we can.
12:00  And then we switch out the underlying model, and
12:04  we adjust the amount of KV cache and the context
12:07  size that's available, subject to that 8 gigabyte VRAM cap.
12:11  So I think the true minimum on KV cache is probably like 64K
12:15  or maybe 48K.
12:17  Uh, but a couple of the configs with smaller
12:20  sizes were tried here, where required.
12:22  So honorable mentions here are reap.
12:25  It really surprised me to try a modification
12:28  to Qwen that did pretty well.
12:31  And as we can see, reap is,
12:34  it's a model where they, they prune the experts
12:38  that get used least frequently.
12:40  But anyways, the reap guys did a really good job of making a
12:43  slightly smaller model that was a little bit dumber and took
12:46  a little bit longer, but overall, it might be within noise and
12:50  it might be just about as good as Qwen 3.6.
12:53  But I don't see a reason to choose it over the
12:56  Qwen 3.6 base.
12:58  Uh, granite, IBM granite was absolutely horrible.
13:02  Bonsai has a couple points on the graph that are sort
13:05  of okay.
13:07  Gemma 4 did not perform well
13:10  in this setup.
13:11  Maybe it deserves being tried with a
13:15  lower model quant, but I doubt that with this
13:18  initial score getting a 55%.
13:21  I doubt that Gemma is a contender.
13:24  And then the dense configurations between Qwen 3.8,
13:28  27B and Bonsai just didn't
13:31  have the chops to complete the task.
13:34  Another one, I tried a couple different setups that had other
13:37  a mixture of experts models in the 30 B class,
13:40  Gemma 4 and the old GLM 47
13:44  Flash, and they didn't, they couldn't hang with
13:47  Qwen 3.6, 35B-A3B.
13:50  Not exactly a surprise.
13:52  Okay, so future work, Bob Bench
13:55  needs some more tasks that are diverse in terms
13:58  of the language used.
14:00  It focuses a lot on C, C+,
14:04  Python, Bash, that kind of thing.
14:07  It would be great to have tasks that have Swift or
14:10  Android, iOS development, embedded, which
14:13  is going to require getting a little bit creative with a test harness, but I think would be very valuable.
14:17  And maybe there are configurations where offloading
14:20  the KV cache system RAM can have better performance, and
14:23  I didn't spend an entire run, because these
14:26  runs take, you know, 5 or 10 hours plus analysis time.
14:31  So I haven't tried one where we offload
14:34  the KV cache's system memory.
14:36  I have my doubts, but, you know,
14:39  there's also other model families to try.
14:41  So, having the BBW be a static set
14:44  of tasks with the static evaluations and having it be
14:48  private is pretty valuable because in the future, say Qwen 4 releases.
14:52  We can easily run it, you know, do the full 10 hour run.
14:56  Then we have this whole family of task to
14:59  evaluate it against.
15:01  And I see a lot of benchmarks online where they say,
15:04  oh, um, you know, Opus 4.6
15:08  got a 95 on this benchmark.
15:12  And it doesn't make any sense to me because Opus 4.6 isn't one thing.
15:16  There are a 1000 different trade-offs on the back end.
15:20  In terms of what inference framework
15:23  they were actually running it on in terms of the model quant that
15:26  they were using in terms of their runtime parameters, and all
15:29  that gets collapsed when we look at these benchmark results.
15:34  And so I struggle to find
15:38  benchmark results that I enjoy online for that reason.
15:41  So, suffice that to say,
15:44  we'll continue adding to the BBW Alpha,
15:48  which is this initial version of BBW.
15:51  So, and it's the same, you know, new techniques, new
15:54  inference features, more inference frameworks,
15:57  I would love to run this against VLLM
16:00  or SG Lang or, you know, there's a number of other backends that you can use.
16:05  I've simply used llama.cpp because I like it.
16:08  I like the fact that it's mostly, you know, C and Cuda.
16:13  And I enjoy working with it because of its simplicity.
16:16  You just compile it, you've got a binary on disk.
16:21  Okay, that's it for 8 gigabyte setups.
16:23  If you enjoyed this, consider commenting, join us on Discord,
16:26  hop in for a live stream.
16:28  Every one of these venues has a lot of high quality technical conversations
16:31  going on every single day, which I'm grateful for.
16:34  If you want to watch this without advertisements, the full video is
16:38  on unbiasedbob.com as a torrent.
16:40  Also, consider a YouTube membership, if you
16:43  want to go above and beyond in your support of the channel, we
16:46  just got accepted into the YouTube partner
16:49  program after 2 months of being a channel.
16:51  This is quite an accomplishment, I think.
16:53  And so thank you for watching and I will see you in the next one.