An 8 GB GPU and about 12 GB of system RAM available for inference. Here is the Qwen3.6 35B-A3B configuration I recommend, what it feels like on a 1070 Ti, and how it compares with the other setups I tested on Bob-Bench Wide.
Want a copy without ads, an account, or tracking? Grab the torrent (112.1 MB).
It includes a webseed, so it downloads even when nobody else is seeding. Please seed it if you can.
Magnet link
Qwen3.6 35B-A3B at UD-IQ3_XXS, 128k context, Q4 KV cache, and CPU expert offloading. This is the command supplied with the video.
llama-server \ --host 0.0.0.0 --port 8091 \ -hf unsloth/Qwen3.6-35B-A3B-GGUF:UD-IQ3_XXS \ -c 131072 -ngl 99 --n-cpu-moe 25 \ -b 2048 -ub 2048 \ --cache-type-k q4_0 --cache-type-v q4_0 \ --cache-ram 0 --parallel 1 --jinja \ --reasoning on --reasoning-budget 8192 \ --temp 0.7 --top-k 20 --top-p 0.8 --min-p 0.0 \ --presence-penalty 0.0 --repeat-penalty 1.0 \ --n-predict 32768
Download the config and notes.
The first run downloads the GGUF from Hugging Face (about 13 GB).
The optional mmproj vision tower can be omitted with --no-mmproj.
Use --n-cpu-moe 25 for the dedicated inference setup shown.
If the same GPU drives your display, the supplied notes recommend 23.
If VRAM is tight, reduce context to -c 65536 first.
The 11 configurations in the video’s generated resource package, dated 23 September 2026. Each uses the same 86-task BBW Alpha set. Score is the mean normalized task score; TO counts task timeouts.
| Configuration | Score | Hours | TO |
|---|---|---|---|
| Qwen3.6-35B-A3B IQ3_XXS (8GB) | 80.59% | 9.94 | 16 |
| Qwen3.6-35B-A3B REAP-48 (8GB) | 78.12% | 10.14 | 16 |
| Qwen3.6-35B-A3B IQ2_M (8GB) | 77.53% | 11.20 | 17 |
| Bonsai-2-27B PTQ1_0 48k (8GB) | 71.18% | 11.23 | 21 |
| Qwen3.5-9B Q4_K_M (8GB) | 66.89% | 11.34 | 23 |
| Gemma-4-26B-A4B IQ3_XXS (8GB) | 63.18% | 12.74 | 19 |
| HauhauCS 35B-A3B IQ2_M (8GB) | 62.28% | 11.07 | 24 |
| Gemma-4-12B IQ4_XS 40k (8GB) | 54.66% | 5.84 | 5 |
| Bonsai-27B Q1_0 128k (8GB) | 50.04% | 13.12 | 36 |
| Granite-4.2-8B Q4_K_M 48k (8GB) | 48.41% | 16.60 | 46 |
| Qwen3.8-27B IQ2_XXS 16k (8GB) | 38.66% | 2.65 | 1 |
The 8 GB label is the measured model-process VRAM envelope. Benchmark runs used 24 GB 3090 Ti and 4090 cards; wall-clock times are not normalized across GPUs. The separate 1070 Ti trial is discussed in the video.
From the final-cut captions. Model names and technical terms have been corrected; timestamp links open the corresponding point on YouTube.
0:00The best model you can run at home naturally depends on the hardware that you have. In this video, we're going to be talking about the 8 gigabyte VRAM class of GPUs with alongside it, about 12 gigabytes of system memory available for inference.
0:14So first, the configuration I recommend. It outperformed a dozen others on my benchmark. Get yourself the Qwen3.6 35B-A3B. We're looking at the IQ3_XXS Model Quant. Download the mmproj Vision Tower if you want, and that'll be resident in main memory.
0:33Use the Q4, KV cache at 128K context, and we'll be using the sampling parameters from the model card. This model's performance degrades gracefully as you offload weights to system memory. The vast majority of the weights in the 35B model are experts that are only loaded at about a 3% rate.
0:53So they can be offloaded to system memory with a pretty small but noticeable loss in performance. In llama.cpp, --n-cpu-moe is the flag that allows us to run a model that's 30 gigabytes or more in full precision on only an 8 gigabyte card.
1:10So the configuration is on screen. It's also in the YouTube video description. I've got it on GitHub. I would encourage you to try this out. If you start downloading the model now, you can probably use it later in the video, if you're on a fast internet connection.
1:23It's fast enough to be interactive. It's a great coding assistant. So far, it's the best that I've found under these hardware budget constraints of 8 gigabytes of VRAM and 12 gigabytes of system RAM.
1:35So, what can you expect in this video? We're going to talk about the hardware. This stuff is incredibly cheap. You can run this on anything. It's basically the potato camera equivalent of GPU LLM setups. 75 bucks to $200 on eBay shipped within the US.
1:52What can you expect in terms of speed and intelligence? We're going to look at some of the difficult problems that this model succeeds at, and then over the cusp, what it fails at, and we're going to compare it in aggregate against a bunch of other models I've tested on similar local hardware.
2:10Third, testing methodology. Disconfirming evidence, how I evaluate a model and all the configurations that we've looked at. And then finally, we'll wrap up with some future work. What I want to improve about the testing framework that I have so far, and what I don't know yet about the performance of models like this.
2:30Okay, a little bit more about hardware configurations. We're talking about a server or a PC with 16 gigabytes minimum of system RAM. You can be running Linux or Windows if you're a savage. The GPUs that we're targeting today, like I said, are very cheap.
2:45The 1080, I saw one for about 80 bucks on eBay. The 2080 goes for a little bit more, but not too much more. At about $100. And then there's the 3050, uh, which is coming in under $200, but be careful because some of the 3050s are 6 gigabyte cards.
3:01So for a lot of people, I think you already have one of these sitting around at home. And if we look at the steam hardware survey data, basically anybody that's buying and playing games on steam. And responding to their survey, already has hardware, like what we're talking about today.
3:18This is a very accessible tier. So I'm lucky enough to have better hardware at home. I have a 4090 and a 3090 that I program on, and luckily the prices of those have gone up since I bought them, so I could probably sell them for a profit now.
3:31But I had not tried the configuration that I am testing today, and I wanted to correct that. So last week on stream live on YouTube, we pulled out the 3090 from my infant server. We stuck a 1070 Ti in there, literally $70 card.
3:51And um, the the card had been sitting around at my house for like 8 years. It's not modern stuff, and we fired it up to do some programming. So we were getting about 400 tokens per 2nd prompts processing, which does feel a little bit slow, and we were using about 10 gigabytes of system RAM.
4:10And on stream. We were getting 25 tokens per second, which for me, that's pretty close to the limit where I'm like, okay, this is usable and interactive. And it doesn't feel as fast as if you're using Claude for sure. And it doesn't, you know, I'm used to a much faster tokens per 2nd output.
4:30So it's not optimal, but we're talking about budget hardware here. So we have to keep our expectations in check. The main issue, I think, that we had with this setup on stream was where the model was doing work. testing its own work, with taking screenshots and then reading them and inspecting them.
4:49And this took a really long time to read each image. It's important to me if I'm going to make a recommendation to actually try it, and so I made sure that we did that live to kind of experience what this thing is like to actually use.
5:02And I'm not going to say I was impressed with the results, but I will say that I expected worse. Obviously, it's better to have a $1000 GPU than a $70 GPU. This isn't rocket science.
5:15Okay, so what about the top end of the intelligence? What can you expect from this? One of the hardest task ladders in my private task set, Bob Bench, involves letting an agent loose on a docker image where there's a working, running, ComfyUI service.
5:30The agents are instructed to do various tasks that involve querying the running system, to see what servers are running, reading the source code to determine what functionality exists. You have to probe the API and make sure that you can automate your request to the API, and then finally...
5:47You know, agents have to verify their work through the file system. So this is a very agentic kind of situation. For this particular task that I'm going to go over, the agent is instructed to run a sweep over a bunch of different parameters, hitting the ComfyUI, and it's not given.
6:03A lot of guardrails or instructions about how to do it. This is more of a, please accomplish this for me, and then you're free to figure out how it works, rather than a, here is a spec, and I want the agent and the model to go off and implement some.
6:16Some thing against the spec. So on this particular task, there's a 900-second timeout, and the 35B A3B is running slowly, so time is a concern. It finished in about 750 seconds. It made about 70 tool calls. And over the course of 64 turns, it did a great job on this task.
6:35Now, it did fuck some shit up along the way. And you know what? That's kind of expected. It detected that there was a problem and it adjusted its course, all excellent behavior that you want in an agent to be able to determine that there was an issue and that it has to undo something, roll back a change.
6:57And you see this with much more intelligent agents like Claude Opus, too. So difficult task, lots of ambiguity, completed in under 900 seconds, can't ask for more in terms of a budget configuration.
7:09So I'll play a little bit of the... the footage from the live stream, from the unbiased Bob channel, where we installed the 1070 Ti, and we did programming with this thing. We created the interactive solar system visualization with a prompt directly YOLO from a subscriber watching on in the chat.
7:26Yeah, and I also want to stress the insightful and intelligent commentary that I gave. As we were doing this live. We need sunspots. At sunspots. And I'm pretty sure that this is nowhere near to scale. There's the sun spots.
7:50This reminds me of those egg bites at Starbucks. It really has that vibe.
7:55Okay, so that's specific tasks, and a couple, you know, basically cherry picked examples of where the model does a good job. But in the aggregate, let's talk a little bit more about Bob Bench Wide, which is my 86 task, held out private, not in the training data task set, and we'll talk about how this model fits in broadly.
8:19So each point on Bob Bench wide is a given model and a given configuration and its score versus how long it took to get there. So models like Qwen3.8 27B, where they change the reasoning and the model thinks forever, they can get a lower score, uh, in terms of time, but then a very high score in terms of total accuracy if you let them cook forever.
8:45My benchmark set penalize his models for thinking for a long time. And the tasks have timeouts of usually 15 or 30 minutes per task, depending on the difficulty. Each point on the bob bench wide chart is a benchmark run on a given model and a given configuration.
9:03The top left quadrant are fast and smart configurations, Qwen 3.6 completes the entire benchmark, in less than 5 hours and gets a 90%. But that's a higher model quant with more memory and with MTP enabled. So the red lines are your trade-offs with 24 gigabytes of memory available to you and a fast graphics card.
9:24It's not exhaustive, but this is. a parado curve for 24 gigaway cards. So, the X axis here measures how it retarded the model is, if you're way out here to the right, say, like IBM granite, it means you took forever to execute, and the model was super slow.
9:41And as we go down, that's the other retarded axis where the models get a lot dumber, like if you're IBM granite, way down here at the bottom, you got a very low score. So that's a little bit about exploring the curves on the BBW, something I'm a really big fan of.
9:56Methodology. Now, I'm not a scientist, but I apply techniques from science to this problem. So, first, get yourself that falsifiable hypothesis, something specific and actionable and something that you can disprove, okay?
10:09And then for small models that fit on consumer hardware, nothing beats Qwen so far. So Qwen's going to be our baseline. So we're going to say, look, Qwen, 3.6, 35B-A3B, is the best model We're gonna do a lot of offloading the CPU on the 8 gigabyte card.
10:28And look, this is a very defensible baseline position. And then we're going to go off and we're going to look for disconfirming evidence, okay? I don't think scientists do that today. I think they're more into publishing papers that they wrote with chat GPT, but what you do is you go and you say, well, if this wasn't true, you know, how could I, how could I prove it?
10:52And so we can get creative by looking at different model families, different quantizations, either extreme high or extreme low end of quantizations, we have both of those here. And so this is how I go about trying to figure out the best configuration.
11:09In my experience, in the workplace and at home, decisions are rarely made with a framework like this. Now with open weights models and open source inference, we have the freedom to methodically and methodologically approach this.
11:26Is methodologically a word? don't actually know, but we can approach it diligently and methodically. and methodologically. So, I tested over six model families and a dozen configurations, all the while knowing that Qwen 3.6, 35B-A3B was probably going to be the best, and it was.
11:46So each test, each one of these blue dots was run on the same hardware, the same harness, the same version of llama.cpp, the same task timeouts, the same system prompts, you get the picture. We hold everything constant that we can.
12:00And then we switch out the underlying model, and we adjust the amount of KV cache and the context size that's available, subject to that 8 gigabyte VRAM cap. So I think the true minimum on KV cache is probably like 64K or maybe 48K.
12:17Uh, but a couple of the configs with smaller sizes were tried here, where required.
12:22So honorable mentions here are reap. It really surprised me to try a modification to Qwen that did pretty well. And as we can see, reap is, it's a model where they, they prune the experts that get used least frequently. But anyways, the reap guys did a really good job of making a slightly smaller model that was a little bit dumber and took a little bit longer, but overall, it might be within noise and it might be just about as good as Qwen 3.6.
12:53But I don't see a reason to choose it over the Qwen 3.6 base. Uh, granite, IBM granite was absolutely horrible. Bonsai has a couple points on the graph that are sort of okay. Gemma 4 did not perform well in this setup. Maybe it deserves being tried with a lower model quant, but I doubt that with this initial score getting a 55%.
13:21I doubt that Gemma is a contender. And then the dense configurations between Qwen 3.8, 27B and Bonsai just didn't have the chops to complete the task. Another one, I tried a couple different setups that had other a mixture of experts models in the 30 B class, Gemma 4 and the old GLM 47 Flash, and they didn't, they couldn't hang with Qwen 3.6, 35B-A3B.
13:50Not exactly a surprise. Okay, so future work, Bob Bench needs some more tasks that are diverse in terms of the language used. It focuses a lot on C, C+, Python, Bash, that kind of thing. It would be great to have tasks that have Swift or Android, iOS development, embedded, which is going to require getting a little bit creative with a test harness, but I think would be very valuable.
14:17And maybe there are configurations where offloading the KV cache system RAM can have better performance, and I didn't spend an entire run, because these runs take, you know, 5 or 10 hours plus analysis time. So I haven't tried one where we offload the KV cache's system memory.
14:36I have my doubts, but, you know, there's also other model families to try. So, having the BBW be a static set of tasks with the static evaluations and having it be private is pretty valuable because in the future, say Qwen 4 releases.
14:52We can easily run it, you know, do the full 10 hour run. Then we have this whole family of task to evaluate it against. And I see a lot of benchmarks online where they say, oh, um, you know, Opus 4.6 got a 95 on this benchmark.
15:12And it doesn't make any sense to me because Opus 4.6 isn't one thing. There are a 1000 different trade-offs on the back end. In terms of what inference framework they were actually running it on in terms of the model quant that they were using in terms of their runtime parameters, and all that gets collapsed when we look at these benchmark results.
15:34And so I struggle to find benchmark results that I enjoy online for that reason. So, suffice that to say, we'll continue adding to the BBW Alpha, which is this initial version of BBW. So, and it's the same, you know, new techniques, new inference features, more inference frameworks, I would love to run this against VLLM or SG Lang or, you know, there's a number of other backends that you can use.
16:05I've simply used llama.cpp because I like it. I like the fact that it's mostly, you know, C and Cuda. And I enjoy working with it because of its simplicity. You just compile it, you've got a binary on disk.
16:21Okay, that's it for 8 gigabyte setups. If you enjoyed this, consider commenting, join us on Discord, hop in for a live stream. Every one of these venues has a lot of high quality technical conversations going on every single day, which I'm grateful for.
16:34If you want to watch this without advertisements, the full video is on unbiasedbob.com as a torrent. Also, consider a YouTube membership, if you want to go above and beyond in your support of the channel, we just got accepted into the YouTube partner program after 2 months of being a channel.
16:51This is quite an accomplishment, I think. And so thank you for watching and I will see you in the next one.