WEBVTT 00:00:00.000 --> 00:00:02.970 The best model you can run at home naturally depends on the hardware that you have. 00:00:02.970 --> 00:00:05.939 In this video, we're going to be talking about the 8 00:00:05.973 --> 00:00:08.942 gigabyte VRAM class of GPUs with 00:00:08.976 --> 00:00:11.945 alongside it, about 12 gigabytes of 00:00:11.979 --> 00:00:13.947 system memory available for inference. 00:00:14.681 --> 00:00:16.650 So first, the configuration I recommend. 00:00:16.650 --> 00:00:19.620 It outperformed a dozen others on my benchmark. 00:00:19.653 --> 00:00:23.624 Get yourself the Qwen3.6 35B-A3B. 00:00:23.657 --> 00:00:26.627 We're looking at the IQ3_XXS Model Quant. 00:00:27.661 --> 00:00:30.631 Download the mmproj Vision Tower if you want, 00:00:30.664 --> 00:00:32.633 and that'll be resident in main memory. 00:00:33.667 --> 00:00:37.638 Use the Q4, KV cache at 128K context, 00:00:37.671 --> 00:00:41.642 and we'll be using the sampling parameters from the model card. 00:00:41.675 --> 00:00:44.645 This model's performance degrades gracefully as you 00:00:44.678 --> 00:00:46.647 offload weights to system memory. 00:00:46.780 --> 00:00:49.750 The vast majority of the weights in the 35B 00:00:49.750 --> 00:00:53.720 model are experts that are only loaded at about a 3% rate. 00:00:53.754 --> 00:00:56.723 So they can be offloaded to system memory with a pretty small 00:00:56.757 --> 00:00:58.725 but noticeable loss in performance. 00:00:59.726 --> 00:01:02.696 In llama.cpp, --n-cpu-moe 00:01:03.697 --> 00:01:06.700 is the flag that allows us to run a model that's 30 gigabytes 00:01:06.700 --> 00:01:09.670 or more in full precision on only an 8 gigabyte card. 00:01:10.537 --> 00:01:12.506 So the configuration is on screen. 00:01:12.506 --> 00:01:14.474 It's also in the YouTube video description. 00:01:14.508 --> 00:01:15.475 I've got it on GitHub. 00:01:15.509 --> 00:01:17.477 I would encourage you to try this out. 00:01:17.511 --> 00:01:20.480 If you start downloading the model now, you can probably use it 00:01:20.514 --> 00:01:23.483 later in the video, if you're on a fast internet connection. 00:01:23.517 --> 00:01:25.485 It's fast enough to be interactive. 00:01:25.519 --> 00:01:27.487 It's a great coding assistant. 00:01:27.521 --> 00:01:30.490 So far, it's the best that I've found under these hardware 00:01:30.524 --> 00:01:34.494 budget constraints of 8 gigabytes of VRAM and 12 gigabytes of system RAM. 00:01:35.295 --> 00:01:37.264 So, what can you expect in this video? 00:01:37.264 --> 00:01:39.232 We're going to talk about the hardware. 00:01:39.266 --> 00:01:40.233 This stuff is incredibly cheap. 00:01:40.267 --> 00:01:41.234 You can run this on anything. 00:01:41.268 --> 00:01:45.238 It's basically the potato camera equivalent of GPU 00:01:45.272 --> 00:01:46.239 LLM setups. 00:01:47.274 --> 00:01:50.243 75 bucks to $200 on eBay 00:01:50.277 --> 00:01:51.244 shipped within the US. 00:01:52.279 --> 00:01:55.248 What can you expect in terms of speed and intelligence? 00:01:55.282 --> 00:01:58.251 We're going to look at some of the difficult problems that this model 00:01:58.285 --> 00:02:01.254 succeeds at, and then over the cusp, what 00:02:01.288 --> 00:02:04.257 it fails at, and we're going to compare it in aggregate 00:02:04.291 --> 00:02:08.261 against a bunch of other models I've tested on similar local hardware. 00:02:10.263 --> 00:02:13.233 Third, testing methodology. 00:02:13.266 --> 00:02:16.236 Disconfirming evidence, how I evaluate a 00:02:16.269 --> 00:02:19.239 model and all the configurations that we've looked at. 00:02:19.272 --> 00:02:22.242 And then finally, we'll wrap up with some future work. 00:02:22.275 --> 00:02:25.245 What I want to improve about the testing framework that I have so 00:02:25.278 --> 00:02:28.248 far, and what I don't know yet about the performance 00:02:28.281 --> 00:02:29.249 of models like this. 00:02:30.517 --> 00:02:32.486 Okay, a little bit more about hardware configurations. 00:02:32.486 --> 00:02:35.455 We're talking about a server or a PC with 00:02:36.490 --> 00:02:39.059 16 gigabytes minimum of system RAM. 00:02:39.092 --> 00:02:42.062 You can be running Linux or Windows if you're a savage. 00:02:42.062 --> 00:02:45.032 The GPUs that we're targeting today, like I said, are very cheap. 00:02:45.065 --> 00:02:49.036 The 1080, I saw one for about 80 bucks on eBay. 00:02:49.069 --> 00:02:52.039 The 2080 goes for a little bit more, but not too much more. 00:02:52.472 --> 00:02:53.440 At about $100. 00:02:54.041 --> 00:02:57.010 And then there's the 3050, uh, which is coming 00:02:57.010 --> 00:02:59.980 in under $200, but be careful because some of the 3050s 00:03:00.013 --> 00:03:00.981 are 6 gigabyte cards. 00:03:01.982 --> 00:03:05.952 So for a lot of people, I think you already have one of these sitting around at home. 00:03:05.952 --> 00:03:09.923 And if we look at the steam hardware survey data, 00:03:09.956 --> 00:03:12.926 basically anybody that's buying and playing games on steam. 00:03:13.093 --> 00:03:17.064 And responding to their survey, already has hardware, 00:03:17.064 --> 00:03:18.031 like what we're talking about today. 00:03:18.065 --> 00:03:20.667 This is a very accessible tier. 00:03:20.700 --> 00:03:23.670 So I'm lucky enough to have better hardware at home. 00:03:23.670 --> 00:03:26.640 I have a 4090 and a 3090 that I program on, and 00:03:26.673 --> 00:03:29.643 luckily the prices of those have gone up since I bought them, 00:03:29.676 --> 00:03:31.645 so I could probably sell them for a profit now. 00:03:31.978 --> 00:03:34.948 But I had not tried the 00:03:34.948 --> 00:03:38.919 configuration that I am testing today, and I wanted to correct that. 00:03:38.952 --> 00:03:41.922 So last week on stream live on YouTube, 00:03:41.955 --> 00:03:45.926 we pulled out the 3090 from my infant server. 00:03:46.960 --> 00:03:51.364 We stuck a 1070 Ti in there, literally $70 card. 00:03:51.398 --> 00:03:54.367 And um, the the card had been sitting around 00:03:55.368 --> 00:03:58.338 at my house for like 8 years. 00:03:58.338 --> 00:04:01.341 It's not modern stuff, and we fired it up to do some programming. 00:04:01.908 --> 00:04:04.878 So we were getting about 400 tokens per 2nd 00:04:04.878 --> 00:04:07.848 prompts processing, which does feel a little bit slow, and 00:04:07.881 --> 00:04:10.851 we were using about 10 gigabytes of system RAM. 00:04:10.884 --> 00:04:12.853 And on stream. 00:04:12.886 --> 00:04:15.856 We were getting 25 tokens per second, which for 00:04:15.889 --> 00:04:18.859 me, that's pretty close to the limit where I'm like, okay, 00:04:18.892 --> 00:04:20.861 this is usable and interactive. 00:04:21.895 --> 00:04:24.865 And it doesn't feel as fast as if you're using Claude for sure. 00:04:25.899 --> 00:04:30.871 And it doesn't, you know, I'm used to a much faster tokens per 2nd output. 00:04:30.904 --> 00:04:34.875 So it's not optimal, but we're talking about budget hardware here. 00:04:34.908 --> 00:04:36.877 So we have to keep our expectations in check. 00:04:38.345 --> 00:04:41.314 The main issue, I think, that we had with this setup on 00:04:41.314 --> 00:04:44.284 stream was where the model was doing work. 00:04:44.317 --> 00:04:47.287 testing its own work, with taking screenshots 00:04:47.287 --> 00:04:49.256 and then reading them and inspecting them. 00:04:49.289 --> 00:04:52.259 And this took a really long time to read each image. 00:04:52.592 --> 00:04:55.562 It's important to me if I'm going to make a recommendation to actually try 00:04:55.562 --> 00:04:58.532 it, and so I made sure that we did that live to kind of experience 00:04:58.565 --> 00:05:01.535 what this thing is like to actually use. 00:05:02.569 --> 00:05:05.572 And I'm not going to say I was impressed with the results, but 00:05:05.572 --> 00:05:07.541 I will say that I expected worse. 00:05:08.008 --> 00:05:10.977 Obviously, it's better to have a $1000 00:05:10.977 --> 00:05:13.947 GPU than a $70 GPU. 00:05:13.980 --> 00:05:14.948 This isn't rocket science. 00:05:15.816 --> 00:05:17.784 Okay, so what about the top end of the intelligence? 00:05:17.784 --> 00:05:18.752 What can you expect from this? 00:05:19.786 --> 00:05:22.756 One of the hardest task ladders in my private task set, 00:05:22.789 --> 00:05:25.759 Bob Bench, involves letting an agent loose on a docker image 00:05:25.792 --> 00:05:29.763 where there's a working, running, ComfyUI service. 00:05:30.330 --> 00:05:34.301 The agents are instructed to do various tasks that involve querying 00:05:34.301 --> 00:05:37.270 the running system, to see what servers are running, reading the 00:05:37.304 --> 00:05:39.272 source code to determine what functionality exists. 00:05:40.307 --> 00:05:43.276 You have to probe the API and make sure that you 00:05:43.310 --> 00:05:47.113 can automate your request to the API, and then finally... 00:05:47.147 --> 00:05:51.117 You know, agents have to verify their work through the file system. 00:05:51.117 --> 00:05:54.087 So this is a very agentic kind of situation. 00:05:54.487 --> 00:05:57.457 For this particular task that I'm going to go over, the agent is instructed 00:05:57.457 --> 00:06:00.427 to run a sweep over a bunch of different parameters, hitting 00:06:01.461 --> 00:06:03.430 the ComfyUI, and it's not given. 00:06:03.597 --> 00:06:05.565 A lot of guardrails or instructions about how to do it. 00:06:05.565 --> 00:06:08.535 This is more of a, please accomplish this for me, and then you're 00:06:08.568 --> 00:06:11.538 free to figure out how it works, rather than a, here 00:06:12.572 --> 00:06:15.542 is a spec, and I want the agent and the model to go off and 00:06:15.575 --> 00:06:16.543 implement some. 00:06:16.776 --> 00:06:18.712 Some thing against the spec. 00:06:18.712 --> 00:06:21.681 So on this particular task, there's a 900-second timeout, 00:06:21.681 --> 00:06:25.652 and the 35B A3B is running slowly, so time is a concern. 00:06:25.685 --> 00:06:27.654 It finished in about 750 seconds. 00:06:28.154 --> 00:06:30.123 It made about 70 tool calls. 00:06:30.123 --> 00:06:34.094 And over the course of 64 turns, it 00:06:34.094 --> 00:06:35.061 did a great job on this task. 00:06:35.095 --> 00:06:38.064 Now, it did fuck some shit up along the way. 00:06:38.098 --> 00:06:39.065 And you know what? 00:06:39.099 --> 00:06:40.066 That's kind of expected. 00:06:40.100 --> 00:06:43.069 It detected that there was a problem 00:06:43.103 --> 00:06:46.072 and it adjusted its course, 00:06:46.106 --> 00:06:49.075 all excellent behavior that 00:06:49.109 --> 00:06:52.078 you want in an agent to be able to determine that there was an 00:06:53.113 --> 00:06:57.083 issue and that it has to undo something, roll back a change. 00:06:57.117 --> 00:07:00.086 And you see this with much more intelligent agents like Claude Opus, too. 00:07:00.120 --> 00:07:03.089 So difficult task, lots of ambiguity, 00:07:04.090 --> 00:07:07.060 completed in under 900 seconds, can't ask for more 00:07:07.093 --> 00:07:09.062 in terms of a budget configuration. 00:07:09.296 --> 00:07:10.263 So I'll play a little bit of the... 00:07:10.363 --> 00:07:13.333 the footage from the live stream, from the unbiased 00:07:13.333 --> 00:07:17.304 Bob channel, where we installed the 1070 Ti, and we did programming with this thing. 00:07:18.338 --> 00:07:22.309 We created the interactive solar system visualization with 00:07:22.342 --> 00:07:25.312 a prompt directly YOLO from a subscriber watching on 00:07:25.345 --> 00:07:26.313 in the chat. 00:07:26.379 --> 00:07:29.349 Yeah, and I also want to stress the insightful and intelligent 00:07:29.349 --> 00:07:31.318 commentary that I gave. 00:07:31.885 --> 00:07:33.853 As we were doing this live. 00:07:35.155 --> 00:07:37.123 We need sunspots. 00:07:37.157 --> 00:07:38.124 At sunspots. 00:07:40.427 --> 00:07:43.396 And I'm pretty sure that this 00:07:43.430 --> 00:07:48.268 is nowhere near to scale. 00:07:48.301 --> 00:07:49.269 There's the sun spots. 00:07:50.270 --> 00:07:53.239 This reminds me of those egg bites at Starbucks. 00:07:53.273 --> 00:07:54.240 It really has that vibe. 00:07:55.775 --> 00:07:58.745 Okay, so that's specific tasks, and a couple, 00:07:58.778 --> 00:08:02.749 you know, basically cherry picked examples of where the model does a good job. 00:08:02.782 --> 00:08:05.752 But in the aggregate, let's talk a little bit 00:08:05.785 --> 00:08:08.755 more about Bob Bench Wide, which is my 86 00:08:08.788 --> 00:08:11.758 task, held out private, not in the training 00:08:11.791 --> 00:08:14.761 data task set, and we'll talk 00:08:14.794 --> 00:08:17.764 about how this 00:08:17.797 --> 00:08:19.766 model fits in broadly. 00:08:19.799 --> 00:08:22.769 So each point on Bob Bench wide is a 00:08:22.802 --> 00:08:25.772 given model and a given configuration and its score versus 00:08:25.805 --> 00:08:27.774 how long it took to get there. 00:08:28.775 --> 00:08:32.746 So models like Qwen3.8 27B, 00:08:32.779 --> 00:08:35.749 where they change the reasoning and the model thinks forever, they 00:08:35.782 --> 00:08:38.752 can get a lower score, uh, in terms 00:08:39.786 --> 00:08:42.756 of time, but then a very high score in terms of 00:08:42.789 --> 00:08:45.759 total accuracy if you let them cook forever. 00:08:45.792 --> 00:08:48.762 My benchmark set penalize 00:08:48.795 --> 00:08:50.764 his models for thinking for a long time. 00:08:50.797 --> 00:08:53.767 And the tasks have timeouts of 00:08:53.800 --> 00:08:56.770 usually 15 or 30 minutes per task, depending on the difficulty. 00:08:57.771 --> 00:09:00.740 Each point on the bob bench wide chart is a benchmark 00:09:00.740 --> 00:09:02.709 run on a given model and a given configuration. 00:09:03.476 --> 00:09:06.446 The top left quadrant are fast and smart configurations, 00:09:06.446 --> 00:09:10.417 Qwen 3.6 completes the entire benchmark, in 00:09:10.450 --> 00:09:12.419 less than 5 hours and gets a 90%. 00:09:13.453 --> 00:09:17.424 But that's a higher model quant with more memory and with MTP enabled. 00:09:17.457 --> 00:09:20.427 So the red lines are your trade-offs with 00:09:20.460 --> 00:09:24.364 24 gigabytes of memory available to you and a fast graphics card. 00:09:24.397 --> 00:09:26.066 It's not exhaustive, but this is. 00:09:26.066 --> 00:09:29.035 a parado curve for 24 gigaway cards. 00:09:30.036 --> 00:09:33.006 So, the X axis here measures how it retarded the model is, 00:09:33.039 --> 00:09:36.009 if you're way out here to the right, say, like IBM 00:09:36.042 --> 00:09:39.012 granite, it means you took forever to execute, and 00:09:40.046 --> 00:09:41.014 the model was super slow. 00:09:41.047 --> 00:09:44.017 And as we go down, that's the other retarded axis 00:09:44.050 --> 00:09:47.020 where the models get a lot dumber, like if you're IBM 00:09:47.053 --> 00:09:50.023 granite, way down here at the bottom, you got a very low score. 00:09:51.057 --> 00:09:55.028 So that's a little bit about exploring the curves on the BBW, 00:09:55.028 --> 00:09:55.995 something I'm a really big fan of. 00:09:56.496 --> 00:09:57.464 Methodology. 00:09:57.464 --> 00:10:01.434 Now, I'm not a scientist, but I apply techniques from science to this problem. 00:10:01.468 --> 00:10:05.438 So, first, get yourself that falsifiable hypothesis, 00:10:05.472 --> 00:10:08.441 something specific and actionable and something that you 00:10:08.475 --> 00:10:09.442 can disprove, okay? 00:10:09.476 --> 00:10:13.446 And then for small models that fit on consumer hardware, 00:10:13.480 --> 00:10:15.448 nothing beats Qwen so far. 00:10:15.482 --> 00:10:17.450 So Qwen's going to be our baseline. 00:10:17.484 --> 00:10:20.487 So we're going to say, look, Qwen, 3.6, 00:10:20.487 --> 00:10:23.456 35B-A3B, is the best model 00:10:23.490 --> 00:10:27.460 We're gonna do a lot of offloading the CPU on the 8 gigabyte card. 00:10:28.495 --> 00:10:31.464 And look, this is a very defensible 00:10:31.464 --> 00:10:32.432 baseline position. 00:10:34.467 --> 00:10:38.438 And then we're going to go off and we're going to look for disconfirming evidence, okay? 00:10:38.471 --> 00:10:40.440 I don't think scientists do that today. 00:10:40.473 --> 00:10:43.443 I think they're more into publishing papers that they wrote with chat GPT, 00:10:43.476 --> 00:10:46.446 but what you do is you go and you 00:10:46.479 --> 00:10:50.450 say, well, if this wasn't true, 00:10:50.483 --> 00:10:52.452 you know, how could I, how could I prove it? 00:10:52.485 --> 00:10:55.455 And so we can get creative by looking at different model 00:10:55.488 --> 00:10:58.458 families, different quantizations, 00:10:58.491 --> 00:11:02.462 either extreme high or extreme low end of quantizations, we have both of those here. 00:11:03.463 --> 00:11:07.433 And so this is how I go about trying to figure 00:11:07.467 --> 00:11:08.434 out the best configuration. 00:11:09.168 --> 00:11:12.138 In my experience, in the workplace and at 00:11:12.138 --> 00:11:15.108 home, decisions are rarely made with a 00:11:15.141 --> 00:11:16.109 framework like this. 00:11:17.143 --> 00:11:20.113 Now with open weights models and open source inference, 00:11:20.146 --> 00:11:23.116 we have the freedom to methodically 00:11:23.149 --> 00:11:25.118 and methodologically approach this. 00:11:26.152 --> 00:11:28.121 Is methodologically a word? 00:11:28.154 --> 00:11:32.125 don't actually know, but we can approach it diligently and methodically. 00:11:32.158 --> 00:11:33.126 and methodologically. 00:11:35.161 --> 00:11:38.164 So, I tested over six model families and a dozen configurations, 00:11:38.164 --> 00:11:41.134 all the while knowing that Qwen 3.6, 00:11:41.167 --> 00:11:45.138 35B-A3B was probably going to be the best, and it was. 00:11:46.139 --> 00:11:49.108 So each test, each one of these blue dots was 00:11:49.142 --> 00:11:52.111 run on the same hardware, the same harness, the same version 00:11:52.145 --> 00:11:55.114 of llama.cpp, the same task timeouts, 00:11:55.148 --> 00:11:57.116 the same system prompts, you get the picture. 00:11:57.150 --> 00:11:59.118 We hold everything constant that we can. 00:12:00.153 --> 00:12:03.122 And then we switch out the underlying model, and 00:12:04.157 --> 00:12:07.126 we adjust the amount of KV cache and the context 00:12:07.160 --> 00:12:11.130 size that's available, subject to that 8 gigabyte VRAM cap. 00:12:11.197 --> 00:12:15.168 So I think the true minimum on KV cache is probably like 64K 00:12:15.168 --> 00:12:17.136 or maybe 48K. 00:12:17.170 --> 00:12:20.139 Uh, but a couple of the configs with smaller 00:12:20.173 --> 00:12:22.141 sizes were tried here, where required. 00:12:22.976 --> 00:12:25.945 So honorable mentions here are reap. 00:12:25.945 --> 00:12:28.915 It really surprised me to try a modification 00:12:28.948 --> 00:12:30.917 to Qwen that did pretty well. 00:12:31.951 --> 00:12:34.954 And as we can see, reap is, 00:12:34.954 --> 00:12:37.924 it's a model where they, they prune the experts 00:12:38.958 --> 00:12:40.393 that get used least frequently. 00:12:40.426 --> 00:12:43.396 But anyways, the reap guys did a really good job of making a 00:12:43.396 --> 00:12:46.366 slightly smaller model that was a little bit dumber and took 00:12:46.399 --> 00:12:49.369 a little bit longer, but overall, it might be within noise and 00:12:50.403 --> 00:12:53.373 it might be just about as good as Qwen 3.6. 00:12:53.406 --> 00:12:56.376 But I don't see a reason to choose it over the 00:12:56.409 --> 00:12:58.278 Qwen 3.6 base. 00:12:58.311 --> 00:13:02.282 Uh, granite, IBM granite was absolutely horrible. 00:13:02.282 --> 00:13:05.251 Bonsai has a couple points on the graph that are sort 00:13:05.284 --> 00:13:07.253 of okay. 00:13:07.286 --> 00:13:10.256 Gemma 4 did not perform well 00:13:10.290 --> 00:13:11.257 in this setup. 00:13:11.290 --> 00:13:15.261 Maybe it deserves being tried with a 00:13:15.294 --> 00:13:18.264 lower model quant, but I doubt that with this 00:13:18.298 --> 00:13:20.266 initial score getting a 55%. 00:13:21.300 --> 00:13:23.269 I doubt that Gemma is a contender. 00:13:24.304 --> 00:13:28.274 And then the dense configurations between Qwen 3.8, 00:13:28.308 --> 00:13:31.277 27B and Bonsai just didn't 00:13:31.310 --> 00:13:34.180 have the chops to complete the task. 00:13:34.213 --> 00:13:37.183 Another one, I tried a couple different setups that had other 00:13:37.183 --> 00:13:40.153 a mixture of experts models in the 30 B class, 00:13:40.186 --> 00:13:43.156 Gemma 4 and the old GLM 47 00:13:44.190 --> 00:13:47.160 Flash, and they didn't, they couldn't hang with 00:13:47.193 --> 00:13:50.163 Qwen 3.6, 35B-A3B. 00:13:50.196 --> 00:13:51.164 Not exactly a surprise. 00:13:52.065 --> 00:13:55.034 Okay, so future work, Bob Bench 00:13:55.034 --> 00:13:58.004 needs some more tasks that are diverse in terms 00:13:58.037 --> 00:14:00.006 of the language used. 00:14:00.039 --> 00:14:04.010 It focuses a lot on C, C+, 00:14:04.043 --> 00:14:06.012 Python, Bash, that kind of thing. 00:14:07.046 --> 00:14:10.016 It would be great to have tasks that have Swift or 00:14:10.049 --> 00:14:13.019 Android, iOS development, embedded, which 00:14:13.052 --> 00:14:17.023 is going to require getting a little bit creative with a test harness, but I think would be very valuable. 00:14:17.457 --> 00:14:20.426 And maybe there are configurations where offloading 00:14:20.426 --> 00:14:23.396 the KV cache system RAM can have better performance, and 00:14:23.429 --> 00:14:26.399 I didn't spend an entire run, because these 00:14:26.432 --> 00:14:30.403 runs take, you know, 5 or 10 hours plus analysis time. 00:14:31.437 --> 00:14:34.407 So I haven't tried one where we offload 00:14:34.440 --> 00:14:36.409 the KV cache's system memory. 00:14:36.442 --> 00:14:39.412 I have my doubts, but, you know, 00:14:39.445 --> 00:14:41.414 there's also other model families to try. 00:14:41.981 --> 00:14:44.951 So, having the BBW be a static set 00:14:44.951 --> 00:14:47.920 of tasks with the static evaluations and having it be 00:14:48.955 --> 00:14:52.925 private is pretty valuable because in the future, say Qwen 4 releases. 00:14:52.959 --> 00:14:56.929 We can easily run it, you know, do the full 10 hour run. 00:14:56.963 --> 00:14:59.932 Then we have this whole family of task to 00:14:59.966 --> 00:15:00.933 evaluate it against. 00:15:01.968 --> 00:15:04.937 And I see a lot of benchmarks online where they say, 00:15:04.971 --> 00:15:08.941 oh, um, you know, Opus 4.6 00:15:08.975 --> 00:15:11.944 got a 95 on this benchmark. 00:15:12.979 --> 00:15:16.949 And it doesn't make any sense to me because Opus 4.6 isn't one thing. 00:15:16.949 --> 00:15:19.919 There are a 1000 different trade-offs on the back end. 00:15:20.953 --> 00:15:23.923 In terms of what inference framework 00:15:23.956 --> 00:15:26.926 they were actually running it on in terms of the model quant that 00:15:26.959 --> 00:15:29.929 they were using in terms of their runtime parameters, and all 00:15:29.962 --> 00:15:32.932 that gets collapsed when we look at these benchmark results. 00:15:34.967 --> 00:15:38.938 And so I struggle to find 00:15:38.971 --> 00:15:41.941 benchmark results that I enjoy online for that reason. 00:15:41.974 --> 00:15:44.944 So, suffice that to say, 00:15:44.977 --> 00:15:47.947 we'll continue adding to the BBW Alpha, 00:15:48.948 --> 00:15:51.918 which is this initial version of BBW. 00:15:51.951 --> 00:15:54.921 So, and it's the same, you know, new techniques, new 00:15:54.954 --> 00:15:57.924 inference features, more inference frameworks, 00:15:57.957 --> 00:16:00.927 I would love to run this against VLLM 00:16:00.960 --> 00:16:04.931 or SG Lang or, you know, there's a number of other backends that you can use. 00:16:05.965 --> 00:16:08.935 I've simply used llama.cpp because I like it. 00:16:08.968 --> 00:16:11.938 I like the fact that it's mostly, you know, C and Cuda. 00:16:13.973 --> 00:16:16.943 And I enjoy working with it because of its simplicity. 00:16:16.976 --> 00:16:19.946 You just compile it, you've got a binary on disk. 00:16:21.280 --> 00:16:23.249 Okay, that's it for 8 gigabyte setups. 00:16:23.282 --> 00:16:26.252 If you enjoyed this, consider commenting, join us on Discord, 00:16:26.285 --> 00:16:28.254 hop in for a live stream. 00:16:28.287 --> 00:16:31.257 Every one of these venues has a lot of high quality technical conversations 00:16:31.290 --> 00:16:34.260 going on every single day, which I'm grateful for. 00:16:34.293 --> 00:16:37.263 If you want to watch this without advertisements, the full video is 00:16:38.297 --> 00:16:40.266 on unbiasedbob.com as a torrent. 00:16:40.299 --> 00:16:43.269 Also, consider a YouTube membership, if you 00:16:43.302 --> 00:16:46.272 want to go above and beyond in your support of the channel, we 00:16:46.305 --> 00:16:49.275 just got accepted into the YouTube partner 00:16:49.308 --> 00:16:51.277 program after 2 months of being a channel. 00:16:51.310 --> 00:16:52.278 This is quite an accomplishment, I think. 00:16:53.312 --> 00:16:56.282 And so thank you for watching and I will see you in the next one.