WEBVTT 00:00:00.000 --> 00:00:02.970 Were you looking for America's next top model? 00:00:02.970 --> 00:00:04.938 Meta's Muse Spark might be for you. 00:00:04.972 --> 00:00:07.941 On this channel, we look at AI models that you can run on your own 00:00:07.975 --> 00:00:10.944 infrastructure, and typically that means the GPUs and 00:00:10.978 --> 00:00:12.946 the Apple Silicon Macs that you already have at home. 00:00:13.981 --> 00:00:14.948 Today it doesn't. 00:00:14.982 --> 00:00:18.952 We're going to look at something that was run on 2 rented B200 GPUs. 00:00:18.986 --> 00:00:22.956 Now, Meta's Spark Open is basically Lennie from Of Mice and Men. 00:00:23.991 --> 00:00:26.960 It's a giant at 842 billion parameters. 00:00:26.994 --> 00:00:29.963 It's mentally handicapped, getting about the same score as 00:00:29.997 --> 00:00:32.966 a model literally 30 times smaller than it 00:00:33.000 --> 00:00:33.967 on Bob-Bench. 00:00:34.968 --> 00:00:37.938 And if you've read the book recently, you'll know that Lennie absolutely 00:00:37.971 --> 00:00:39.940 adores small fluffy animals. 00:00:39.973 --> 00:00:41.942 And so does Spark. 00:00:41.975 --> 00:00:44.945 All right, the end of last year, we're like, all right, we are quite 00:00:44.978 --> 00:00:46.947 excited about the progress. 00:00:46.980 --> 00:00:48.949 We haven't seen the models yet. 00:00:48.982 --> 00:00:50.951 You sure this is gonna work? 00:00:51.985 --> 00:00:54.955 Now, Bob-Bench Wide is a specific comparable 00:00:55.989 --> 00:00:58.959 test set, okay, it measures a model configuration 00:00:58.992 --> 00:01:01.995 on the same 86 tasks on the same hardware, 00:01:01.995 --> 00:01:02.963 a single 4090. 00:01:03.997 --> 00:01:06.967 And let's take a look at a few specific dots on this plot. 00:01:06.967 --> 00:01:09.937 I know it's complicated, but even if you're a consumer of Meta's 00:01:10.971 --> 00:01:13.941 AI offerings, I know that you're smart enough to follow along. 00:01:13.941 --> 00:01:15.943 The highest score in Bob-Bench goes down to 00:01:16.977 --> 00:01:20.948 Qwen 3.8 Flash Next, running on a fork of Strata. 00:01:20.981 --> 00:01:24.952 On specifically, the IQ3_XXS quant. 00:01:24.985 --> 00:01:26.987 It got a 92.9% in almost 5 hours. 00:01:26.987 --> 00:01:28.956 This ran upstairs at my house. 00:01:29.990 --> 00:01:33.961 On a GPU that is relatively pricey at around $2,300 00:01:33.994 --> 00:01:36.964 and that GPU has been increasing in value 00:01:36.997 --> 00:01:38.966 exponentially since I bought it. 00:01:38.999 --> 00:01:41.969 Another spot on the frontier here goes to Qwen 3.6 00:01:41.969 --> 00:01:44.938 ThinkingCap, which is a 27-billion-parameter model coming 00:01:44.972 --> 00:01:45.939 in at just over 4 hours. 00:01:46.974 --> 00:01:49.943 It got an 89%, which is worse than Spark Open. 00:01:49.977 --> 00:01:50.944 Good job, Lennie. 00:01:50.978 --> 00:01:53.947 You beat up a model 30 times smaller than you by one point. 00:01:53.981 --> 00:01:54.948 So Spark Open. 00:01:55.983 --> 00:01:57.951 We set it up on stream today. 00:01:57.985 --> 00:02:01.955 According to the model card, it needs an incredible 700 gigabytes 00:02:01.989 --> 00:02:02.956 of VRAM to run. 00:02:02.990 --> 00:02:04.958 You could literally buy a house with that kind of hardware. 00:02:05.993 --> 00:02:08.962 Now, regardless, we rented 00:02:08.996 --> 00:02:11.965 2 B200s, and we downloaded the 00:02:11.999 --> 00:02:13.967 IQ3_XXS quant. 00:02:13.967 --> 00:02:16.937 Now, if you're good at math, you'll remember that that's the same quant that we 00:02:17.971 --> 00:02:20.941 used earlier for 3.8 Flash Next, okay? 00:02:20.974 --> 00:02:23.944 Again, 3.8 Flash Next is 00:02:23.977 --> 00:02:26.947 like 4 or 5 times smaller than Lennie. 00:02:27.981 --> 00:02:31.952 Anyways, we fired this thing up on llama.cpp with low reasoning. 00:02:31.985 --> 00:02:34.955 The entire mess fits into VRAM this way on 2 B200s. 00:02:34.988 --> 00:02:37.958 You get 128K context with 00:02:37.991 --> 00:02:39.960 a Q8 KV cache. 00:02:39.993 --> 00:02:42.963 And this was rented for less than the starting average 00:02:43.997 --> 00:02:45.966 wage at In-N-Out Burger. 00:02:45.966 --> 00:02:48.969 I paid $16 per hour on RunPod for this run. 00:02:48.969 --> 00:02:52.939 So the llama commit is here. 00:02:52.973 --> 00:02:56.943 We ran the Unsloth IQ3_XXS quant, Q8 KV 00:02:56.977 --> 00:02:59.946 cache, like I said, 128K context, sampling from the model card. 00:02:59.980 --> 00:03:01.948 No spilling the system memory. 00:03:01.982 --> 00:03:05.952 We were getting 569 tokens per second prompt processing, and 00:03:05.986 --> 00:03:07.954 41 tokens per second decode. 00:03:07.988 --> 00:03:10.957 So, again, let's compare that to Flash Next, all right? 00:03:10.991 --> 00:03:14.961 2500 prompt processing, and 93 tokens per second. 00:03:15.996 --> 00:03:18.965 So it was much faster for Flash Next. 00:03:18.999 --> 00:03:21.968 And this configuration for Lennie used a 00:03:21.968 --> 00:03:25.939 staggering 324 gigabytes of VRAM. 00:03:25.972 --> 00:03:28.942 So moral of the story is, if you 00:03:28.975 --> 00:03:31.945 want to run on tremendously 00:03:32.979 --> 00:03:36.950 expensive hardware and get the same score as Qwen 27B, 00:03:36.983 --> 00:03:39.986 this could be America's next top model for that use case. 00:03:39.986 --> 00:03:41.955 Otherwise, I'm not too interested in it. 00:03:41.988 --> 00:03:44.958 There is a couple problems with comparing it this way, and one is that the hardware 00:03:45.992 --> 00:03:48.962 needed to run Lennie is just impossible to come by. 00:03:50.997 --> 00:03:53.967 And then the other is that we ran it on light reasoning or 00:03:53.967 --> 00:03:56.937 low reasoning as opposed to medium for Qwen 3.8 00:03:56.970 --> 00:04:00.941 Flash Next, but the difference in score is 00:04:00.974 --> 00:04:01.942 phenomenal here. 00:04:01.975 --> 00:04:05.946 And because it's so expensive to test, and I can't run it at home, 00:04:05.979 --> 00:04:08.949 it's not likely for me to go back and revisit this with different 00:04:08.982 --> 00:04:11.952 sampling parameters or with different reasoning. 00:04:11.985 --> 00:04:14.955 It's just not impressing me enough to 00:04:14.988 --> 00:04:19.960 look like it's worth it, especially when it's 842 billion 00:04:19.993 --> 00:04:22.963 parameters, and, you 00:04:22.996 --> 00:04:25.966 know, hundreds of gigabytes to download a different quant. 00:04:26.967 --> 00:04:30.937 So, um, in any case, 00:04:30.937 --> 00:04:33.940 we'll settle this like men do with 00:04:33.974 --> 00:04:34.941 a skateboarding game. 00:04:34.975 --> 00:04:37.944 We'll start here with the version of the 2D skate 00:04:37.978 --> 00:04:41.948 game that was created by Qwen 3.8 Flash Next. 00:04:41.982 --> 00:04:42.949 Check it out. 00:04:42.983 --> 00:04:43.950 I'm not going to read the 00:04:43.984 --> 00:04:44.951 I'm not going to read the instructions. 00:04:45.986 --> 00:04:46.953 Okay, we'll see if that makes any difference. 00:04:46.987 --> 00:04:47.954 Here we go. 00:04:47.988 --> 00:04:48.955 Go off the ramp. 00:04:49.990 --> 00:04:51.958 It looks like we need the ollie. 00:04:56.997 --> 00:04:59.966 Okay, so it looks like the controls for 00:05:00.000 --> 00:05:01.968 ollie are. 00:05:03.970 --> 00:05:07.941 Yeah, so you have to kind of front load it with a spring, which is kind of cool. 00:05:08.975 --> 00:05:09.943 Ouch. 00:05:09.976 --> 00:05:11.945 Okay, don't hit the green box. 00:05:16.983 --> 00:05:17.951 I think I'm getting it. 00:05:17.984 --> 00:05:18.952 Can we clear this gap? 00:05:18.985 --> 00:05:22.956 No, you have to, you have to kind of race the jump. 00:05:25.992 --> 00:05:28.962 So it seems like the mechanic of this game and this version of it 00:05:28.995 --> 00:05:31.965 is all about how you ollie. 00:05:31.998 --> 00:05:33.967 So yeah, hold the charge space. 00:05:33.967 --> 00:05:37.938 You can tap left or right in midair to spin. 00:05:37.971 --> 00:05:38.939 Oh, and you can kickflip. 00:05:43.977 --> 00:05:45.946 Ooh, changes color after you kickflip. 00:05:51.985 --> 00:05:54.955 All right, we kickflip twice and got a big air with a bonus. 00:05:58.992 --> 00:06:01.962 All right, I think I'm I think I'm seeing how it works here. 00:06:01.995 --> 00:06:03.964 I'm going too fast. 00:06:03.997 --> 00:06:07.968 So if you hold If 00:06:07.968 --> 00:06:11.938 you hold D to get a bunch of speed, it's pretty difficult to get through the map. 00:06:11.972 --> 00:06:14.941 So it seems like slowing down and braking 00:06:14.975 --> 00:06:15.942 would really help. 00:06:15.976 --> 00:06:17.944 Can you just do a stationary kickflip? 00:06:19.980 --> 00:06:21.948 Slightly better at skateboarding than I am. 00:06:24.985 --> 00:06:28.955 So, I mean, this seems to be reasonably, 00:06:28.989 --> 00:06:31.958 whoa, front flip. 00:06:31.992 --> 00:06:35.962 So you'll notice that the character did a backflip, but then got points for a front flip. 00:06:35.996 --> 00:06:37.964 So Qwen 3.8 Flash Next. 00:06:37.998 --> 00:06:38.965 Should have figured that out. 00:06:40.000 --> 00:06:40.967 Let's try that again. 00:06:42.969 --> 00:06:46.940 Okay, so that was a double backflip, and then we fell into the gap. 00:06:48.975 --> 00:06:52.946 So if you do a backflip, you get points for doing a front flip. 00:06:52.979 --> 00:06:53.947 All right. 00:06:54.981 --> 00:06:55.949 That's enough. 00:06:55.982 --> 00:06:56.950 That's enough. 00:06:56.950 --> 00:06:57.951 So that's how Qwen 3.8 Flash does it. 00:06:58.985 --> 00:07:00.954 And let's see, Neon Grind. 00:07:00.987 --> 00:07:03.957 So this is implemented by Meta's Muse 00:07:03.990 --> 00:07:04.958 Spark Open. 00:07:04.991 --> 00:07:05.959 You guys think it's going to be good? 00:07:11.998 --> 00:07:13.967 Whoa, bailed. 00:07:13.967 --> 00:07:17.938 We got a 112.97-66-7876. 00:07:20.974 --> 00:07:21.942 There's our guy. 00:07:23.977 --> 00:07:26.946 Um, Oh, 00:07:26.980 --> 00:07:30.950 whoa, you can really get a lot of velocity 00:07:30.984 --> 00:07:31.952 on those turns. 00:07:31.985 --> 00:07:32.952 Holy cow. 00:07:32.986 --> 00:07:35.956 If you get into the air, you can really spin and do a front 00:07:35.989 --> 00:07:36.956 flip like 70 times. 00:07:37.991 --> 00:07:38.958 That's crazy. 00:07:38.992 --> 00:07:41.962 Dude, this is way worse than I thought it was going to be. 00:07:41.995 --> 00:07:45.966 Okay, so W and S allow you to accelerate. 00:07:45.999 --> 00:07:47.968 And then as soon as you press space to ollie. 00:07:48.968 --> 00:07:50.937 You are doing a flip. 00:07:53.974 --> 00:07:54.941 Okay, this is terrible. 00:07:56.976 --> 00:07:57.944 Holy crap. 00:07:59.980 --> 00:08:02.949 This is way worse than I thought it would do. 00:08:03.984 --> 00:08:05.952 The character has like no definition. 00:08:05.986 --> 00:08:08.955 It's just like a air hockey puck. 00:08:09.990 --> 00:08:11.958 And by the way guys, this is the first time I'm seeing this. 00:08:12.992 --> 00:08:16.963 Not at all in the same league as this game, which has some 00:08:20.000 --> 00:08:23.970 had some gameplay, had sort of a loop to it, had some mechanics, 00:08:23.970 --> 00:08:24.938 The scoring seemed pretty reasonable. 00:08:24.971 --> 00:08:27.941 The front flip was not very reasonable. 00:08:28.975 --> 00:08:30.944 So that's actually a backflip. 00:08:32.979 --> 00:08:36.950 But, you know, I can't really complain too much. 00:08:36.950 --> 00:08:39.953 And also, the control scheme is just a lot more a 00:08:39.986 --> 00:08:42.956 lot more sane and comfortable here in terms of using 00:08:43.990 --> 00:08:47.961 A and D to go left and right as opposed to W and S. 00:08:47.994 --> 00:08:50.964 And we'll decide who the real winner is based on 00:08:50.997 --> 00:08:53.967 the results of these 2D skateboarding games. 00:08:53.967 --> 00:08:56.936 In conclusion, I'm not going to spend any more time on this model 00:08:56.970 --> 00:08:58.938 until I get 2 B200s at home. 00:08:58.972 --> 00:09:00.940 What I've seen isn't promising. 00:09:01.975 --> 00:09:05.945 It's possible that in higher reasoning modes, it could act differently, but it's 00:09:05.979 --> 00:09:06.946 not a very verbose model. 00:09:06.980 --> 00:09:09.949 It gives up really early, even when it's successful 00:09:09.983 --> 00:09:11.951 and completes the task fast. 00:09:12.986 --> 00:09:14.954 It does it with very few tokens used. 00:09:14.988 --> 00:09:17.991 So even though it ran slowly on these 2 behemoth GPUs. 00:09:17.991 --> 00:09:18.958 Not that interesting. 00:09:18.992 --> 00:09:20.960 All right, guys, thank you for listening. 00:09:21.995 --> 00:09:24.964 Appreciate you watching the short video on Lennie, 00:09:24.998 --> 00:09:25.965 and I will see you in the next one.