0:00  Were you looking for America's next top model?
0:02  Meta's Muse Spark might be for you.
0:04  On this channel, we look at AI models that you can run on your own
0:07  infrastructure, and typically that means the GPUs and
0:10  the Apple Silicon Macs that you already have at home.
0:13  Today it doesn't.
0:14  We're going to look at something that was run on 2 rented B200 GPUs.
0:18  Now, Meta's Spark Open is basically Lennie from Of Mice and Men.
0:23  It's a giant at 842 billion parameters.
0:26  It's mentally handicapped, getting about the same score as
0:29  a model literally 30 times smaller than it
0:32  on Bob-Bench.
0:34  And if you've read the book recently, you'll know that Lennie absolutely
0:37  adores small fluffy animals.
0:39  And so does Spark.
0:41  All right, the end of last year, we're like, all right, we are quite
0:44  excited about the progress.
0:46  We haven't seen the models yet.
0:48  You sure this is gonna work?
0:51  Now, Bob-Bench Wide is a specific comparable
0:55  test set, okay, it measures a model configuration
0:58  on the same 86 tasks on the same hardware,
1:01  a single 4090.
1:03  And let's take a look at a few specific dots on this plot.
1:06  I know it's complicated, but even if you're a consumer of Meta's
1:10  AI offerings, I know that you're smart enough to follow along.
1:13  The highest score in Bob-Bench goes down to
1:16  Qwen 3.8 Flash Next, running on a fork of Strata.
1:20  On specifically, the IQ3_XXS quant.
1:24  It got a 92.9% in almost 5 hours.
1:26  This ran upstairs at my house.
1:29  On a GPU that is relatively pricey at around $2,300
1:33  and that GPU has been increasing in value
1:36  exponentially since I bought it.
1:38  Another spot on the frontier here goes to Qwen 3.6
1:41  ThinkingCap, which is a 27-billion-parameter model coming
1:44  in at just over 4 hours.
1:46  It got an 89%, which is worse than Spark Open.
1:49  Good job, Lennie.
1:50  You beat up a model 30 times smaller than you by one point.
1:53  So Spark Open.
1:55  We set it up on stream today.
1:57  According to the model card, it needs an incredible 700 gigabytes
2:01  of VRAM to run.
2:02  You could literally buy a house with that kind of hardware.
2:05  Now, regardless, we rented
2:08  2 B200s, and we downloaded the
2:11  IQ3_XXS quant.
2:13  Now, if you're good at math, you'll remember that that's the same quant that we
2:17  used earlier for 3.8 Flash Next, okay?
2:20  Again, 3.8 Flash Next is
2:23  like 4 or 5 times smaller than Lennie.
2:27  Anyways, we fired this thing up on llama.cpp with low reasoning.
2:31  The entire mess fits into VRAM this way on 2 B200s.
2:34  You get 128K context with
2:37  a Q8 KV cache.
2:39  And this was rented for less than the starting average
2:43  wage at In-N-Out Burger.
2:45  I paid $16 per hour on RunPod for this run.
2:48  So the llama commit is here.
2:52  We ran the Unsloth IQ3_XXS quant, Q8 KV
2:56  cache, like I said, 128K context, sampling from the model card.
2:59  No spilling the system memory.
3:01  We were getting 569 tokens per second prompt processing, and
3:05  41 tokens per second decode.
3:07  So, again, let's compare that to Flash Next, all right?
3:10  2500 prompt processing, and 93 tokens per second.
3:15  So it was much faster for Flash Next.
3:18  And this configuration for Lennie used a
3:21  staggering 324 gigabytes of VRAM.
3:25  So moral of the story is, if you
3:28  want to run on tremendously
3:32  expensive hardware and get the same score as Qwen 27B,
3:36  this could be America's next top model for that use case.
3:39  Otherwise, I'm not too interested in it.
3:41  There is a couple problems with comparing it this way, and one is that the hardware
3:45  needed to run Lennie is just impossible to come by.
3:50  And then the other is that we ran it on light reasoning or
3:53  low reasoning as opposed to medium for Qwen 3.8
3:56  Flash Next, but the difference in score is
4:00  phenomenal here.
4:01  And because it's so expensive to test, and I can't run it at home,
4:05  it's not likely for me to go back and revisit this with different
4:08  sampling parameters or with different reasoning.
4:11  It's just not impressing me enough to
4:14  look like it's worth it, especially when it's 842 billion
4:19  parameters, and, you
4:22  know, hundreds of gigabytes to download a different quant.
4:26  So, um, in any case,
4:30  we'll settle this like men do with
4:33  a skateboarding game.
4:34  We'll start here with the version of the 2D skate
4:37  game that was created by Qwen 3.8 Flash Next.
4:41  Check it out.
4:42  I'm not going to read the
4:43  I'm not going to read the instructions.
4:45  Okay, we'll see if that makes any difference.
4:46  Here we go.
4:47  Go off the ramp.
4:49  It looks like we need the ollie.
4:56  Okay, so it looks like the controls for
4:59  ollie are.
5:03  Yeah, so you have to kind of front load it with a spring, which is kind of cool.
5:08  Ouch.
5:09  Okay, don't hit the green box.
5:16  I think I'm getting it.
5:17  Can we clear this gap?
5:18  No, you have to, you have to kind of race the jump.
5:25  So it seems like the mechanic of this game and this version of it
5:28  is all about how you ollie.
5:31  So yeah, hold the charge space.
5:33  You can tap left or right in midair to spin.
5:37  Oh, and you can kickflip.
5:43  Ooh, changes color after you kickflip.
5:51  All right, we kickflip twice and got a big air with a bonus.
5:58  All right, I think I'm I think I'm seeing how it works here.
6:01  I'm going too fast.
6:03  So if you hold If
6:07  you hold D to get a bunch of speed, it's pretty difficult to get through the map.
6:11  So it seems like slowing down and braking
6:14  would really help.
6:15  Can you just do a stationary kickflip?
6:19  Slightly better at skateboarding than I am.
6:24  So, I mean, this seems to be reasonably,
6:28  whoa, front flip.
6:31  So you'll notice that the character did a backflip, but then got points for a front flip.
6:35  So Qwen 3.8 Flash Next.
6:37  Should have figured that out.
6:39  Let's try that again.
6:42  Okay, so that was a double backflip, and then we fell into the gap.
6:48  So if you do a backflip, you get points for doing a front flip.
6:52  All right.
6:54  That's enough.
6:55  That's enough.
6:56  So that's how Qwen 3.8 Flash does it.
6:58  And let's see, Neon Grind.
7:00  So this is implemented by Meta's Muse
7:03  Spark Open.
7:04  You guys think it's going to be good?
7:11  Whoa, bailed.
7:13  We got a 112.97-66-7876.
7:20  There's our guy.
7:23  Um, Oh,
7:26  whoa, you can really get a lot of velocity
7:30  on those turns.
7:31  Holy cow.
7:32  If you get into the air, you can really spin and do a front
7:35  flip like 70 times.
7:37  That's crazy.
7:38  Dude, this is way worse than I thought it was going to be.
7:41  Okay, so W and S allow you to accelerate.
7:45  And then as soon as you press space to ollie.
7:48  You are doing a flip.
7:53  Okay, this is terrible.
7:56  Holy crap.
7:59  This is way worse than I thought it would do.
8:03  The character has like no definition.
8:05  It's just like a air hockey puck.
8:09  And by the way guys, this is the first time I'm seeing this.
8:12  Not at all in the same league as this game, which has some
8:19  had some gameplay, had sort of a loop to it, had some mechanics,
8:23  The scoring seemed pretty reasonable.
8:24  The front flip was not very reasonable.
8:28  So that's actually a backflip.
8:32  But, you know, I can't really complain too much.
8:36  And also, the control scheme is just a lot more a
8:39  lot more sane and comfortable here in terms of using
8:43  A and D to go left and right as opposed to W and S.
8:47  And we'll decide who the real winner is based on
8:50  the results of these 2D skateboarding games.
8:53  In conclusion, I'm not going to spend any more time on this model
8:56  until I get 2 B200s at home.
8:58  What I've seen isn't promising.
9:01  It's possible that in higher reasoning modes, it could act differently, but it's
9:05  not a very verbose model.
9:06  It gives up really early, even when it's successful
9:09  and completes the task fast.
9:12  It does it with very few tokens used.
9:14  So even though it ran slowly on these 2 behemoth GPUs.
9:17  Not that interesting.
9:18  All right, guys, thank you for listening.
9:21  Appreciate you watching the short video on Lennie,
9:24  and I will see you in the next one.