0:00  Strata is a specialized inference engine for one model,
0:02  Qwen 3.8 Flash Next.
0:04  The model itself is incredible, and Strata has really
0:08  amazing performance.
0:09  I'm getting almost 100 tokens per second on a 177 billion
0:13  parameter model on my 4090 at home, which I didn't think was possible.
0:17  The model has incredible caching behavior.
0:20  It anticipates cache misses and starts streaming
0:23  them to the GPU ahead of time, and it uses all the
0:26  resources on your machine to do inference super fast.
0:29  That all sounds amazing.
0:30  I won't be using Strata.
0:32  I don't recommend using it.
0:34  My preference in my deployment model is for local only
0:37  offline inference.
0:39  Strata doesn't fit with that model outline of this video.
0:43  I'm going to put the cliff notes at the very front.
0:45  My recommendation is don't install it.
0:47  that YOLOs random blobs from a 2 week old repo.
0:51  Second, the model itself is incredible.
0:53  It's a new model from Qwen.
0:56  Third Strata and how it works, this is the meat of the video.
0:59  My conclusion is that there are 2 main tricks, parallelism and caching.
1:03  We're going to visualize the results of this.
1:06  Fourth, what does this mean for llama.cpp?
1:09  If this is a race, the GGML guys are running backwards.
1:12  And I think it's extremely relevant to discussing Strata.
1:15  And lastly, my fork of Strata for safe
1:18  offline use.
1:20  So my recommendation in a little bit more
1:23  detail is do not use Strata, or use docker
1:27  to run it, or consider wiping your machine after you install it and run it.
1:32  Inference servers are probably the most critical
1:35  pieces of infrastructure that we have in the home lab community
1:38  or folks who are doing inference in, you know,
1:41  at home or at work.
1:43  And I don't think that they should have broad permissions and
1:46  persistent access to the greater internet.
1:48  So, Strata is violating at least 3 cardinal sins
1:51  of software for these machines.
1:54  One is downloading random binaries and running them from a two-week
1:57  old GitHub repo.
1:58  It's just not a good idea.
1:59  I don't care if it's Strata or something else.
2:01  I don't recommend this.
2:03  Second, is installation by downloading scripts
2:06  and markdown and yeeting that shit into your agent of choice.
2:09  Also not recommended.
2:11  2026 is a low trust environment, all right?
2:15  This is like piping curl into bash on steroids
2:18  and crack and a huge dose of creatine.
2:20  Just, no, it's the mother of all attack vectors.
2:23  The thing has 17,000 GitHub stars.
2:25  It could have 10 X that and the number of installs.
2:28  It's a very juicy attack target, okay?
2:31  So even if it's not compromised today, treat it with suspicion.
2:36  And guys, this is not specific to Strata.
2:39  I felt the same way about OpenClaw.
2:41  Okay, the 3rd cardinal sin is Strata's setup.py,
2:45  download shit automatically from GitHub without asking you, maybe not
2:48  malicious, but absolutely not cool.
2:50  So, uh, Niko, I chatted
2:53  with the gentleman who appears to be in control of the GitHub repo,
2:56  and he seems like a very nice guy, and he said that he would address these issues.
3:01  So in general, like be skeptical, okay?
3:04  At the simplest end, use a container.
3:06  I don't see enough people using common sense about this
3:09  kind of stuff, like OpenClaw, dots, whatever
3:12  metas, free AI agent is called.
3:16  This is why I love llama.cpp.
3:18  I download the source code.
3:20  I rsync it to my inference machine, and then I run make.
3:23  Done.
3:24  Can you guys imagine how fun I am at parties?
3:27  I'm the guy who shows up and then I'm going through your cupboard for the spice
3:30  cabinet and checking the expiration date on everything.
3:33  Base model intelligence.
3:34  It's incredible.
3:35  It made this beautiful pagoda.
3:37  It made this super Dario platformer where Dario
3:40  runs around and turns all the lakes into money.
3:44  We'll do a full video on that.
3:45  It made this graphic calculator.
3:47  It's really smart.
3:49  But like I said earlier, it's a 177B parameter
3:52  model from Qwen, it's going to slap.
3:55  Okay, Strata.
3:56  Yeah, the main part of this video is about Strata, how it works, how it's so fast.
4:00  I got 90 tokens per second and 2500 prompt processing.
4:04  It does go down as you fill the context window.
4:06  That's like 2 to 3 times as fast as llama.cpp
4:09  on the same model with essentially the same settings.
4:12  There's one catch here.
4:13  I ran it on a RunPod container and on a container on my machine at home.
4:18  And the physical machine had 128 cores
4:21  and 256 threads, and I had like 32
4:25  of those for my, uh, container
4:28  that was allocated to me on RunPod.
4:31  And Strata spawned way too many loader threads and
4:34  nuked its own performance.
4:35  So check the num_workers flag and that can
4:40  severely hamper your day if you're trying to run.
4:42  My agent found it really quickly.
4:44  It wasn't a big deal.
4:45  In terms of design, everything
4:48  follows from this one idea, keep the GPU
4:51  busy at all times during decode and prompt processing.
4:55  This clip visualizes the performance data captured by NVIDIA
4:59  Nsight, my agent, I believe it was Astra
5:02  or Fable assembled the frames into Swift, adding
5:05  some metadata to them, but the key tool
5:08  here is NVIDIA Nsight.
5:10  This is a real-time render of llama.cpp performing
5:14  a pre-fill of 4096 bytes.
5:17  The 1st of all, the function that's currently executing is being shown.
5:20  So mul_mat_q is the matrix by matrix
5:23  kernel in llama.cpp, and for a given 1/30th of a
5:26  second window, which is each frame in the in the
5:29  video, it shows the kernel that had the largest total
5:32  overlap with the window, so not necessarily the longest kernel
5:36  or the most recent one, but the one that had the most overlap.
5:40  The blue teal seafoam green swim lane, the GPU being busy.
5:44  We want to see that full.
5:45  It's not full.
5:46  If I had to give llama a grade on this metric, I
5:49  would give it a D for didn't do it.
5:51  The ambiguously orange yellow lane is moving memory
5:54  over PCIE triggered by CUDA's cudaMemcpy.
5:59  And then pink is the GPU being idle.
6:02  this is constructed after the fact.
6:03  Retrospectively, ex post facto.
6:06  later, okay?
6:07  The CPU is the white line.
6:09  If it's full to the top, that's one core in use, and then the gray
6:12  means more than one core of CPU is being used.
6:16  So, guys, is llama doing a good job?
6:18  Now, the next one is Strata.
6:20  Look at that beautiful teal mass, and it's
6:23  done in only 4.8 seconds.
6:26  Gorgeous.
6:27  I don't know why people are playing AION 2 when they can look at this.
6:31  All right, so how does it do this?
6:34  If you want, you can go watch Codacus's video on the topic.
6:39  I watched it after inspecting the code and spending, I
6:42  don't know, 5 or 6 hours between reading it, having my
6:45  agents look at Strata and running both
6:49  Strata and llama and llama forks that are purporting
6:52  to do things similar to Strata.
6:54  As I was watching the video,
6:56  I could feel myself getting smarter in the areas
6:59  of my brain, which are small but present and know things about back propagation.
7:03  And I could feel my visual cortex physically
7:06  decaying and getting stupider from watching these AI
7:09  generated stick figures run around in a kitchen.
7:13  So maybe just listen to the video from Codacus
7:16  and don't watch the AI generated cooks pass things around in the pantry.
7:20  My point is, he covers it in really great detail, and
7:24  talks about a lot of the mechanisms within Strata,
7:28  and I think the video is actually pretty accurate, according to
7:31  my understanding of how it works.
7:34  Now, to be clear, I talk
7:37  about llama.cpp and running inference engines a lot.
7:40  So asking me how they work is kind of like asking a truck driver
7:43  to design a Mack truck.
7:45  So what are the main tricks?
7:47  Keep the hot experts in VRAM, okay?
7:50  This is this is critical.
7:52  Recalculate which ones are hot as you are going.
7:55  It does not keep a simple LRU cache.
7:58  It maintains a distribution of the experts that get used the most.
8:03  Anticipate the upcoming cache misses and stream them into VRAM.
8:07  This is going to allow us to see that ambiguously yellow orange region
8:11  stay busy as it's constantly loading the
8:15  next layer's experts that are going to get mixed.
8:18  Now, it does a few interesting things that I view
8:22  as corollaries to that.
8:23  And one is that it has a separate binary pack file for the model.
8:27  I thought this was really dumb at 1st because we have GGUFs, okay?
8:31  And the GGUF contains everything that you need to run it.
8:35  But it turns out that within a GGUF,
8:38  a single expert is contained
8:41  within 3 different tensors.
8:42  And so you need to create the pack file, just to keep the
8:46  experts in a format on disk, such that you can easily cache them.
8:51  It actually makes a lot of sense.
8:53  So like I said, everything follows from this one idea.
8:57  Keep the GPU busy at all times.
8:59  And llama.cpp just fails at this.
9:02  So, on that note, like,
9:05  this is why we should care about llama.cpp and talk about it here because
9:08  I think that frameworks
9:12  like ninfer, like Strata, like FreeToken.
9:17  Oh, and DeepSeek or Dwarf Star,
9:20  these are going to become more and more prevalent and important and
9:24  be the choice that people reach for far more frequently.
9:27  When it comes to new frameworks,
9:30  I don't go off and review every single one.
9:32  But that's why I'm spending the time here
9:35  on Strata because this is an example of
9:39  this new, very interesting class of specialized inference frameworks.
9:44  And I think it's going to overtake llama.cpp and GGML
9:48  if they don't change the way that they are operating.
9:51  So, I predict that we're going to see more cumulative
9:55  GitHub stars on things like ninfer and Strata than
9:58  on llama.cpp.
10:00  And one of the problems that they're having is they don't view contributions from the community.
10:06  Update from the future, and it looks like all the attention to Strata got
10:09  to them, and they started paying some attention to these PRs,
10:13  and Georgi actually merged this GPU
10:17  cache for MoE experts earlier this week on Wednesday.
10:21  And one of the problems that they're having is they don't view contributions
10:26  from the community, period.
10:28  I looked at 15 different pull
10:32  requests to llama.cpp for this MoE
10:35  expert caching in VRAM.
10:38  And I don't know if they're choosing not to merge them or not looking at them,
10:41  but something's going wrong.
10:43  And llama.cpp is so desperately
10:46  behind on this, this one thing, execution
10:49  of MoE efficiently.
10:52  So, I don't want llama to lose relevance.
10:55  I love the fact that I can clone it and copy it to my server and
10:58  run make, and then I'm good to go.
11:01  But it's not looking good for our heroes.
11:03  So in addition, If llama
11:07  were to lose relevance, blind
11:10  trust in highly internet connected new frameworks,
11:13  like Strata, it's definitely a bad thing.
11:17  Okay, so my fork of Strata.
11:21  I would rather have my daily driver Windows machine auto
11:24  run a USB stick that I found in the parking lot that said
11:27  Stuxnet on the side and Sharpie than install Strata
11:30  on my home inference machine.
11:33  I want to run it though.
11:34  I want all the good stuff from it.
11:35  So I took Strata, and I told my agents
11:38  to remove all the Python, and all the downloading
11:41  random shit at build time, and it
11:45  works a lot like the llama.cpp model.
11:47  So the tokenizer is now in C++.
11:50  And the templating and the front-end
11:53  server, basically all the Python junk got deleted
11:56  and replaced with C++ by my agents.
11:58  And it works pretty well, but
12:01  I haven't really stress tested it to the extent that others have
12:04  probably stress tested Strata.
12:07  So, take it with a grain of salt.
12:09  I think we should also be careful about influencers and content creators
12:12  who are pitching us on products that they are using.
12:16  Maybe they're vibe coding something and putting it out there.
12:19  I don't feel great about that, but in this case, if
12:23  you download this and use it the way it's intended, I will never know,
12:26  because you'll clone it, you'll copy the
12:29  tarball to a machine that's completely offline and then you'll run make.
12:34  And I won't be involved.
12:37  So, the fork requires no
12:40  network access to build and it doesn't download random binaries.
12:43  It does an auto update and replace stuff on disk without asking you.
12:48  It's a simplified version of Strata, without
12:51  any additional smarts or anything.
12:53  So, get your CUDA drivers on your inference
12:56  box, get the CDROM that came with your 5090,
12:59  install that stuff, and then you'll be good to go.
13:03  If you want to use it, you can, if you want to use Strata, you can also do that.
13:07  It's a free country, but it used to be.
13:10  In conclusion, I had this moment when I read the PageRank paper.
13:14  And again, when I read the Chord distributed
13:18  hash table paper, where I felt like
13:21  this kind of experiencing a
13:24  spark of genius, you know, where I was reading this paper and I was like, wow.
13:28  That's amazing, and people are using this to actually build real products,
13:31  like, you know, the massive databases
13:34  behind Google and indexing the web.
13:37  And then, so that's one way to experience something
13:41  new technologically.
13:42  The other way to do it.
13:44  is the way Garry Kasparov experienced the
13:48  genius of chess engines.
13:49  And when he finally lost IBM's Deep Blue, he accused the team of cheating.
13:54  And he thought that they had been secretly using Grandmasters.
13:58  Because Garry saw deep genius in
14:01  what the computer was doing for the 1st time.
14:04  Now, today, if you're playing chess and they accuse you of cheating is because you're using a computer.
14:08  not because you're using a grandmaster, but it was different back then.
14:11  And so I don't know which one it is for me.
14:14  Because when I sat down and I understood what Strata
14:17  was doing and how it was working, I had that
14:20  same moment of realization of like, wow, this is, this is incredible.
14:25  It was especially highlighted to me when I saw how poorly
14:29  llama.cpp does the same thing.
14:32  Watching the traces, and seeing llama.cpp
14:35  drop the ball on pre-fetching, and
14:38  on keeping the GPU busy.
14:42  I was like, wow, I get it.
14:45  My opinion, is that somebody with deep technical
14:49  expertise sat down for weeks planning and executing
14:52  and designing Strata.
14:56  And I would like to see the planning
15:00  traces and conversations from it because they're not committed.
15:03  It starts with the 1st commit is source code.
15:07  So, I told this to Astra and
15:10  I was like, wow, man, isn't this genius?
15:12  I mean, they do all this pre-fetching and uh,
15:16  they've got the VRAM cache.
15:18  Like, this is great.
15:19  And Astra was like, eh.
15:21  They cite a couple papers and they're using some ideas from llama.cpp
15:25  and they're pulling in some kernels.
15:27  Isn't that great?
15:29  I was like, fuck you, Astra.
15:31  Okay, to wrap up.
15:32  Strata is incredible.
15:34  Don't use it.
15:35  The guy who seems to control the GitHub repo chatted with me
15:38  and agrees about the security issues.
15:41  He says he's going to resolve some of them.
15:42  If you enjoyed the video, like the video,
15:45  consider subscribing to the channel, if you didn't like the video, press the down vote
15:48  button, make sure you press it twice because it only works if you press it twice.
15:52  I'm going to see you guys in the next one, but 1st I want to give a shout out to our 2
15:55  YouTube members, zirconprime and Kruz_AI.
15:57  Thank you guys for supporting the channel in this way, and I'll see you in the next one.