Strata runs Qwen 3.8 Flash Next, a 177B parameter MoE, at almost 100 tokens per second on a single 4090. Nsight traces show how: hot experts stay in VRAM and the next cache misses stream in before the GPU needs them. I still won’t run it on my inference box, so I forked it, deleted the Python, and made it build offline.
Want a copy without ads, an account, or tracking? Grab the torrent (151.5 MB).
It includes a webseed, so it downloads even when nobody else is seeding. Please seed it if you can.
Magnet link
From the final-cut captions. Names and technical terms have been corrected; timestamp links open the corresponding point on YouTube.
0:00Strata is a specialized inference engine for one model, Qwen 3.8 Flash Next. The model itself is incredible, and Strata has really amazing performance. I'm getting almost 100 tokens per second on a 177 billion parameter model on my 4090 at home, which I didn't think was possible.
0:17The model has incredible caching behavior. It anticipates cache misses and starts streaming them to the GPU ahead of time, and it uses all the resources on your machine to do inference super fast.
0:29That all sounds amazing. I won't be using Strata. I don't recommend using it. My preference in my deployment model is for local only offline inference. Strata doesn't fit with that model outline of this video.
0:43I'm going to put the cliff notes at the very front. My recommendation is don't install it. that YOLOs random blobs from a 2 week old repo. Second, the model itself is incredible. It's a new model from Qwen. Third Strata and how it works, this is the meat of the video.
0:59My conclusion is that there are 2 main tricks, parallelism and caching. We're going to visualize the results of this. Fourth, what does this mean for llama.cpp? If this is a race, the GGML guys are running backwards. And I think it's extremely relevant to discussing Strata.
1:15And lastly, my fork of Strata for safe offline use.
1:20So my recommendation in a little bit more detail is do not use Strata, or use docker to run it, or consider wiping your machine after you install it and run it. Inference servers are probably the most critical pieces of infrastructure that we have in the home lab community or folks who are doing inference in, you know, at home or at work.
1:43And I don't think that they should have broad permissions and persistent access to the greater internet. So, Strata is violating at least 3 cardinal sins of software for these machines. One is downloading random binaries and running them from a two-week old GitHub repo.
1:58It's just not a good idea. I don't care if it's Strata or something else. I don't recommend this. Second, is installation by downloading scripts and markdown and yeeting that shit into your agent of choice. Also not recommended.
2:112026 is a low trust environment, all right? This is like piping curl into bash on steroids and crack and a huge dose of creatine. Just, no, it's the mother of all attack vectors. The thing has 17,000 GitHub stars. It could have 10 X that and the number of installs.
2:28It's a very juicy attack target, okay? So even if it's not compromised today, treat it with suspicion. And guys, this is not specific to Strata. I felt the same way about OpenClaw. Okay, the 3rd cardinal sin is Strata's setup.py, download shit automatically from GitHub without asking you, maybe not malicious, but absolutely not cool.
2:50So, uh, Niko, I chatted with the gentleman who appears to be in control of the GitHub repo, and he seems like a very nice guy, and he said that he would address these issues. So in general, like be skeptical, okay? At the simplest end, use a container.
3:06I don't see enough people using common sense about this kind of stuff, like OpenClaw, dots, whatever metas, free AI agent is called. This is why I love llama.cpp. I download the source code. I rsync it to my inference machine, and then I run make.
3:23Done. Can you guys imagine how fun I am at parties? I'm the guy who shows up and then I'm going through your cupboard for the spice cabinet and checking the expiration date on everything.
3:33Base model intelligence. It's incredible. It made this beautiful pagoda. It made this super Dario platformer where Dario runs around and turns all the lakes into money. We'll do a full video on that. It made this graphic calculator.
3:47It's really smart. But like I said earlier, it's a 177B parameter model from Qwen, it's going to slap.
3:55Okay, Strata. Yeah, the main part of this video is about Strata, how it works, how it's so fast. I got 90 tokens per second and 2500 prompt processing. It does go down as you fill the context window. That's like 2 to 3 times as fast as llama.cpp on the same model with essentially the same settings.
4:12There's one catch here. I ran it on a RunPod container and on a container on my machine at home. And the physical machine had 128 cores and 256 threads, and I had like 32 of those for my, uh, container that was allocated to me on RunPod.
4:31And Strata spawned way too many loader threads and nuked its own performance. So check the num_workers flag and that can severely hamper your day if you're trying to run. My agent found it really quickly.
4:44It wasn't a big deal. In terms of design, everything follows from this one idea, keep the GPU busy at all times during decode and prompt processing. This clip visualizes the performance data captured by NVIDIA Nsight, my agent, I believe it was Astra or Fable assembled the frames into Swift, adding some metadata to them, but the key tool here is NVIDIA Nsight.
5:10This is a real-time render of llama.cpp performing a pre-fill of 4096 bytes. The 1st of all, the function that's currently executing is being shown. So mul_mat_q is the matrix by matrix kernel in llama.cpp, and for a given 1/30th of a second window, which is each frame in the in the video, it shows the kernel that had the largest total overlap with the window, so not necessarily the longest kernel or the most recent one, but the one that had the most overlap.
5:40The blue teal seafoam green swim lane, the GPU being busy. We want to see that full. It's not full. If I had to give llama a grade on this metric, I would give it a D for didn't do it. The ambiguously orange yellow lane is moving memory over PCIE triggered by CUDA's cudaMemcpy.
5:59And then pink is the GPU being idle. this is constructed after the fact. Retrospectively, ex post facto. later, okay? The CPU is the white line. If it's full to the top, that's one core in use, and then the gray means more than one core of CPU is being used.
6:16So, guys, is llama doing a good job?
6:18Now, the next one is Strata. Look at that beautiful teal mass, and it's done in only 4.8 seconds. Gorgeous. I don't know why people are playing AION 2 when they can look at this.
6:31All right, so how does it do this? If you want, you can go watch Codacus's video on the topic. I watched it after inspecting the code and spending, I don't know, 5 or 6 hours between reading it, having my agents look at Strata and running both Strata and llama and llama forks that are purporting to do things similar to Strata.
6:54As I was watching the video, I could feel myself getting smarter in the areas of my brain, which are small but present and know things about back propagation. And I could feel my visual cortex physically decaying and getting stupider from watching these AI generated stick figures run around in a kitchen.
7:13So maybe just listen to the video from Codacus and don't watch the AI generated cooks pass things around in the pantry. My point is, he covers it in really great detail, and talks about a lot of the mechanisms within Strata, and I think the video is actually pretty accurate, according to my understanding of how it works.
7:34Now, to be clear, I talk about llama.cpp and running inference engines a lot. So asking me how they work is kind of like asking a truck driver to design a Mack truck. So what are the main tricks? Keep the hot experts in VRAM, okay?
7:50This is this is critical. Recalculate which ones are hot as you are going. It does not keep a simple LRU cache. It maintains a distribution of the experts that get used the most. Anticipate the upcoming cache misses and stream them into VRAM.
8:07This is going to allow us to see that ambiguously yellow orange region stay busy as it's constantly loading the next layer's experts that are going to get mixed. Now, it does a few interesting things that I view as corollaries to that.
8:23And one is that it has a separate binary pack file for the model. I thought this was really dumb at 1st because we have GGUFs, okay? And the GGUF contains everything that you need to run it. But it turns out that within a GGUF, a single expert is contained within 3 different tensors.
8:42And so you need to create the pack file, just to keep the experts in a format on disk, such that you can easily cache them. It actually makes a lot of sense. So like I said, everything follows from this one idea. Keep the GPU busy at all times.
8:59And llama.cpp just fails at this.
9:02So, on that note, like, this is why we should care about llama.cpp and talk about it here because I think that frameworks like ninfer, like Strata, like FreeToken. Oh, and DeepSeek or Dwarf Star, these are going to become more and more prevalent and important and be the choice that people reach for far more frequently.
9:27When it comes to new frameworks, I don't go off and review every single one. But that's why I'm spending the time here on Strata because this is an example of this new, very interesting class of specialized inference frameworks.
9:44And I think it's going to overtake llama.cpp and GGML if they don't change the way that they are operating. So, I predict that we're going to see more cumulative GitHub stars on things like ninfer and Strata than on llama.cpp.
10:00And one of the problems that they're having is they don't view contributions from the community.
10:06Update from the future, and it looks like all the attention to Strata got to them, and they started paying some attention to these PRs, and Georgi actually merged this GPU cache for MoE experts earlier this week on Wednesday.
10:21And one of the problems that they're having is they don't view contributions from the community, period. I looked at 15 different pull requests to llama.cpp for this MoE expert caching in VRAM. And I don't know if they're choosing not to merge them or not looking at them, but something's going wrong.
10:43And llama.cpp is so desperately behind on this, this one thing, execution of MoE efficiently. So, I don't want llama to lose relevance. I love the fact that I can clone it and copy it to my server and run make, and then I'm good to go.
11:01But it's not looking good for our heroes. So in addition, If llama were to lose relevance, blind trust in highly internet connected new frameworks, like Strata, it's definitely a bad thing.
11:17Okay, so my fork of Strata. I would rather have my daily driver Windows machine auto run a USB stick that I found in the parking lot that said Stuxnet on the side and Sharpie than install Strata on my home inference machine.
11:33I want to run it though. I want all the good stuff from it. So I took Strata, and I told my agents to remove all the Python, and all the downloading random shit at build time, and it works a lot like the llama.cpp model.
11:47So the tokenizer is now in C++. And the templating and the front-end server, basically all the Python junk got deleted and replaced with C++ by my agents. And it works pretty well, but I haven't really stress tested it to the extent that others have probably stress tested Strata.
12:07So, take it with a grain of salt. I think we should also be careful about influencers and content creators who are pitching us on products that they are using. Maybe they're vibe coding something and putting it out there.
12:19I don't feel great about that, but in this case, if you download this and use it the way it's intended, I will never know, because you'll clone it, you'll copy the tarball to a machine that's completely offline and then you'll run make.
12:34And I won't be involved. So, the fork requires no network access to build and it doesn't download random binaries. It does an auto update and replace stuff on disk without asking you. It's a simplified version of Strata, without any additional smarts or anything.
12:53So, get your CUDA drivers on your inference box, get the CDROM that came with your 5090, install that stuff, and then you'll be good to go. If you want to use it, you can, if you want to use Strata, you can also do that.
13:07It's a free country, but it used to be.
13:10In conclusion, I had this moment when I read the PageRank paper. And again, when I read the Chord distributed hash table paper, where I felt like this kind of experiencing a spark of genius, you know, where I was reading this paper and I was like, wow.
13:28That's amazing, and people are using this to actually build real products, like, you know, the massive databases behind Google and indexing the web. And then, so that's one way to experience something new technologically.
13:42The other way to do it. is the way Garry Kasparov experienced the genius of chess engines. And when he finally lost IBM's Deep Blue, he accused the team of cheating. And he thought that they had been secretly using Grandmasters.
13:58Because Garry saw deep genius in what the computer was doing for the 1st time. Now, today, if you're playing chess and they accuse you of cheating is because you're using a computer. not because you're using a grandmaster, but it was different back then.
14:11And so I don't know which one it is for me. Because when I sat down and I understood what Strata was doing and how it was working, I had that same moment of realization of like, wow, this is, this is incredible. It was especially highlighted to me when I saw how poorly llama.cpp does the same thing.
14:32Watching the traces, and seeing llama.cpp drop the ball on pre-fetching, and on keeping the GPU busy. I was like, wow, I get it. My opinion, is that somebody with deep technical expertise sat down for weeks planning and executing and designing Strata.
14:56And I would like to see the planning traces and conversations from it because they're not committed. It starts with the 1st commit is source code. So, I told this to Astra and I was like, wow, man, isn't this genius? I mean, they do all this pre-fetching and uh, they've got the VRAM cache.
15:18Like, this is great. And Astra was like, eh. They cite a couple papers and they're using some ideas from llama.cpp and they're pulling in some kernels. Isn't that great? I was like, fuck you, Astra.
15:31Okay, to wrap up. Strata is incredible. Don't use it. The guy who seems to control the GitHub repo chatted with me and agrees about the security issues. He says he's going to resolve some of them. If you enjoyed the video, like the video, consider subscribing to the channel, if you didn't like the video, press the down vote button, make sure you press it twice because it only works if you press it twice.
15:52I'm going to see you guys in the next one, but 1st I want to give a shout out to our 2 YouTube members, zirconprime and Kruz_AI. Thank you guys for supporting the channel in this way, and I'll see you in the next one.