Thanks for sharing, tito! I figured it was time I chimed in properly, as the lead on this project.
Quick mission statement, since it explains everything below: Deltafin runs the full, never-pruned, 2.8-trillion-parameter Kimi K3... all 16 experts, every token, no shortcuts, nothing re-encoded down to "close enough." Speed is not allowed to come from quality.
A few similar K3 projects have shown up claiming to be the same thing, just faster. But they all get there by shrinking the model - quantizing it down, trading fidelity for speed, calling the results close enough. Those are great projects, and have real value - but they're very different projects. Deltafin's purpose remains the real full K3, unaltered and uncut, as fast* as possible.
* it's not fast
Next, an update since this was posted five days ago. Thanks to everyone who filed issues, including a couple of you running M3 Ultra and M5 Max boxes, which helped a lot!
Since this was posted, throughput on the M1 Max laptop went from 0.0141 tok/s to 0.2847 tok/s - about 20x, and still climbing. Most of that came from routine hardening: fixing a storage race, a rust/c++ rewrite, and tolerating a float32 rounding drift that looked like a routing bug until it wasn't.
None of that is a moonshot (get it?). It's the boring, necessary work of making something this huge survive on consumer hardware.
Now one huge thing to get out of the way, because this is about half the comments: is this a viable business product, the kind of thing that makes people in suits happy because it helps them sell widgets? No. My interest in computing goes beyond that, and I suspect a lot of yours does too. I'd like to think that's still what hacker news is largely about.
A research project usually isn't the useful thing itself. It's what falls out of chasing it.
It is genuinely hard to squeeze even a small speed gain out of a model this size on a machine this small. Yes, an M1 Max is a good laptop. But Kimi's own recommended setup is 16 nodes at ~300GB of VRAM each. That's $2,000,000 of hardware. Next to that, this laptop is an ant lifting a car. And that is the point of this project - the insane challenge.
My hope is that if we push as hard as we can to get the largest public model ever released running on the smallest machine that can plausibly hold it, some of what falls out along the way could be useful elsewhere... maybe to smaller models, on smaller machines, running faster than they do today.
But first I wanted to find out whether this was even possible at all. K3 is the largest public model that exists. Nobody had run something this size on a laptop before. The closest comparable projects like colibri and DwarfStar ran quantized models at half this size. Turns out: yes, it's doable, but slow af.
But I've spent a long time frustrated with the quality ceiling on local models, and quantization is a big part of that ceiling. The hallucinations. The reasoning that almost holds together, but then doesn't.
So I'm done compromising on that axis. This isn't going to be another quantized, slightly-worse copy of the real model wearing its name.
Maybe no techniques we’re using are groundbreaking at all in the end. Bummer. But, at least I tried. Try and fail at life, people. That’s okay. Just try, and at least you’ll have lived a little. It's a decent way to spend the time you get.
If people want to keep following this, I'll post updates here instead of just on Reddit like I have been.
We've made some updates to deltafin since this was shared 5 days ago. I wonder if you'd be able to get better speeds now? https://github.com/gavamedia/deltafin
Hi! I'm one of the deltafin devs. The biggest difference is that this is not actually a 100% "pure" uncut Kimi K3. This is requantized to 3-bit residual, whereas deltafin is the full real unaltered k3, through and through.
WASTE reads about 17 GB/token versus Deltafin’s 25.8 GB/pass—roughly. That's 34% less expert traffic, and some could argue a 34% reduction in quality.
My biggest complaint about HN is the users with enough karma to down-vote that do so just because they disagree with you.
I've had legit discussions here about real tech topics (for example web apps vs standard EXE apps), and the guy who continually disagreed with me clearly down-voted every comment I made, and replied with out-dated counterarguments. I wound up losing almost all my karma from one discussion, all while remaining civil.
It's very discouraging to be new here, and try to openly express an opinion. It should take more than one to down-vote a comment successfully.
Security e.g. users shouldn't trust a client based HTML5 with any serious data collection.
- I'm not sure why you say that. I do online banking all the time. It is possible to code something secure and tight, but I'll admit it's not easy. I think security will always be an issue, even with native apps.
Local Data - Storage is limited & it can be manipulated
- True, storage needs to be unlimited. As for manipulating it, this is also true of native apps. Without jailbreaking, I can manipulate a save file for almost any iPhone game. The security issue (again) depends on the developer, and how they protect the data.
Syncing offline apps - Determining which is the latest version etc
- Yup, that's a challenge too. But also a challenge native apps face. When I make a note on an offline iPhone in the notes app, and then edit the same note on my Mac, it creates a duplicate when the iPhone is back online. It should probably be up to the developer how to handle this, whether the app is native or HTML5.
If the browser vendors won't implement something, it won't get implemented. This has also lead to
- Yes, this I think is one of the biggest challenges. People buy a computer, it comes with a browser default, and they never change or update it. It's hard enough to get the W3C to add new features, let alone browsers to implement them, and even harder still to get users to update their browsers.
Format Incompatibilities e.g. try audio/video tags across the major browsers
Regarding your Security point, the fundamental problem with HTML5 and the way online banks use their security is that with HTML5 ultimately the user has control over the code which is running on their machine. Browsers come with decent debugging tools making it easier to abuse (and there are even better ones as plugins).
Using the browsers debugger (or your one of choice) you can simply go to a website running on HTML5 and insert a few breakpoints and watch what happens. All someone would have to do is edit the variable(s) which hold the data to anything they'd desire.
You can argue there are limits to these potential exploitations but some of the tools are as complicated as the complier itself - and when companies are trying to push out a feature as quickly as possible, security issues are often overlooked. For example have a look (if you haven't already) at http://plaintextoffenders.com/ which shows websites which store passwords in plain text. Sure my example relates to a different issue but if websites are prepared to store passwords in plain text then inevitably, they will overlook other potential security issues.
When money is involved, any minor potential exploitation is taken advantage to the fullest extent. Hence I still believe HTML5 apps shouldn't be trusted with any serious data collection.
True, which is another problem web apps face - waiting for the W3C to adopt something new. Native apps run on OSes that change several times a year, introducing new features. New HTML5 features seem to take two years - and worse still, every user must update their browser to get them.
security causes a lot of limits for users and developers
- Security will probably always be an issue, even with native apps. Anything specific you think could be changed to make web apps safer?
unhealthy competition between browser vendors, example: WebGL@InternetExplorer
- Agreed. Especially when one browser stays from what's supposed to be "standard", yet rarely is.
its limited to 3 languages: HTML, CSS, JS (and its derivatives)
- Agreed. Google's working on a compiled web language, but it will only work in chrome. They're attempting to make it a standard, but I'm not sure if any other browsers would put the effort into supporting it. And a whole new system like this would take years to implement. But still possible....
Quick mission statement, since it explains everything below: Deltafin runs the full, never-pruned, 2.8-trillion-parameter Kimi K3... all 16 experts, every token, no shortcuts, nothing re-encoded down to "close enough." Speed is not allowed to come from quality.
A few similar K3 projects have shown up claiming to be the same thing, just faster. But they all get there by shrinking the model - quantizing it down, trading fidelity for speed, calling the results close enough. Those are great projects, and have real value - but they're very different projects. Deltafin's purpose remains the real full K3, unaltered and uncut, as fast* as possible.
Next, an update since this was posted five days ago. Thanks to everyone who filed issues, including a couple of you running M3 Ultra and M5 Max boxes, which helped a lot!Since this was posted, throughput on the M1 Max laptop went from 0.0141 tok/s to 0.2847 tok/s - about 20x, and still climbing. Most of that came from routine hardening: fixing a storage race, a rust/c++ rewrite, and tolerating a float32 rounding drift that looked like a routing bug until it wasn't.
None of that is a moonshot (get it?). It's the boring, necessary work of making something this huge survive on consumer hardware.
Now one huge thing to get out of the way, because this is about half the comments: is this a viable business product, the kind of thing that makes people in suits happy because it helps them sell widgets? No. My interest in computing goes beyond that, and I suspect a lot of yours does too. I'd like to think that's still what hacker news is largely about.
A research project usually isn't the useful thing itself. It's what falls out of chasing it.
It is genuinely hard to squeeze even a small speed gain out of a model this size on a machine this small. Yes, an M1 Max is a good laptop. But Kimi's own recommended setup is 16 nodes at ~300GB of VRAM each. That's $2,000,000 of hardware. Next to that, this laptop is an ant lifting a car. And that is the point of this project - the insane challenge.
My hope is that if we push as hard as we can to get the largest public model ever released running on the smallest machine that can plausibly hold it, some of what falls out along the way could be useful elsewhere... maybe to smaller models, on smaller machines, running faster than they do today.
But first I wanted to find out whether this was even possible at all. K3 is the largest public model that exists. Nobody had run something this size on a laptop before. The closest comparable projects like colibri and DwarfStar ran quantized models at half this size. Turns out: yes, it's doable, but slow af.
But I've spent a long time frustrated with the quality ceiling on local models, and quantization is a big part of that ceiling. The hallucinations. The reasoning that almost holds together, but then doesn't.
So I'm done compromising on that axis. This isn't going to be another quantized, slightly-worse copy of the real model wearing its name.
Maybe no techniques we’re using are groundbreaking at all in the end. Bummer. But, at least I tried. Try and fail at life, people. That’s okay. Just try, and at least you’ll have lived a little. It's a decent way to spend the time you get.
If people want to keep following this, I'll post updates here instead of just on Reddit like I have been.