Hacker Newsnew | past | comments | ask | show | jobs | submitlogin

George Hotz already implemented LLaMA 7B and 15B on Twitch yesterday on GPU in Tunygrad llama branch:

https://github.com/geohot/tinygrad/tree/llama

The only problem is that it's swapping on 16GB Macbook, so you need at least 24GB in practice.



There are two coding sessions streamed so far, very interesting to watch

George Hotz | Programming | can we fit a LLaMA inside a tinygrad? https://www.youtube.com/watch?v=0kRDs9BW2NU

George Hotz | Programming | ChatLLaMA: get in losers we're building a chatbot https://www.youtube.com/watch?v=nctqc8FBJ2U


Both are very impressive. A nice thing about Gerganov's implementation is that it is written in almost pure C. Arguably easier for deployment.


What does almost mean in this case?


There is also C++


Iow it probably wouldn’t compile with an actual C only compiler, but by and large it looks more like C than like C++?


There is also a gpu-acelerated fork of the original repo

https://github.com/remixer-dec/llama-mps


> For 7B model, it always goes above 32gb of RAM,

That's double of what Tinygrad uses


Tinygrad is using openCL right?


It's not tinygrad really, it is PyTorch.


Unfamiliar with the domain, but is it? Looking through the changes (https://github.com/geohot/tinygrad/compare/master...llama) it seems to not mention pytorch at all but tinygrad + numpy.


Can you link to the Twitch stream VoD?


https://www.twitch.tv/georgehotz/videos?filter=archives&sort...

although, there is a VOD channel on YT that might be better.


why does it say video unavailable?


Hmm. I'm not sure. Presumably this will be updated soon: https://www.youtube.com/@geohotarchive/videos


I'm pretty sure he only makes his past twitch streams available to subscribers. It's weird they are available on Youtube, maybe he doesn't know that's something he can change.


I don't think he runs the YouTube channel - it's managed by some of his fans. But like what's he gonna do about it? Send them a takedown notice?


Not sure about his more recent content but he used to have a policy that the VODs could be uploaded in full as long as they were uncut and not modified. If you go watch some older VODs he says so at the beginning.


This is such a refreshing and neat way to use Twitch.


that is so impressive!




Guidelines | FAQ | Lists | API | Security | Legal | Apply to YC | Contact

Search: