Gemma 30B with 256K context runs at 20 tok/sec on my M3 Max with 128GB RAM so I think there’s something wrong with your setup. This should run at ~30-40 toks. Maybe your inference engine is not optimised for Mac.
then used it in a very minimalistic pi with a very small system prompt.
Didn't spend much time to try to optimize it tbh, but my issue was not the speed. it just could not make a decision on how to implement the task, kept going on an on.
Yes, I use LM Studio with MLX support, which is specifically faster on M series Macs. I am not sure if llama is the same, but I guess what I’m saying is if you want the performance to be good on M series you have to use models packaged in the right format.
Maybe you have to set reasoning effort to low. 3.8 27B on x-high (default) reasons forever on anything complex. I asked it to write down the answer plan so far leaving open questions as open and it wrote the plan twice in reasoning (and more times partially) while it dilly-dallied about open questions before realizing “ok the user just asked to leave questions open.” That was a Q6o quant with unquantized KV cache.
Neat. This is pretty close to what I’d be looking for, if I could really afford to pull the trigger on something like this (realistically I can’t - iMessage on laptop is too important for work).
If it natively plugged into the Apple ecosystem, now that would be difficult to refuse.
1) If you want to make Claude usable how about addressing the fact that OpenAI 20x lasts all day on Astra, yet Claude 20x pips out in 20 mins of agentic with Fable 5.1, which is totally useless.
And unlike Sol: Opus like yoda, anything for unusable he is.
2) The first AI company to reduce their token pricing such that Astra/Fable can run for less than $150/day from credits would see a 5x usage increase and 20x spend increase from me. It’s currently around $500 for Astra and $5000 for Claude.
3) They seem to recently have addressed the fact that their safeguards were OTT so kudos to them for realising and now Fable is usable for wire and memory level programming.
I am a huge advocate of this kind of app paradigm:
It’s now really easy to spin a terminal emulator and make cross platform terminal apps that work the same on Windows, macOS and Linux regardless of users terminal. You can also make sure the term sends everything required to make the app “nice”, like mouse hover events.
I started using this paradigm for some of my own company’s apps around 3 months ago because it was neat to allow the app to spawn DWM windows for sub windows, dialogs etc. but the app can also fall back to standard TTY for using over SSH or from within a docker or any other terminal. Also, using your own terminal window gives proper taskbar integration (stacking windows) and you can set your own app name/icon.
(It’s possible to 1-shot this in Python using the prompt “use ghostling as a reference, write a terminal emulator in Python using raylib and libghostty that hosts my app” and the first pass was around half the speed of Ghostty.)
I disagree. First we got everything as HTML/Electron apps, now the trend is everything as TUIs. Cross-platform, better, but still worse somehow.
What falls to the wayside are the actual and real productivity and usability improvements afforded by "real" GUI apps over the last 40-50 years.
Don't get me wrong I'm a huge terminal user and absolutely love the clarity of a terminal and flexibility of a shell. But if I have to "learn" to use a TUI app - and they're not as simple to use as their authors might believe - I might as well learn to use the full GUI app with all the niceties available because of it.
Depends on what you're selling. If you do it right, Google will give you plenty of organic customers who want to buy your product - no ads needed. You can market as much as you want on Meta also without having to pay for ads.
But if you need stronger marketing than that, there will always be willing affiliates.
SEO is not nearly as effective as it used to be. Organic click through rates have been trending way down since AI responses were added to the top of the search results. So this really isn't just a case of "you're holding it wrong".
If you're selling something, you love having the AI talking about your product. The move to AI is not a negative for businesses wanting to reach customers organically - on the contrary.
reply