I've learned things I wouldn't otherwise have learned because I hadn't considered using the tools the LLM recommends. It's also a way to eliminate some hallucinating, given that critical questions are posed as unbiased as possible. For that, I also like to open a new chat with a different model and asking open-ended questions about a recommended tool I don't know much about, to double-check that the original LLM was likely correct in its recommendation in the first place.

Individual guidelines can live in a dot-prefixed subdirectory or the users $HOME.

This is what they already provide.

So it's possible (but not necessary) to add personal pre-prompts, but the shared best-practises and, more importantly, learnings, can be comitted and pushed.

What's exposed on a web server is a different topic, and when that is your repo's root dir, you've done something wrong either way.

Instead, a scroll should give you a break before heading into the next video. I'm willing to bet this would help severely with addiction. People are then forced to reconsider whether they actually want to play the next video. "Done" should not always lead to "here's the next stimulus". That's what's addictive. The brain isn't made to break out of that loop easily.

This discussion makes me both laugh and feel sad, because we all know this is bad for us and gives us zero ROI for our time... and yet there's a whole thread developing here to justify the pattern.

I don't think I have a point on that, just that observation.

There is no allowable constitutional authority for actions like this. CA is literally overstepping the 1A limits of the Constitution here.

• It is content neutral.

• The government can probably show a significant government interest in reducing the harms infinite scroll often leads to.

• It is narrowly tailored. It achieves the goal without burdening more speech than is substantially necessary to achieve the goal. Arguably it doesn't burden any speech since every word you can have on an infinite scroll page you can have on a paginated site.

• There are alternate channels. The speakers still can get their message across. In this case they can get it across to the exact same audience in the exact same place. They just have to stick in page breaks.

While the court has once or twice extended protections to people using private property as a public forum, to my knowledge they have never done so with time/place/manner restrictions.

Hint: the physical world is very different from the virtual world, and has different limitations.

Hint #2: if I crank up the amp to 1000db and shout into it, it’s obviously not a question of speech anymore. This is obviously an extreme example (the energetic release just destroyed the planet), so dial it back to where it’s reasonable and concerns are balanced. Are you still facing actual physical discomfort? Did you dial it back enough?

Hint #3: is my nighttime writing keeping you awake in your home?

To be honest I suspect much of the support for this bill here is inorganic, and I do feel extreme contempt for the people pushing it.

In the end I’m not really trying to convince these posters—they have obviously made up their mind—but rather to entertain and educate the nonaligned audience.

Why not link to those instead of a youtube video?

Should/do we allow foreign propaganda radio stations? If we accept that the government can (and very much does) impose itself on content platforms for "national security", what exactly is the difference between deliberately insidious information warfare, and collateral damage from market incentives?

I agree that its better to find solutions that involve protections instead of restrictions though. I think it means forced decoupling of indices/curation from advertising. This would make advertising funded addiction feeds compete with paid feed applications.

How do you define this? Is it foreign owned? Noncitizens are not guaranteed the same rights, especially citizens of hostile foreign powers.

> If we accept that the government can (and very much does) impose itself on content platforms for "national security"

Why should we accept this?

> what exactly is the difference between deliberately insidious information warfare, and collateral damage from market incentives?

None, both are concepts not found in the Constitution (if you’re talking about domestic speech by citizens) and both are protected by 1A.

I don’t care in the slightest about your fearmongering national security nonsense.

It's also telling that only corporations seem to be the ones demanding the right to infinite scroll; what's the scenario where an individual can only express themselves and their ideas through implementing infinite scroll on a social media?

We draw lines in the sand all the time for the sake of public safety, I'd like to hear a specific case of harm here.

The First Amendment is clear: there shall be no law abridging freedom of speech. Courts have bent around that in the past, in earlier eras, but they were wrong to do so. Their mistakes have mostly been corrected although there’s still a few left.

The document that governs this country spells it out: it can’t be done. Public safety be damned. There’s no public safety exemption in the Constitution. If you want it done, pass an amendment. There’s a process for it.

I personally dislike infinite scroll, but I dislike the camel’s nose in the tent even more. No speech laws.

Seriously, this diffusion of individual liberties into corporations has no presence in the constitution, and courts have fabricated this wholesale. There is no idea, no concept, no notion that infinite scroll provides. We regulate the size, location, and brightness of billboards; is this also a matter of speech?

Oh is the presentation of text and images not “speech” because it’s “addictive”? I didn’t realize that.

Your strategy with billboards is more clever than I’ve usually seen from you lot; I’ll give you credit for that. A billboard is actually a physical structure. The message on the billboard is the speech. If I stopped here you’d have a “gotcha”; the software must be like the billboard! But no, because first of all, code is speech, and secondly, the layout of items on the screen and how they interact is also just speech. It’s just graphic and UX design! There is no physical structure here. You’re attempting to regulate the presentation of information—design.

But courts are willing to find that certain speech that is apolitical can be limited (the previous "fire in a crowded theatre" example). Basically the courts have recognized 1A established freedom of speech to protect political dissent and political ideas. Porn, for example, has limitations that would never apply to political ideas.

Limitations on porn still exist in a few areas, but they are gradually being rolled back—obscenity laws were once widespread and highly restrictive. Most still standing carveouts are pretzel twists that probably need to be corrected with a clarifying amendment; they are on very shaky ground.

The court has recognized speech protections outside of politics many times, including protections for authors and creators who were not explicitly aiming for political statements. For example, Brown v. Entertainment Merchants Association established that video games are protected expressive speech, even if they are violent trash that aren’t attempting any political point whatsoever.

A law doesn’t just get a constitutional bypass because it’s addressing known harms or “anti-social” behavior or whatever. This is not the UK.

Illuminated signs exist in the real, physical world. They can beam bright light into your home, involuntarily. Design and presentation exists in the realm of a printed page, or on the display of your device. Can we regulate how a book lays out its type?

The First Amendment is quite possibly the most uniquely American thing about our Constitution, and its most defining feature. It’s worth defending.

Does society benefit from mass ownership and unlimited access to fully automatic rifles and grenade launchers? If it does, what country allows it?

Are the above constraints explicitly decided as constitutional though years of legal decisions at all levels of the courts? Yes? Then we can observe that we can reasonably constrain constitutional rights through law and legal opinions. The line may be hard to draw and may shift, see the AR ban, but it is accepted that constitutional guaranteed rights have bounds that can be articulated and clarified through the legal and political system.

We put upper bounds on the rights and freedoms of individuals and corporations because we all must live within proximity to each other. These bounds may be authoritarian at times, and of course that's bad. But we collectively can limit freedoms because the alternative is actively and disproportionally harmful to society.

When it comes to the rights and the freedoms of the largest and wealthiest corporations, we already live in an era where these entities are shaping major aspects of our lives. Infinite scroll is one small mechanism by which they're hacking our biology; this is more than just pixels on a screen but a component in a system that was A/B tested to maximize behavior modification.

Help me understand - do you believe that it is possible to regulate these entities in any form? Or do we need to say that the folks that yeeted tea into a harbor were fine with infinite corporate power and regulatory capture?

We’re talking apples and oranges because a website is more like a book than an illuminated sign. You have to decide to view it, and it doesn’t shine through your window at night, disturbing the peaceful enjoyment of your home.

> A law doesn’t just get a constitutional bypass because it’s addressing known harms or “anti-social” behavior or whatever. This is not the UK.

First, the harm arguments are regularly made in front of the supreme court. And sometimes, when it suits them, justices make their own harm or sociality arguments. No, USA is worst. It gets to be constitutional if it advances conservative right wing agenda and unconstitutional otherwise.

> The First Amendment is quite possibly the most uniquely American thing about our Constitution, and its most defining feature.

You dont defend it by redefining its meaning to unrecognizable to encompass things non-speech of corporations. All the while making it so that in practice, poorer people have no defense anyway.

I must also mention that courts are not Congress and states are not Congress. The first amendment does not say "there shall be no law" - that is your poor paraphrasing - it says "Congress shall make no law"

If you’re just going to pick and choose what rights you apply, then it’s not much of a governing document, is it? Is this just “Parliament is Sovereign” with extra fluff? Might makes right?

It’s almost like we need a robust system of checks and balances, governed some kind of rigid framework to ensure that everyone plays by the rules. Or we could just continue to ignore that and see what happens.

That was a little hyperbolic. The government already can regulate "speech" to some extent in limited, targeted ways, as this is. For example: they can (and increasingly will) require ADA accessibility standards on web and mobile sites and apps-- even private sector-- that deal with the public.

It may be that this isn’t as settled as you think when speech concerns are present. The existence of alternative accessible formats, or sufficient assistive technology in the marketplace, may be just as compliant. It’s likely that these will be favored over mandating changes that affect design or presentation, given the Court’s prior decisions on balancing speech concerns in other areas.

And things like image crafting are not even necessarily intentional or malicious. Somebody who posts a pic of their filet mignon dinner probably isn't posting much in the way of their microwaved leftovers mashed up in a bowl. And adults already get mistaken perceptions of others because of this bias, let alone children.

It needs to be baked into the OS.

At that point, HN users start screeching about it, so it's lose/lose, really.

It's annoying, and often rather infuriating, but I understand that one of the motivations for people buying Apple stuff, is for that very reason, so I'm really sawing off the branch that I'm sitting on, by trying to work around it.

Might be why we're already rarely seeing models output an "I don't know".

I've noticed a few things compared to large models like Claude. For starters, you really need to know what you're asking, and be precise; it doesn't do much thinking for you. Any assumptions left open, and it'll take the easiest route to reach the goal (e.g. CSS in HTML), often not the best in terms of architecture.

It gets into loops quite often, and surprisingly often gets the edit tool call wrong, after which it will spend lots of thinking tokens and re-read files instead of retrying (despite the system prompt suggesting so).

Comparing agentic Qwen3.6 35b to Claude Opus is like a junior with knowledge across the board, that you really need to guide, versus a senior that thinks with you on architecture. If Opus gives a 15x speedup, local and fully offline Qwen gives a 5x speedup. Which, given that it's completely free, is still mind-boggling to me :)

I've never used the frontier models in earnest, I don't believe in using proprietary tools for my programming, so I can't really compare.

And I'm still a AI skeptic, so I'm doing more testing and kicking the tires than I am actually using it. That means I spend a lot of time trying to break various models, probe them for strengths and weaknesses, etc.

But I find that when I do try to use it for real for agentic coding, Qwen 3.6 35B-A3B is definitely the one I reach for the most often.

For other chat tasks and translation, I'll frequently use Gemma 4 31B.

For audio, I'll use Gemma 4 12B.

I keep a bunch of other models around to try out every once in a while (Qwen 3.5 122B-A10B, Qwen 3.6 27B, Nemotron 3 Super 122B-A12B, Step 3.7 Flash and Minimax M2.7 both at somewhat more aggressive quants, and GPT-OSS 120B if I want super fast but not terribly smart), but so far Qwen 3.6 35B-A3B is really the sweet spot for coding on a setup like this.

The re-processing context every turn problem is definitely something I've hit. Some of the causes have been solved upstream in llama.cpp; make sure you're up to date.

But another cause of the issue that has a big effect is that older Qwen models didn't support preserving thinking. This means that each time you have a long sequence of tool calls with interleaved thinkging, as soon as you had your next turn in the chat, it would have to re-process all of that as it would drop all of the reasoning.

Qwen 3.6, however, now supports preserving thinking. This can use a bit more context, becasue you're not dropping the thinking every turn, but it re-uses the cache better, not causing you to have to reprocess a whole turn at a time each time.

In my models.ini, I have this for the Qwen3.6 models:

  chat-template-kwargs = {"preserve_thinking": true}

I'll have to give the preserve_thinking a shot.

but for caching, all you are doing is leaving off a fraction of the most recent assistant message generation, which will have little/no impact on cache hit rate.

True, but not a tiny fraction, qwen is very verbose in its thinking traces. And it basically means that for every (nonthinking) generated token you have to compute the KV twice (once as tg, the second one as pp).

  --chat-template-kwargs '{"preserve_thinking":true}'

I've had the best luck with Pi so far, but it comes without some bells and whistles you might be used to (e.g. plan mode, subagents, MCP client support)

Isn't this the nature of how LLMs work? Or do you mean that it recalculates the entire KV cache instead of saving the old KV cache, in which case the problem is likely in your executor (llama.cpp, vllm, e.g.) configuration or capabilities?

Qwen 3.6 has finally been trained both with and without preserving thinking, so you can optionally enable preserving thinking. This will use up a bit more context, but it will avoid having to do this re-processing of long agentic turns, and also the preserved thinking can avoid having to re-do some of the same reasoning over again in later turns.

Besides that, modern LLMs don't only use full attention (apparently, attention is not all you need). Full attention is very expensive to compute and store (0(n^2)). But additionally, full attention is actually bad at certain kinds of reasoning; keeping track of some value that gets replaced over the course of time, for example. So most models these days use various forms of local attention which is fixed length and gets updated as you go; sliding window attention, Mamba-2 state space models, etc.

But one advantage of attention is that you can go back and reprocess by truncating the KV cache and starting over. You can't do that with other forms of local attention; you've lost the state earlier in the sequence.

So to allow you to go back without fully recomputing the cache all over again, your engine will save snapshots of the local attention state at various times, so if you need to go back to recompute the cache, you can start from the last snapshot. However, these snapshots can get large, you can't keep too many of these, so sometimes you need to go back quite far to get to one, or they're all past the point you need to go back to and you need to start over again from the beginning.

There have been particular bugs in llama.cpp that have caused this to be triggered more often than it should; for instance, it wouldn't take snapshots before turns that included images at one point, so if you had an image heavy agentic workflow, that issue plus the lack of preserving thinking would mean you would frequently have to go back and start over from scratch.

Some of these issue have been fixed, some are addressed by preserving thinking. There are still some issues sometimes; for instance, one that's hard to fix is that the tokens generated autoregressively don't always parse the same when doing prefill. For instance, you could generate something as two tokens "pre" and "fill", but it turns out that "prefill" is also a single token so the tokenizer will use that, so when you send that back again on the next turn, it will see a divergence and have to recompute from that point. It might be possible to ignore that and use the not fully greedy tokenization that's in the cache, but I've definitely seen llama.cpp have to do some cache recomputation due to that.

use that to install assuming you have whatevers needed to set up.

ive a much fancier next gen thing i hope to make available as a sass mid to late summer, but if you have any feedback or questions on mine do reach out

The jinja template is what renders the openai-format request sent by the harness, into the actual string of text that will be tokenized and fed to the model. For models without preserve thinking support, the jinja template drops the reasoning from all but the current turn.

Here is the default jinja for Gemma 4: https://huggingface.co/google/gemma-4-31B-it/blob/main/chat_...

    {#- Render reasoning/reasoning_content as thinking channel -#}
    {%- set thinking_text = message.get('reasoning') or message.get('reasoning_content') -%}
    {%- if thinking_text and loop.index0 > ns_turn.last_user_idx and message.get('tool_calls') -%}
        {{- '<|channel>thought\n' + thinking_text + '\n<channel|>' -}}
    {%- endif -%}

Here's Qwen 3.6 by comparison: https://huggingface.co/Qwen/Qwen3.6-35B-A3B/blob/main/chat_t...

        {%- if (preserve_thinking is defined and preserve_thinking is true) or (loop.index0 > ns.last_query_index) %}
            {{- '<|im_start|>' + message.role + '\n<think>\n' + reasoning_content + '\n</think>\n\n' + content }}
        {%- else %}
            {{- '<|im_start|>' + message.role + '\n' + content }}
        {%- endif %}

It's possible to modify the jinja file that you're using with a model. Some people do that with models that haven't been specifically trained for it, and report good results; but some report that because it wasn't trained for that, they get worse results if they include thinking from previous turns.

So for models like Gemma, you would have to modify the default jinja to enable this. For Qwen, you can just set the preserve_thinking flag to get this behavior; and apparently they have trained in this mode so you get better results than models that have not trained this way.

Anyhow, I've heard mixed results on any method of supplying reasoning traces beyond the current turn to models not trained on them. For some models, I've heard that it works fine this way, for others I've heard it degrades performance. But I don't know of anyone who has any kind of reliable benchmark for how well this works.

https://sebastianraschka.com/llms-from-scratch/ch04/08_delta...

There are other reasons to prefer vllm to llama-cpp as well

What does this mean in June 2026 wrt coding?

To me it sounds like being a "rice cooker skeptic". Some people don't like using rice cookers, some do.

For example you can just tell it to make a website for a business with a webshop and it'll just generate thousands of lines of code and you have no control over anything. Or you can spend hours/days writing the specification and then have it generate it.

Or you can do what I do and work iteratively one feature at a time making sure everything is exactly the way you want it. I generally solve the problem myself then tell it what to do, or if I'm not sure what the best solution is I might discuss with the AI until we agree on a plan and then have it execute it. Often this leads to me learning useful things, like it will suggest a tool/feature that I didn't know about that's perfect for my usecase or it will identify a problem in my plan that I wouldn't have found until after spending hours on the implementation.

I've always been very detail oriented and I care a lot about code quality, I want my solutions to be clean, consistent and as simple as possible while solving the problem. To me, AI tools let me do that more quickly and better, it's not a compromise it's just flat out better in every dimension. It's about how you use it.

A lot of people seem to think that it's a binary choice, either hand craft a high quality bespoke solution or just vibe code a pile of trash. There's a whole spectrum in between those two, and I think there's a sweet spot where you still maintain control and understanding, it's just much faster and the result is actually better because it's not just you and the knowledge in your brain it's also the AI that practically knows everything - it will teach you things and suggest solutions you wouldn't have thought about, it makes you a better developer. It's a force multiplier and the smarter you are the better you will be at using it.

It's not a replacement it's an enhancement. It's like imagine a developer with Google vs one without, obviously the one with Google will be better because they have access to more information. The AI is like automatic google that just googles everything all the time, things you wouldn't have even thought to Google or things you couldn't possibly formulate a good search term for. With AI you can just show it a screenshot or describe an issue in detail and get a really solid answer a lot of the time. It's like having an expert on standby all the time, sure it's sometimes wrong but most of the time it's not and if you're smart you'll recognize when it isn't.

I'd say anyone who isn't using AI today aren't using their full potential. I don't see how anyone could possibly perform better without this tool than with it. I do see how someone who doesn't care could produce a lot of slop, but the people who refuse to use it aren't that guy. That guy has been using it to produce slop for years already. You can use it to produce top quality code if you choose to.

This isn't about using rice cookers or not, that's a personal choice for how you cook your food, and choosing to do so or not really only affects the person cooking and cleaning. A rice cooker probably uses a similar amount of energy as cooking it by hand, possibly even less.

But when people using LLMs are causing active harm, and are making it more difficult to collaborate on a team, it's a lot harder to accept that it's just a personal preference.

If you wanted to use the rice cooker analogy, imagine if rice cookers let you cook rice in just one minute. Faster, don't have to wait for the rice to be done, great! But in order to do so, you have to cook 50 pounts of rice, but throw out the majority of it, and use a thousand kilowatt hours of energy to do so. You'd better believe I'm going to be skeptical of everyone deciding that they suddenly have to use these 1-minute rice cookers that burn so much energy and generate so much waste.

I didn't do much benchmarking, but anecdotally, I found it to be making less edit errors. YMMV

Not sure if you intended this to be this philosophical, but this is basically the slogan for modern life now.

Not everyone can plough $$$$ into hardware right now (more power to those who can), so choosing to rent is an A-Ok strategy.

You can. You just don't want to. Huge difference.

You may be, but the topic of discussion is whether anyone is using a local model as their main coding tool.

For corporate use, if the corporation would break the law sending anything to the open internet or to the US, then you can't use any model that's not hosted in house. And there are many such cases.

And sounds like you haven't factored in the cost of electricity to run that Mac Studio as an LLM machine. Probably get a few more years.

I find that running better quantization, like Q8 tend to prevent this even though its a bit slower to run, it saves overall time with less churn

Using 3.6-27b is even slower again than 3.6-35b, but I find the accuracy really pays off

Qwen seems better at one-shotting things based on vague prompts to an acceptable degree, but thats literally not what I use these things for!

One thing if people do play with it, is it seems very very sensitive to quantisation of the K part of the KV cache. F16 K and Q8 V got rid of a lot of the loops that it was otherwise hitting.

There's also a regression in llama.cpp wrt. Step Flash, where quantisation is getting worse KLD and Perplexity than it otherwise was previously, for the exact same quants. Very odd, but it's being looked into at least!

What's your evaluation setup like? It sounds like maybe the best thing to do is have a realistic evaluation that resembles your actual intended workload and workflow, and then just try everything.

That is quite literally what I have setup :)

I have a few codebases I've written over the years that I attempt a suite of specific tasks: code analysis/bug finding, bug fixing, adding features, that kind of thing. I keep track of the results, including wall clock time

>Do you think the choice of quantization matters that much for other models

It hugely matters. Lots more than r/LocalLlama would have you believe, sadly. Some model architectures can handle more aggressive quantisation than others, and it's hard to know ahead of time.

Step handles it surprisingly well (sparse MoE models seem to generally, when the particular layers are chosen to be quantised carefully). Qwen 3.6 27B handles it okay, but FP8 was better... except annoyingly Qwen's official FP8 has worse KLD/perplexity numbers/accuracy than it otherwise should. RedHat's one was better in my testing, though not by a huge amount.

I have a custom assert for loop/repeat detection that works well:

    def count_repeats(text: str, length: int) -> int:
        n = len(text)
        pattern = text[n - length : n]
        count = 1 # Include the end of the string as matching the substring.

        text = text[: -length]
        while text.endswith(pattern):
            text = text[: -length]
            count = count + 1

        return count


    def repeats(output: str, context: dict[str, any]) -> bool|float|dict[str, any]:
        threshold = context.get('config', {}).get('threshold', 3)
        count = 0
        length = 0

        for n in range(1, (len(output) // 2) + 1):
            n_count = count_repeats(output, n)
            if n_count > count:
                count = n_count
                length = n

        if count >= threshold:
            return { 'pass': True, 'score': 1.0, 'reason': f'Output repeats {count} times with length {length}.' }
        else:
            return { 'pass': False, 'score': 0.0, 'reason': f'Output doesn\'t repeat {threshold} or more times.' }


    def no_repeats(output: str, context) -> dict[str, any]:
        result = repeats(output, context)
        result['pass'] = not result['pass']
        result['score'] = 1.0 - result['score']
        return result

    defaultTest:
      assert:
        - # ----- The output doesn't repeat/get stuck in a loop.
          type: python
          value: file://asserts/repeat.py:no_repeats

Ds4 is impressive for what it is, but it loops and over thinks even more, burning massive wall clock time to not even get great outcomes. It’s also limited to a slow speed on my Spark

Step 3.7 is notably better than 3.5

1. Use the official StepFun GGUF, IQ4_XS - theirs is better tuned in my experience than the other quants

2. Temp 1.0 top_p 0.95 sampling parameters for reasoning/agentic coding

3. It's really quite important that you don't quantise the KV cache: it made a surprising amount of difference to the looping and over thinking I found, at least for the quantised version of the model. I'm using the full F16 for K, and Q8 for V

4. Note that it now supports `reasoning_effort: low|medium|high` in your chat_template_kwargs; this is super useful :)

It's what I use. Fixes the problem

https://github.com/day50-dev/petsitter

that's why i use the frontier models because its a senior co-worker vs a junior. if you use the junior for the sake of privacy i think you're missing out on the best insights for a specific task.

Consumer-grade subscriptions of the frontier models give you superb capabilities per dollar, them being heavily subsidized. But if you're working in an enterprise setting, that won't work. You need to upgrade, and that gets significantly more expensive.

Furthermore, basing the SDLC on leveraging the bargain subscriptions risks falling apart in the future, both from a cost perspective as well as the question of availability (e.g. Mythos).

So from a strategic perspective, going local on the LLM and still achieving great results with the right approach is very relevant.

I do (unscientifically) experiment whenever a new capable local LLM (<=130b) releases with a license that permits commercial use. As for knowing my models require more work than Opus, I don't mind still having to puzzle on getting the architecture right. In any case, it forces me to stay in the loop of what's being built, which is a good thing.

I'm not familiar with Pi, and not sure which kind of container you are referring to. Something mainstream like docker, or more classic like a BSD jail?

I started to experiment with locale LLMs, through ollama and Lemonade. Enough to throw simple prompts with code excerpts and get small scope code refactors. Though I still struggled to make them work with external tools, like my IDE, so they can be leveraged on to an agentic level with access to a full repository.

That's mainly for work, as they push for using LLMs, though with the new copilote license they provide it doesn't take me even a week to burn the whole token credit.

The tool can be useful, but in my experience without heavy guard rails and loops over tests. I suspect late models to also burn many token into rabbit hole of nonsense hypothesis, instead of doing straight forward correct implemention as you would expect from any entity with such a huge cumulated resources eaten and experimental playground to leverage on. Maybe incentives don't help model provider to minimize sold token, maybe it's just so hard to tame the beast all these bright minds with virtually infinite resources are not good enough.

Anyway, sorry for digression, but I would be extremely interested with a step by step tutorial to make a local LLM work in agentic level, including which kind of hardware is required to make it work properly.

https://github.com/DeepBlueDynamics/nemesis8

Another POV is that most of the code written in most of my codebases were generated by Codex/Claude, so they would be "stealing data from themselves" in a sense.

I've been working with Transformers/LLM training in 2018-2021 and then now, more recently again. Things are far different. I think they would be more interested in the "how" you got your code to be satisfactory with your guidance than the actual code generated. But mostly I personally trust that they are not really using my trajectories for that (unless I explicitly allow it in the configs)

Is it that in your case is it different?

Yeah, that edit inability is weird. I’ve updated AGENTS.md to limit editing (as opposed to rewriting) and that helps a little.

Maybe even more useful than Opus when I have all the constraints to an issue. There is less "knowledge" in the model (I get by with 48GB of RAM allocated to an 8b quant), so it has fewer things to hallucinate about.

I've been getting to know its limits pretty well over the last few weeks and would say it's an excellent code search/replacement/generation* engine.

It's got the "in-context script generation" flow down as well, so it will easily help automate tasks that you describe with text and perhaps example commands, or tools, or skills* that you provide.

*Think of it + Pi as an NLP abstraction layer over grep, or a shell, rather than a jack of all trades + world knowledge all-in-one.

I've wanted the latter quite a bit for Pi, because weaker models like Deepseek V4 have extreme issues with obeying prompts (e.g. I'll instruct it to find a bug but not fix it, and it'll "helpfully" try to fix it anyway), so having a "read-only mode" actually backed by the OS would be very useful.

[1] https://blog.fidelramos.net/software/how-i-sandbox-ai-agents...

All of these models also seem to get stuck in long thinking loops, sometimes tripling the tokens of a frontier closed model which is really painful when inference is already on the slow side (on my Macbook).

Hold on, what are the specs of your rig? How much RAM?

I've been considering getting an old refurbished 2018 Mac Mini with 64Gb of DDR4 RAM but everything I've read suggests this will be way slower than my 16Gb M1 Pro Macbook.

I've been meaning to write a blog post but well whatever here's the md.

https://gist.github.com/hparadiz/f3596d00a62d8ebb2dadcc46ee5...

Qwen3.5 9B performed best.

You can absolutely still use this to do some basic stuff like tell opencode to convert a video file from one format to another. But frankly you're better off getting two AMD GPUs. Say a dual 7900XT would get way better performance.

So there's this really amazing program called "man"

So that it's done correctly instead of wrongly

> man plows field by hand

Yes, you surely can read man, docs, whatever, then DIY. The point is that in many areas people don’t really want to become an expert, like in ffmpeg cli arguments, they just want the work to be done. Above is an example of agent being able to do it locally, and I think it’s great

matches my experience and a deal breaker

also the context window sizes are too low. I can't operate in 65,000 windows any more because even just reading the code's file structure overruns it and gets me nowhere. Definitely its own art form.

200k context windows and above for me now

I saw a paper last night that should help this a lot though

In Pi, /new is my best friend and most-used command for sure. For simple tasks (I decompose complex ones anyway since I don't trust small local LLMs to do this for me), the model doesn't need much context, given that I'm proficient in my codebase myself: "I'd like Feature X. Look into files 1, 2 and 3 to make your edits."

  > you really need to know what you're asking, and be precise

Thank you.

For the time being, off the top of my head, I'd say:

- Prompt Engineering tips & tricks apply here (like being complete in the relevant context you provide in your question, and the specific task(s) the agent should do like reasoning, modifying one file, or trying to fix a complex task all at once (not recommended)).

- If you already know which files the agent should look into, mention them to save time and potentially context.

- In my personal workflow, I write down lots of atomic TODOs needed to solve a problem. As I write it down, I'll notice assumptions I'm making, or the fact that the TODO could still be decomposed further into (atomic) subtasks.

- It's best to get a feeling yourself for how Qwen handles your repository. I noticed if I don't specify an architecture for development, it'll make quick & dirty fixes. If I don't tell it to remove debug statements, it won't. This is what was meant with "be precise" – Claude Opus might think for you and act in your best interest. Smaller Qwen models will just do what you ask them to, and no more. They have design knowledge, but you have to explicitly ask them to "activate" that part of their knowledge.

I look forward to that blog post!

Two years later, and I'm running Qwen3.6 35b agentically to develop the start of a repository and automatically run tests to then improve on itself. I never thought we'd get here so quickly with LLMs back then.

I'm pretty sure in two years we'll have current Opus-like quality in the 30-100b parameter model range. But at that point, Opus 6.3 will reason along for us so much better still, that we'll still look at those models in awe. It's great to look ahead, but let's not forget to appreciate how effective the current local models already are :)

It's personal, but I prefer CapEx over OpEx for this. If you can purchase a device upfront that runs a decent local LLM, you get the peace of mind that your setup won't suddenly change over time and can only get better.

Now, there's a bit of a degree to which some of the open source models do some benchmaxxing, and bigger models with more params may always feel like they have more depth. But anyhow, right now you have something that is arguably comparable to Claude 4 Opus on your laptop. I can't really compare myself because I never used it. It looks like Claude 4 Opus is still available on OpenRouter, so you could try it out and compare yourself if you're interested.

It will likely always be the case that there are proprietary cloud models that are more powerful than what you can run on a laptop. You can just do a whole lot more with terabytes of VRAM on multi-GPU clusters than you can do on a laptop. So for folks who must have the most capable, you're probably not going to want to leave Anthropic.

But right now, the models you can run on your laptop are comparable to the cloud models that were popular when vibecoding and Claude Code first took off.

Anyhow, feel free to try them out head to head on OpenRouter. I'd love to see someone write up their results, of a modern local sized open source model vs. frontier models from ~a year ago, on something other than the standard benchmarks.

I'm not affiliated, I just like his style and have found it handy. I know it's not very rigorous, but it's good enough for me and I've found his examples to pretty closely match the results I see in real life.

Claude 4 Opus: https://youtu.be/J7omabtqnBM?t=193

Qwen 3.6 35B A3B: https://youtu.be/gVU-DQeqkI0?t=215

Qwen 3.6 produced far more working functionality than Claude 4 Opus did.

Obviously, just one test of a single one-shot prompt of a silly toy OS, but yeah, this particular test shows Qwen 3.6 running locally dramatically outperforming Claude 4 Opus, which was a frontier model a year ago.

Will this trend continue? Who knows. Both the frontier and local model will probably continue to get better. Which one will hit the top of the S-curve first? Hard to say, really. But what you can do right now locally is better than what you could do a year ago on the frontier, and lots of people were already using it pretty heavily a year ago.

Hoever, November is when most folks agree that the frontier models got good enough for much of their work. Local models aren't quite there yet (where by "local" I mean "can run at reasonable speed and quant on a system less that $10,000 with today's RAM and GPU prices"). The biggest open weights models are getting there, but those require something like an 8x H100 server to reasonably run.

It's likely that there will always be a gap between frontier and local if you're comparing models at the same time, you can just do a lot more with terabytes of HBM than gigabytes of DDR. But will local models get good enough to be usable for useful work? For many folks, they already are.

For web development (or anything else with an extreme amount of training data) it's number one for sure. You can't beat it at its costs. US companies will not be able to compete on a competitive market, which is why they rely on so much US government protection + corporate welfare.

But it is still available on Google Vertex according to OpenRouter (though it's possible that info is just out of date, it's currently quoting 3tps which is unusably slow): https://openrouter.ai/anthropic/claude-opus-4

OpenAI was offering 2x usage at one point and I still used opus just because it's so much more effective.

Anthropic has been releasing models named Opus since 2024 with Claude 3 Opus.

Opus has gotten vastly more capable since then.

Local model far surpass Opus 3. They even surpass Opus 4 on most benchmarks.

Sure, if you compare to the latest Opus 4.8 or even 4.6, they're not there yet. But there's a huge difference in performance between 4 and 4.8.

When I colloquially say Opus level I really mean Opus 4.5 or later

More and more specialized and ultra-performant chips are going to flood the consumer market. Especially once new hardware foundries will start producing (well if we don't die from WW3 in the interval).

In 10 years from now, when even basic computers will have 128 GB of memory, and phones will have super optimized tuned models, then what will be the point of Anthropic ?

Just use Gemma/Gemini/Siri or whatever.

Pornography and uncensored models is also pushing toward local models.

It's not like needs of people grows exponentially, the needs follow an asymptote instead (they are capped).

The real revolution is offline robots and self-driving cars, but LLMs are already quite maxed.

For programmers, now, what Anthropic offers is like 3% improvement on a known test (like this pelican riding a bicycle), or on questions leaked from benchmark insiders.

It's ok but not like revolutionary (Fable was better but it was unusable, easy 20 minutes per one prompt due to overthinking).

https://ikyle.me/blog/2026/how-to-setup-a-local-coding-agent...

One thing I did change was the context length to 256k rather than 64k.

I've read a bit on what the various components are. What I don't see in your comment is what you're using to run your model locally. Ollama?

And then also, sometimes the tool call errors are because of something like a file was changed out from under it; the larger model is probably going to do a better job of figuring that out and fixing it up.

Finally, in Pi, you can always just use the /tree command to skip back to before a series of failed tool calls, with a summary if you want to let the model know what happened. The Pi /tree command is pretty powerful in managing your context

I'll experiment more with the effectiveness of AGENTS.md rules for local Pi agents. I feel like smaller (local) LLMs just lack in attentiveness to elements in the context window, like precise instructions, compared to e.g. Claude models.

Full octane isn't gonna fit on much of anything south of a 128GB machine once adding KV cache.

[1]: https://huggingface.co/unsloth/Qwen3.6-35B-A3B-MTP-GGUF

We truly live in the dumbest timeline.

You are paying for the extra power draw.

First I tried OpenCode but they unexpectedly make external requests (!) even when using Ollama (I noticed when Ollama wasn't properly connected and I still got a title generated).

So I settled for Pi, but I strongly disliked the idea that the agent could, at any point, decide to delete files or exfiltrate .env secrets. So I created Picosa (https://github.com/GreenpantsDeveloper/Picosa), containerizing and sandboxing Pi, with firewall rules such that it could only ever reach the local network (for Ollama), scoped by just the current working directory, and nothing else. Combined with Qwen3.6:35b, it works surprisingly well, and I could ask it to improve itself when run on its own repository.