Yes! Your modified version would stay the same, if I have an update which conflicts with your modified version you can simply select what to add to your modified version and then modify would take care of the rest.
nice pattern with the stored queries for writes. but who defines them? if the app author can create their own stored queries, the write restriction is basically honor system.
There are actually two types of stored query: regular and "trusted".
Any query you save is a regular query. It operates under the permissions of the viewer, and checks that the viewer has the necessary permissions - read access to the database, or more finely grained write access which checks the individual tables they will be writing to.
The problem with that is that it means you can't build an app which other, signed out or unprivileged users, can use.
So there's a second category: "trusted" queries. These are current only configurable by the site administrator who controls the Datasette deployment, as they go straight in the configuration file: https://docs.datasette.io/en/latest/sql_queries.html#trusted...
I'm planning to add a way for trusted users to create these through the UI via another permission, with a very strong UI warning to only use this feature if you understand the implications.
so the 11% miss rate - do users actually notice when the agent drops a memory? like if someone already said they tried X and the agent suggests it again.
It's been a lot of trial & error. A quick aside: running these tests/evals/call them what you will at scale has been fascinating to me. Going back and trawling through the logs has been like speed-running through hundreds of usability tests with people, full of the same types of "aha! Of course you'd try and do that, why didn't I think of that already?" moments of insight and inspiration.
Which is also how we've gone about working out how to improve the CLI. It's usually one or more of:
* rethinking the subcommands and hierarchy to something more obvious and aligned to the task
* providing clear documentation upfront (i.e, in the skills file)
* keeping help text concise, but not too concise. You can't assume the reader is already a power user and it's simply looking for a reminder/reference. So include usage examples for common use cases
* where possible on errors, suggest the likely commands the person meant.
* In general offer affordances on what likely next steps will be. This goes for help output, success, and errors.
> cli help text is usually massive
That doesn't have to be true.
> could eat a lot of the savings on retries
This doesn't have to be true either. You don't need to give the same full help output on every single error, once they've got it once they've got it. Also the size of the entire help output for most CLIs is generally insignificant compared to even just a couple of source files in most repos.
review agents have the same training biases as the one writing the code. you get 30 findings about error handling and edge cases, but wrong domain assumptions slip right through.
the bigger problem is agents defaulting to the broadest command possible. kubectl get -o yaml when a jsonpath query would give 1/50th the tokens. filtering after the fact works, but you're still paying for the round trip. better to teach the agent to ask narrow questions in the first place.
allowlisting breaks once the agent has messaging tools. you can deny all outbound from the agent, but if it can post to teams or slack or email, link previews will fetch whatever URL the injection puts in. messaging is usually the first tool anyone adds to an enterprise agent so you end up with strict network controls that don't actually prevent anything.
