My main use case is for example in my market inteligence system, there are multiple agents, they scrap, they curate, they enrich but my output was missing big part of informations and i saw in logs that i scraped a lot of data but on the output there was almost nothing, so i needed to go inside the system to see what each agent exactly hand off, what tools did agents used and i found 2 things. First was that one agent didnt know what he got from one tool so he returned nothing and second was that plan agent didnt come up with correct plan to invoke whole system. So that was my use case although there is a lot of bugs etc it still helped me so i decided to publish it on github so anyone can like at least read about my problem and creat his own tool

How about OpenAI's agent SDK, do they provide what you envision? They say: "OpenAI's SDK offers built-in tracing that logs agent interactions, tool calls, and handoffs"

Yes and that tracing is the input, not the competition. Detective ingests OTEL spans including ones the Agents SDK exports.

Yeah thats the real failure mode and it took me a while to name it too. A threshold isnt a test, it is a coordinate system that assumes a fixed origin. When the judge moves the filter keeps passing but it is selecting a different population and nothing throws because no invariant anywhere names that population. So i define it as: quality broke when an observable moved without a change that authorizes it. Thanks for trying it out i am really happy for feed back :D

labels here too, about 100 articles with the band they should get. what i still cannot see is a move with no change to blame.

Yes but i am thinking about all datacenters, AI datacenters are same as meta datacenters or netflix etc. Everyone wants theyr private netflix account to watch films and movies and post nonsense on socials but AI that actually helps is the problem... sure. I think that eco maniacs and everyone who is against AI should first consider not using any aps/servers and then they can talk

My framework can discover where your agents failed and why, you can easily debug your agents with help of agent detective. It is in beta so i am still testing it by using it at my work and it already can help. Although i always find something that failed it still is helping me with understanding what is wrong with my agent. It is python library with CLI, in update 0.4.0 i prepared langchain and langgraph adapter for better implementation.

What is your oppinion?