This is obviously a space we are very much interested in, so it is definitely nice to see approaches that attack the same problem. I haven't played with dbtcharts in depth yet but it looks very similar to DaC in principle, and also in the actual spec. I believe the industry definitely needs solutions like this to help get out of the legacy BI tools as Bİ is one of the biggest bottlenecks for AI adoption in large orgs.
Excited to see further competition in the space, nice launch!
I launched Epho a few weeks ago as an API like this but for all harnesses: https://epho.io
I built it primarily for ourselves: we are building an AI data engineer, and we need a way to run many of them in parallel securely. An API for this seemed like the most obvious path forward. It makes it trivial to bring agentic capabilities into any product surface without having to deal with sandboxes, reliability issues, compatibility problems, and more.
I think it also makes sense from OpenAI's perspective to do this, but also we did find ourselves needing to change models and harnesses quite a bit, which is why I think this needs to be a layer of its own above the labs. It also needs to be a layer above the sandboxes, since many of them are quite brittle.
Overall, I expect a lot of the agent implementations to move in this direction. I think this is a lot saner for engineers to implement and maintain, and it makes it trivial to build agentic stuff into products.
I've commented about this before, I think many LLM based apps nowadays are at risk of being replaced by a product straight from the labs once they prove to be successful.
We've seen this pattern with Apple making their own version of an app that was previously popular on the app store.
The labs are in a perfect position to do this - they have a bunch of data on what's being used and they have direct access to their own models/compute. If an external service is popular, it's relatively trivial for them to estimate how much additional profit they're leaving on the table.
If the usecase isn't far from their core business (and things like this absolutely aren't), with their size, why wouldn't they eat other people's lunches?
I wasn't expecting this, and I am pleasantly surprised. I think AWS has been a bad player in the data space and very much behind all the other warehouse providers for many years now. This could lead into interesting integrations of DuckDB on top of all the data that lives in S3 and provide a leaner, faster and a more modern alternative to Athena.
Feel like AWS got a lot of the concepts right with S3 Tables* ("serverless" Iceberg catalog on top of S3) ... would love, love to see a version of S3 Tables that lets you choose between Ducklake or Iceberg as the catalog.
* Cloudflare also seems to have gotten the same idea right with R2 Data Catalog
It seems like a relatively straightforward marketing article. I was pleasantly surprised to learn about Erathos though, nice product!
I am personally not a big fan of CDC in prod. Streaming data movement is generally prone to confusion, and it feeds into bad data patterns like hard deletes without any audit logs, no timestamps on updates or deletes, etc. which are usually the reason why batch loads cannot be utilized. They require a decent operational understanding of the underlying database, and have some gotchas like the Erathos folks mentioned in the article. We offer CDC both in our cloud platform, as well our open-source tools, but if I could, I would always pick an incremental batch load with a cursor value over a CDC connection.
I understand it is sometimes required due to organizational complexity or legacy database reasons, mine is just a personal preference.
> I would always pick an incremental batch load with a cursor value over a CDC connection.
What's the benefit vs. something like Postgres's logical replication for CDC? IMO, the hard part of CDC is maintaining consistency in the face of potential network issues or downstream slowdowns. One is forced to choose between scylla: generate excess trx logs if replication slows, and charybdis: lose consistency. I don't see how an open transaction helps here?
Hey! Glad to hear about Bruin, competition always pushes us to make our product better for our customers :)
Frankly, I am not a big fan of streaming either. But CDC and streaming are two different things. Streaming database replication is generally built on CDC, but you can use CDC with micro-batches too (which is what we do). I wrote about this in another article: https://www.erathos.com/en/blog/cursor-based-sync-vs-change-...
Imo, the key reason for offering CDC is that most databases aren't designed around "how can we extract data efficiently", therefore they often lack a reliable "updated_at" column for us to use as a cursor.
2) downstream consumers consuming a business-events table. Kind of like inverted event-sourcing pattern.
CDC is sound for sure but only in the lowest technically sense. Downstream consumers consuming business events gets you all the benefits of event sourcing(Namabilty, Replayability etc) while keeping relational guarantees and transactional safety.
One would think. Conceptually they are nice, practically it gets very dirty across different databases + all the infra around them. My dislike stems from the operational complexity and years of trauma around it.
I got excited for this not because this didn't exist before, but because Docker putting their weight on this would imply a broader adoption and better integration in the industry. I am sad that they are asking for a login here though, which doesn't make any sense to me.
Hi all, this is Burak, co-founder at Bruin. We built a new feature called data apps, and I recorded a quick video showing how I built the actual Scranton office of Dunder Mifflen from the show The Office to report on data warehouses.
For context: Bruin is a data platform that does ETL + BI. We do a lot of things open-source, and recently we launched Dashboard-as-Code: https://github.com/bruin-data/dac
The idea behind DaC was that using AI agents would require a more code-driven approach. That turned out to be directionally right, however it fell short in terms of unlocking even more value as people became more ambitious.
That's why we built data apps:
- These are React apps that connect to your database / data warehouse
- They work with all kinds of databases + external APIs
- They can make external requests in a secure way without the credentials being embedded into the app itself.
This allows building all kinds of interesting and capable UIs with them. The dashboards you build become interactive internal apps that works with your data.
I would love to hear what you all think about generating more flexible dashboards on top of warehouses and what could be different. I wanted to offer this as a Show HN for folks to try out but it is currently a bit too expensive to run this publicly for us, that's why we are initially enabling access to the feature only to a select set of customers.
Let me know if you have any thoughts or what would you imagine seeing in this space.
It is not obvious from the post but it seems like the allow list for the scripts supports whitelisting packages instead of a global setting. This should make it easier to maintain org-wise rules to allow scripts only for specific packages.
Is there a linter that could be used for scenarios like this to prevent unsafe default on package manager config?
Hi, this looks interesting, thanks for sharing. I am the builder of ingestr (https://github.com/bruin-data/ingestr), so I am very much in the same space.
I really like that you did this in Go, and I'll definitely dig a bit more into the source code to see how you tackled the CDC stuff, given that there is not many reliable CDC libraries in Go, and there are quite a few gotchas when it comes to doing CDC right. We also hand-rolled ours in ingestr, or I must say clanker-rolled, and we got quite a few things wrong in the first place.
Curious about the postgres-compatible query option: what's the usecase you have in mind there? My perception is that any org that would use Iceberg also has one or a few query engines in place, is this more for debugging stuff?
Hello, I checked ingestr repo, and it is in my bookmark. Small world.
Agree, CDC is like Death by a thousand cuts. I believe Debezium has a Java library.
My initial need was Postgres compatibilty. Wanted to give an endpoint that BI and dashboard teams can use to query as if they are querying a Postgres replica. Added more context here https://news.ycombinator.com/item?id=48350820
This is obviously a space we are very much interested in, so it is definitely nice to see approaches that attack the same problem. I haven't played with dbtcharts in depth yet but it looks very similar to DaC in principle, and also in the actual spec. I believe the industry definitely needs solutions like this to help get out of the legacy BI tools as Bİ is one of the biggest bottlenecks for AI adoption in large orgs.
Excited to see further competition in the space, nice launch!
reply