It’s been wild watching smart people have their brain cooked on a constant diet of X and LLMs in the past two years.
Models are excellent at code but they are not people managers. My guess this will ultimately lead a bunch of companies to revert to average returns or fold.
Yeah, models aren't good at code any more than they are good at people. They turn it out very fast, but that isn't actually useful when the code needs to be carefully reviewed to find the land mines and questionable design decisions.
I always got voted down when I posted the evaluation of the parent articles I got from my Ouija board. I just want to help people understand whether they should just reject bad articles, without having to bother reading them.
I'm moving on to evaluating articles with a modified lie detector test and tarot cards, I'm sure that'll help my credibility and give my public rejections more authority.
Do you have evidence Pangram is unreliable? There are independent evaluations [1, 2] showing it works, and it's getting used more and more scientific papers. Have you used it or evaluated it yourself? What do you think these other evaluators are doing or getting wrong?
I don't doubt that those detectors are generally correct. Pangram seems to be particularly accurate. I see independent evaluations ranging from 97% accurate to over 99%. Frankly, I somewhat doubt those numbers, but I do agree that LLM usage can be fairly accurately detected.
But procedurally, there are huge issues involved with automated tools used to harm other people. You are one of the 0.5% percent of people whose article was flagged as LLM-generated when it wasn't, one of the false positives. What do you do? Argue? The accusers will claim that you're 99.5% likely to be lying.
It's the same issue we have with automated customer service, automated insurance claims, and so forth. It is usually correct, and terrifically unjust when it fails... at which point there is no recourse. In a perverse sense, its accuracy can be a drawback, because if the false positive rate low enough, nobody is going to believe you when you're falsely accused. And people will be falsely accused.
I think it's ironic that it seems like it capitalizes on the same flaw that most LLM-posting does... "Chat GPT is usually right, I'm going with it." You shouldn't post an LLM article without independently validating its claims, so that there is a responsible person in the loop. The same is true for rejections and accusations, but more so, because they're more damaging.
> Also, my main gripe with parquet (single table per file) is not even addressed, so, also the name is a bit hyped up.
This is really more of an expectation that has been put on file formats by the query engines. Spark/Datafusion/DuckDB wouldn't really know what to do with a multi-table file.
> Parquet is unfortunately very good just by virtue of being first, and so widely supported
IMO that is not how technology works. It is great that Parquet is so good at a lot of things, but that does not mean just because it came first that it deserves to be the only analytic file format forever.
> Its main result seems to be improved random access which, although certainly welcome, is not the point of columnar storage, as columnar storage was invented to exchange random access for something else: fast analytics
Fast analytics, as well as newer ML-shaped workloads, are inherently mix of batch scans and random access.
Some of the authors of F3 previously authored another paper that goes into the details of the shortcomings of Parquet
All of the newer formats that popped up recently (Vortex, Lance, F3 now) have been working on solving the problems outlined in that paper.
Lance has some interesting ideas, Vortex focuses on extensibility and performance by replacing all of Parquet's black-box encoders with fully transparent encodings. This solves the tradeoff between bulk and element decoding, allowing you to have efficient full scans and really fast random access.
Disclaimer: I work on Vortex, so a lot of these questions about "what is the point of building a new format" are things that I have grappled with myself.
Sure it would, you can attach a multi-table sqlite database in duckdb
> that does not mean just because it came first
I agree with most of your points, I am not stating my opinion but my observations. I am the target audience here, I want to use this, but I don't really care too much about the file format itself, at least not as much as I care about the data inside.
That means access, which means compatibility with my tooling.
FWIW I think if you are just doing pure analytics and nothing else, Parquet will probably continue to do the job for you just fine, and you don't need to touch your workloads at all.
These new formats I think will find a niche where people aren't just running Spark jobs, but doing lots of systems building over large tables. If you're building a PB-scale data warehouse, you care a lot about the file format b/c it is a big factor in your performance curve, and you're willing to ship new experimental codecs in response to new datatypes you want to support that the system wasn't originally designed for, or you want to use a newly invented compressor.
> > Parquet is unfortunately very good just by virtue of being first, and so widely supported
> IMO that is not how technology works. It is great that Parquet is so good at a lot of things, but that does not mean just because it came first that it deserves to be the only analytic file format forever.
I think you and vouwfietsman (https://news.ycombinator.com/item?id=48649412) are actually saying the same thing in different words—I think their "unfortunately" means "it is unfortunate that, by virtue of coming first, this now has a support lead that will make it difficult for anyone else to catch up."
Yeah that point about "random access is not the point of columnar formats" fell flat for me for this same reason. Almost since the first day I started using columnar data, I've been interested in solutions that strike this balance between batch and random access. This comes up all the time (in my experience) in data science / ML, where we have use cases for both access patterns against the same data.
So I'm with you, I'm very unconvinced that parquet (and the various things that are parquet or essentially-parquet under the hood) are the end of the line here.
> “I never got my driver's license, and I rely on Waymo to commute to an office every day," said Sarah Paige Roland, a Waymo rider in Phoenix. "I get privacy, time back, a safe ride, and I'm not obligated to talk to someone that I don't want to talk to."
I recognize that this is a luxury product but I kind of laughed out loud at this testimonial. The amount of privilege you need to have to grow up and live in *Arizona* without ever learning how to drive is insane.
I appreciate this comment immensely - too many people seem to mindlessly assume that every other person shares their own situations, and it could not be less true.
So what you're saying is this hypothetically disabled person who is physically incapable of driving chose NOT to mention that, but instead chose to provide "I never got my drivers license" as their reason?
And you're asking someone to consider this because I presume you think this is a likely enough scenario to consider?
I think I'm missing the excitement. This is an artist's rendering of a supposed massive orb in the sky? I am more impressed by the actual UAV footage that has been released previously.
The entire site is meant to distract you from asking where are the other files they’ve been required by law to disclose but have refused to. Mixing artist renderings with photography is just par for course MAGA conspiracy stuff.
If you can somehow get your hands on a dozen NVL72 racks and duct tape them together in such a way to rent them out as a service, you can make your money back in less than 2 years at current demand pricing. $50M is more than enough to get this going.
Removed means they were there before which means comparison to other maps means nothing. It's possible Apple never had them in the first place. It's completely unverifiable with the link or your links.
OSM is a foundational data layer for GIS. If you're building a mapping service, you're almost certainly using OSM augmented by satellite imagery and other sources to find population zones that OSM has not found yet.
If you look at the Apple Maps satellite layer, you see thousands of structures spread across the area.
It is a reasonable assumption that these population centers were labeled and Apple (or one of its data partners) has withdrawn the labels.
> If you look at the Apple Maps satellite layer, you see thousands of structures spread across the area.
and if stop fixating on South Lebanon and go to other places where israel is not invading and destroying villages, let's say Syria, you'll notice the exact same issue: https://imgur.com/8p1ANAZ
so your argument actually go against what you attempt to demonstrate.
now why is apple bad at maps in this area and possibly other areas elsewhere in the world is a different matter.
I don't know either but here is the answer from Wikipedia: "The main provider of map data is TomTom, but data is also supplied by Automotive Navigation Data, Getchee, Hexagon AB, IGN, Increment P, Intermap Technologies, LeadDog, MDA Information Systems, OpenStreetMap, and Waze."[1]
close tab