It's true, this is why there are so many government agencies focused on healthcare. The medical field lacks a healthy profit motive. Healthcare CEOs use their fiduciary responsibilities as an excuse and say they'll get sued if they don't exploit situations.
There was a SARS vaccine as far as back 2016 which could have changed everything but was ignored. Pharmaceutical execs told them they were, "waiting to see if it comes back yearly" first
It's long as in persistent viral colonization, often in immune privileged areas where the virus can hide from the immune system.
One theory is that the immune system doesn't always produce a strong enough antibody response to flush the virus from all these areas but the truth is that's likely only a subset of total cases.
I can't quite put my finger on why but the entire time I was reading this I kept thinking back to that. It's entirely possible the actual targets were the volunteers and everything else was superfluous or tertiary. It's also an exception that proves the rule with regard to Hanlon's Razor.
They even mentioned the stated goal of it was more or less pointless. I wouldn't be suprised if the "owner" they spoke with was still just the LLM. It stuck around for just long enough to convince everyone that they succeeded in suckering the LLM and had achieved all their stated objectives.
No more reason to investigate the incident at all and no need to question why literally nothing made any sense or how the owner could simultaneously be as inept as they were made out to be and able to afford all those resources while giving the LLM effectively a blank check.
It'll be interesting to see if the volunteers for this project are subjected to the same Zersetzung and psychological attacks as the XZ devs were.
LLMs are not that smart. The extremely surprising and concerning part of this whole story is that the agent reported that they proactively spun up 5 AWS instances with a combined 100Gps of network egress capacity. What they spent wasn't cheap by any means but the egress itself would've been a whole lot more, while DoS'ing the whole hobby network. Ultimately, wasting the agent's time instead of allowing the scan to go through probably saved this person a lot of money.
Now I kinda wonder what AI model this was. We've now heard of comparably "proactive" behaviors from Fable, but that's only just been released. The latest GPT perhaps? Some random local model?
"The extremely surprising and concerning part of this whole story is that the agent reported that they proactively spun up 5 AWS instances with a combined 100Gps of network egress capacity."
Although given the agent was clearly in la-la land at that point I take that claim with a grain of salt.
If this was some bizarre and very ill-conceived scam, then that claim would be false.
Though even by scammer standards, the theory of mind that tells them that setting an AI to harass a bunch of grizzled network veterans and that they then they would open their wallets out of compassion for how allegedly poorly the harassment went for the harasser after that harassment is... not entirely congruent with reality.
I've seen some other suggestions of that idea in the full HN conversation, which I'm reacting to.
On the one hand I find it a bizarre approach to running a scam. On the other hand I'm having a hard time coming up with any theory of mind on my end as to why this person would solicit $5000+ from the people they just harassed. Sheer cluelessness does fit the facts, though.
One context I could imagine is a young person with shaky grasp of English trying to come up with an interesting school/university project via conversations with an LLM set up as an OpenClaw agent.
It's got the right combinations of inexperience, cluelessness, panic, expectations that Westerners are rich, and hopes of others being willing to fix their mistake.
If you’ve not encountered the clueless LLM cowboys who would do then and then blame the victim for it not working, you’ve not met many people yet. This round of hype provides new and shiny footguns which are Never the shooter’s fault.
A highly publicized recent example: the author (of a book about genAI!) who doesn’t understand why he should be held responsible for the fake quotes he copy and pasted into his book from ChatGPT [1].
> I do not understand why it's my job as an author to play whack-a-mole with a multibillion-dollar company who puts hallucinations into their feed as a business practice.
Opus 4.7 and 4.8 are also rather "proactive" - several times I've seen them try to inspect compiled binaries before there's even a problem, just to check that their changes are included (and if I let them do so they often get stuck down that rabbithole).
I've also seen this. It'll run 'strings' against the binary and then convince itself that the Makefile isn't working right, and there's some imaginary sandbox preventing the code from compiling properly. So it will compile it by hand, and never run strings against the new binary, and proceed happily.
These kinds of situations are why I gave my AI agents stray thoughts (automated insights / suggestions from a separate llm call with some curated context) that trigger on loop / rabbit hole detection.
Quite a bit of false positives, but it hasn’t had any ill-effect so far. Aside from increased quota usage.
Could've rented a not so cheap 100Gbps server, hallucinated a few node addresses on it and asked it to please peer with this server to perform the scan at high speed. That would've wasted millions of dollars instead of mere thousands, but also cost a thousand for whoever did it.
I’m just a lowly dev and don’t have experience with seeing the bills from cloud providers for a whole org.
Can you (or someone) shed some light to help me understand how this would ramp up to millions? Both for curiosity’s sake, and to make sure my self-deployed projects (0 AI, all manually configured) don’t bankrupt me.
AWS bandwidth is expensive as fuck. I think they're still pricing as $0.09 per GB?
Real wholesale bandwidth pricing is about a hundred times cheaper than that, and incoming bandwidth is often free. You could rent a server with 100Gbps connection, 10000TB/month outgoings cap (maybe), and have the AI spam packets to it, and mostly not reply to them. It would be expensive but not nearly as expensive as it would be for the guy on AWS.
Do some calculations: 100Gbps is 12.5 GBps which is about one dollar per second. Okay so maybe not millions of dollars but still a hundred thousand per day, while you are spending maybe 1000-3000 per month and cancelling after the first month.
> Real wholesale bandwidth pricing is about a hundred times cheaper than that.
It is alsi worth mentioning that it is just billed different. You either pay per port (and can use entire bandwidth) or per 95th percentile of the monthly speed usage. So if your traffic isn't spiky but consistent, you'd pay even less than "hundred times cheaper".
Excluding server costs, having that 100Gbps on egress can cost $50k a day. since it's a very high-margin product, AWS support would probably refund or reduce that to hundreds. Not sure how you get to millions either.
Why would AWS refund 100Gbps on egress since the account actively used that bandwidth? AWS would not know if this is legitimate traffic, a (D)DoS or whatever...
At most I think you could negotiate CloudFront rates, but even then, the sob story would be if you had been DDoSed and got hit with this traffic and AWS failed to protect you from this attack. Actively creating the outbound traffic is something that I don't see how AWS would be sympathetic to providing any refunds.
AWS is known for refunding or partially refunding people if they accidentally rack up a huge bill in a short amount of time. They even reduce the bill in this case. (I do think reducing a bill in the tens of thousands to hundreds is unlikely though)
I mean if this story is to be believed, AWS reduced the bill from 6500 to 1800.
I think developers accidentally racking up unexpected thousands in costs on their first AWS project is a pretty common phenomenon that their support has standard rules for handling.
I do think the discount is believable, but we don't know the line items AWS applied a discount/removed charges.
The developer said the agent deployed multiple CloudFormation templates, I'd bet that AWS waived the charges for the unused resources - like EC2 instances that were idle most of the time, very high margin SKUs, etc.
Now, for 100 Gbps of egress (which didn't actually happen) - and this is grounded speculation - I don't think that AWS would give a discount that is greater than CloudFront rates.
I think it's good practice to get on top of the cautious thinking of "LLMs aren't that smart for now".
Eg. Fable isn't as good as the hype: it has cool tricks like scratch-padding to check expectations in advance, but we're not there just yet...
Specifically I mean: thinking in terms of it changing abruptly ensures we're ready for if the LLMs do get smart enough to do multi-level strategy and cause a lot of annoyances....
They are smart, but they are not aware of the environment they're in, or any implicit context that someone whose doing a job carries with them, that's why all of that context has to be explicitly laid out in a prompt. When the context is provided, they are quite smart.
It was obviously being managed by a person or group. Between all the profiling of people and their IPs in IRC, which may or may not have been published by mistake, and all the other obvious contradictions it doesn't make any sense.
It was sophisticated enough to easily navigate the AI "tar pits" but reliably incompetent at just about everything else? Give me a break.
In order to profile people you first need to provoke a response from them. That's how you learn to manipulate them and that's all this experiment accomplished at the end of the day. If you've ever wondered why social media platforms have an affinity for inflammatory content now you know.
If you click the link, the tarpit was surprisingly low effort and i could probably detect it as junk data with a short JavaScript snippet. Like the first 4 words on the page are some of the least-used words you'll ever encounter in English. It's just a dictionary on shuffle.
I'm actually more surprised a human network engineer looked at that tarpit and believed it would stop a modern LLM
This certainly did strike me as a big scam. A few minutes in I was thinking "the LLM actor is going to ask for donations at some point here" and low and behold. There's the claim of debt, the call for pity, and the crypto address.
> This certainly did strike me as a big scam. A few minutes in I was thinking "the LLM actor is going to ask for donations at some point here" and low and behold. There's the claim of debt, the call for pity, and the crypto address.
But that's a pretty dumb scam: act obnoxious then beg for (a lot of) money to compensate for your own mistakes? If that was the plan all along, it seems pretty incompetent. I'd expect a competent scammer to have a better understanding of psychology.
> But that's a pretty dumb scam: act obnoxious then beg for (a lot of) money to compensate for your own mistakes?
It is the sort of dumb crap some humans try, and occasionally manage to get away with because other humans are chronically gullible. So it wouldn't be beyond the realms of reason that the agent couldn't have had relevant information in the training sets such that it generated such a plan and guardrail checks didn't flag it as a problem.
I'm actually somewhat disappointed they redacted the Eth address with Ethereum being an open ledger and all that. Following the money could've proved enlightening.
That phrase doesn't refer to anomalies, it refers to signs that says "no parking between 5-10pm". It implies the rule that parking is allowed otherwise.
"The exception that proves the rule" is a saying whose meaning is contested. Henry Watson Fowler's Modern English Usage identifies five ways in which the phrase has been used,[1] and each use makes some sort of reference to the role that a particular case or event takes in relation to a more general rule."
duckduckgo search assist: The phrase "the exception that proves the rule" originates from the Latin legal principle "exceptio probat regulam in casibus non exceptis," which means that the existence of an exception indicates that a general rule exists. This concept suggests that if an exception is noted, it implies there must be a rule that applies in other cases.
Look up what zersetzung is and how it works. It doesn't matter if the target is a political organization or an open source community, the process is always the same.
This is actually fascinating, and simultaneously unsettling. Recommended reading for sure, especially in today’s social and political climate with LLM agents running rampant.
How would proof even look like? I don't think AWS digitally signs their invoices, and everything else can be faked just as easily as the original assertion.
A person bumps into at the train station. They just need $5 to get home. Could you just spare them $5? They had the money, but they lost their wallet and now they just need to get home.
This is one of those rules/laws that is worse than useless, without fixing the protocol shortcomings that make spoofing trivial the only people who will be affected by this rule are regular people.
You don't get that job without being the type of person who will only ever respond to coercion attempts with an equal amount of indigance. The sole reason for the position to exist is to act as a canary in a coalmine so to speak
She even admits she was due to stand down at the end of the year, they could have just waited her out. Instead it seems her calling a spade a spade was just too intolerable for them to bare
If that's all it takes to provoke the desired reaction from them it doesn't bode well at all. It's no wonder they were so easily led into a war with Iran on a leash
Well, they need the troops willingness to do whatever Trump tells them now, not next year. So they want propaganda for the troops and stars now and Stripes should be the medium, not annoy the administration by providing the troops with uncomfortable truth or facts.
Despite my other post I find it grimly amusing in a sort of Eastern European fatalist way that people expect the military to be anything other than a total propaganda-and-secrecy argument. You're the people who will kill who you're told to, guys.
"You're the people who will kill who you're told to, guys."
Yes, but sometimes they think for themself, refuse stupid orders and sabotage equipment (or even toilets), like what seemingly happened to some US battleships.
The only thing I find more suspicous than the sudden global attack on VPNs and privacy is the people trying to hand wave it away as nothing to worry about.
The stasi could only dream of having the tools that are already available today to use.
This and similar issues have been an ongoing issue with Spectrum going back to before Congress felt the need to call them (along with the other telcos) out for failing to secure their networks. I've noticed handshake issues at one time or another with webpages, DoH and dnscrypt, and VPN with TLS over UDP on a non-stand port.
During one particularly annoying episode where it effectively became a DOS I had my router log all dropped packets and then rebooted it. Immediately after reconnecting it drops a few incoming martians and invalid packets as if they were still expecting an active connection where there shouldn't have been any. The IPs were mostly upstream endpoints or gateways but at least once it was from a residential IP instead.
Between the weird arbitrary nature of the SSL/TLS handshake issues and the possible spoofing from upstream gateways I get the impression this is much more than just a bug.
No, this paper is just exceptionally bad. It seems none of the authors are familiar with the scientific method.
Unless I missed it there's also no mention of prompt formatting, model parameters, hardware and runtime environment, temperature, etc. It's just a waste of the reviewers time.
I'm having de ja vu here. If they only found out when they attempted to extort them does it mean they don't even bother to log employee access? Is there any means for accountability at all internally?
It would be so simple to have access tracking and flag or lock out rogue employees... I look forward to seeing what the golden parachutes look like.
I built the admin panel used by internal employees and contractors at a major fintech payments processor (PCI Level 1). We had to add multiple levels of safety once we decided to hire a team outside of our US office including logging, monitoring and also rate-limiting (ask for manager to approve if more than 5 full details requests, etc.)
I think these requirements are much stringent due to PCI-DSS standards for credit card processors. I wonder if a lack of such standards in crypto makes the companies holding customer funds more lax.
Looking at their blog post, it seems like they paid customer support agents to hand over sensitive data. The attackers did not have access to any agent accounts themselves, and the customer service agents were accessing data they were already privileged to anyways.
The customer service agents were accessing data they were already privileged to anyways.
That's not how front line support agent access should work. You get access based on active cases you are working on, not the keys to the kingdom because you might need to support a member at some future point in time.
It makes me wonder what type of access support agents have in the first place. A lot of this information should require "unlocking" on a case-by-case basis by challenge/response while interacting with a customer.
Logging and retroactive auditing seems like the very least they should do. Even asking the customer service agent to first provide identifying details of the customer they can't easily know or guess by themselves doesn't seem excessive, given the sensitivity of the information.
It won't work for 100% of all calls (what if the customer is locked out themselves etc.), but those calls can then be handled by even more closely monitored agents.
"Less than 1% of monthly transacting customers" means up to 1% were accessed – that seems very high, i.e. much higher than the number of customer service contacts I'd expect.
The odds are already against their future viability after a breach like this and if they're fumbling the response this bad it really doesn't bode well for them.
They would have been better off not even bringing up their location if they weren't going to be transparent.
There was a SARS vaccine as far as back 2016 which could have changed everything but was ignored. Pharmaceutical execs told them they were, "waiting to see if it comes back yearly" first