I learned from an electrical engineer that works on systems for the power grid that running the whole OS in CPU cache is a requirement for some of those systems. I guess that makes sense when you are working with things that move at the speed of electricity.
No, the system is probably using tightly coupled memory; the CPU has single cycle access to all of system memory, effectively making a cache redundant.
And the reason is likely "we are building something realtime, we can't afford variable latency".
For the electricity grid in today's world, that's over-engineering things. The 60Hz electricity grid doesn't really need anything being changed at more than 600Hz, and considering your CPU core runs at 3,000,000,000Hz, you can totally waste a lot of cycles before you start missing deadlines.
"This meeting is to announce we've settled on an architecture for the next gen grid optimization program: Electron UI with a Ruby on Rails app underneath the hood that calls to AWS Lambda for anything that's cpu-heavy. State is persisted to Firebase as serialized YAML, which should scale really well. And to our normal crowd of performance zealots, don't even start with your normal "latency" and "outages" griping. We're comfortable with this stack and besides, we can waste a _lot_ of cycles before we start missing deadlines!"
Even if your CPU cycle budget is large, designing a realtime system requires proving thet it won't be exceeded, by adding up the worst-case running time of the worst-case sequence of events.
Avoiding high latency memory gives a few orders of magnitude of margin, which could be needed to afford good functionality on a reasonably cheap CPU; it's not a mere useless improvement from "fast enough" to "as fast as possible".
In hard realtime systems, the problem is not "we have enough time to do the computation" but "we can prove the computation will run in the available time".
The complexity in modern mulit user operating systems (many processes, a run queue, spin locks) along with modern processors (multi tier memory architecture, micro-code, branch prediction, long stalls for memory access) mean it's very hard to prove that anything will run in finite time.
IMO the terms "hard realtime" and "soft realtime" are unnecessarily confusing to beginners. I think that "deterministically bounded latency" and "probabilistically bounded latency" are more descriptive. But they're long and difficult to type, so we'd inevitably call them DBL and PBL, and be back at confusion.
And yet, so much software on modern CPUs takes seconds rather than milliseconds to perform and output simple calculations. Especially with the mantra "developer time is more expensive than computing power", few developers nowadays have recent experience of optimizing programs for sub-ms reaction times. You could probably make it work with standard hardware. But since developers have to be retrained for that field anyway, why not use hardware that's battle-tested and gives you more leeway?
In addition, the advantage of optimizing at every point is that you can use hardware not designed at the limit of what's possible, likely giving you lower error rates.
