UNIX? You mean that pile of crap that replaced much better and saner systems, so that many decades later we still have the same stupid problems they solved in the 70s?

x86, which is basically crap?

Enterprise Java, which all Big Players use, that is a pile of bloat wasting computing resources, fueled by countless armies of code monkeys typing it dumb code that starts to work if there's enough of it, much like uranium gets dangerous if you put enough of it in one place?

Modern web development, with bullshit hipster fads, even more bloat than Enterprise Java, where half-life time of any library or framework is counted in months, if not weeks? Where people who have no understanding of what they're doing invent dumb solutions than then become de-facto standards (see e.g. template languages)?

The Lisp folks who rant about good old times and develop some software of various quality, but nobody even notices so you won't get to apply any of that in your job?

What exactly is the state of the art?

I think, you somewhat answered your own question, albeit indirectly and ex negativo. Widely ignoring what we already know or could have learnt, and pushing the next vanity-boost feature seems more attractive than learning from our experiences and drawing the consequences. And the people doing it always find rosy language that makes it seem like a heroic feat of progress, and an audience that falls for the BS.

There is no such thing as practical state of the art in our field for the problem being discussed (which is the software requirements capture/design/engineering side of things, not the software construction side of things), most of the so called "state of the art" focuses on a problem of very limited utility:

problem a) if I have a complete, consistent and fixed requirements, can you write code that provably implements it it? Sure. But very little software is developed under those conditions (avionics, safety critical stuff, some crypto stuff, etc)

problem b) code/methodologies to make writing easier/more concise/more predictably once as the specification is discovered. Nice, but it is optimizing the 10% of the task, whereas the 90%, the requirements discovery and requirements consistency management between requirements discovered amongst different branches of the problem is where the bulk of the time is spent.

Those are the "assume spherical cow" of our space. Interesting, but limited.

Or going with the bridge analogy that seems to be popular here, you're asked to build a bridge. Nobody tells you over what, what load it has to carry, under what weather conditions. So you don't know whether they want a rope bridge over a tiny stream or a multilane car/train combined suspension bridge. And as you ask for the requirements it is now a bridge over the Atlantic with a stop in Australia. And when you query the Australia part they confirm and casually mention that they might be planning on a stop in Pluto too and can you fit that in the schedule.

Anyway, you're absolutely right about "state of the art" in requirements capture / design / engineering, and this is the larger context of discussion. But I admit, my original post was an assertion that there's no sensible, practical "state of the art" on the construction side either - it's mostly either fads or working with lowest-common-denominator tools (basic Java). The situation seems to look better in embedded in particular, but that's either because you're really, really limited in tools there, or because I have very little experience there and only think it makes sense...

Maybe there is a reason why we seem to see the most impressive code work on "constrained" devices like the C64.

This is an often repeated and completely false assertion.

Just think about the number of deaths every year due to bad road and highway design - level rail crossings, highway exits and entries from the left (in North America).

The idea that civil engineering is any better than software is just nonsense.

Oh, and don't forget about the civil engineers at Fukushima who put the backup power systems in the same area as the reactors, guaranteeing that they would also be destroyed in any serious disaster.

And civil engineering is just one branch of engineering, there are many and all of them have to work with the same balance between available resources, time and other constraints and they do substantially better than your average software project.

A bridge collapsing is headline news, a computer program crashing is so normal that you would probably not even mention it to another person if it happened to you.

That's the main point that I would like to refute.

Given the regulatory barriers, enormous costs, and long planning timelines for most civil engineering projects, isn't it astonishing that bad designs impacting human lives are still being implemented.

If the process of civil engineering was so well understood, so carefully documented and supervised, and so rigorously taught to new engineers, why are many hundreds of people killed every year?

The uncomfortable answer is that the civil engineering process is very far from well understood - just like software.

Well, depending on context, country and corruption: those are the exceptions, not the rule and the engineers typically did their work as well as was possible given the constraints. They know what they know, and more importantly, they know what they do not know and they will engineer in safety factors.

Buildings and bridges collapsing, machinery exploding: those are the exceptions, not the rules.

As a rule, highways function, as a rule, bridges withstand their design loads and excess of those loads and so on.

As a rule: software is buggy, frequently crashes, hogs memory, is slow and has inconsistent user interfaces, updates will randomly break stuff and it is insecure to boot.

If you feel that software is on par with the rest of the engineering disciplines then I probably won't be able to convince you that it isn't.

But whoever designed this bridge and the foundations did a pretty good job of it:

https://www.youtube.com/watch?v=RAv-rYB5qrc

In contrast, if you look cross-eyed at your average piece of software it will misbehave in unpredictable ways.

I DO feel that software engineering is on par with other engineering disciplines - if writing software can be described as "engineering".

The important difference with software is that "the code is the design". The code is not the product.

The manufacturing, building and deployment steps in software are actually quite well understood, and has been mastered by most organizations and individuals.

If we substitute 'some' for 'most' then we're in agreement, unfortunately my experience to date does not give me the confidence required to subscribe to your version, but that could easily be local variation.

Yes, that's exactly the problem. So now we have an unsafe industry that is producing a terrible that powers a very large chunk of the worlds commerce. It's a house built on quicksand and our hope seems to lie in hoping that we can move to another house before this one collapses.

The areas in which this is most visible: security, reliability, maintainability, documentation and testing, and finally a complete lack of warranty for suitability from the various manufacturers/service providers.

Software is 'eating the world', but is it strong enough to support the world in the longer term if we continue down the road that we are on? I'm not bullish on that.

It is however extremely hard to imagine the world that would be, if we were to take a different route. What would be the effects of slower improvements. I assume (since I sometimes do) many people look at the software industry as narcissistic and self-agrandizing when "delivering expriences" and "solving global problems with scalable midware". But what third party industries would be held back if this one moved much slower? Could there be loss of medicinal advancements, monetary, safety, third world advancements?

But as you say, I'm also very hesitant on the stability if we can't find some middleground between the two.

I think the feeling of fresh and new development (before complexity sets in) is so compelling that it tends to drive us away from actual progress. After all, it is so much easier to launch yet another half-baked language, frame work or product than it is to pick up something existing and to really improve it or to update it in a way that it will last much longer. That's relatively thankless and anonymous work compared to slapping your handle on a new framework.

This is part of what makes good software hard: good software isn't sexy (think: erlang verus node.js, as just an example, I've tried very hard to keep this discussion brand and tech free but I feel that an example may help to illustrate what I'm getting at).

Security bug bounties are also an important advance in software engineering process.

I also think there is a growing interest in these issues outside the industry. For example, we are likely to have as our next President someone all too familiar of the risks of poorly secured email systems. Maybe this will be the impetus for a more serious focus on security, at least.

Bug bounties are not such a hot idea in general. There is the issue of entitlement on the white hat, disputes about severity levels, potential for blackmail, and the two engineers that are clearing the false positives. Better to put them in the pen test team.

Civil engineering has existed for more than 3,000 years; the romans were building cross-continental roads 2,000 years ago. The Chinese were building cross-continental defense structures 2,500 years ago.

I mean a lot of the problem is we generally spend most of our time thirty abstractions away from what is actually happening, and often times those abstractions leak. For goodness sakes few of us are even protected from cosmic rays changing bits in our ram, which happens fairly frequently.