People appear to be missing the point that _the website_ is literally the smallest part of the project.
While it's easy for people to crap on things like js-minifying, cache control, etc - the hard reality is the stuff that is the hardest is so hard that most people have never done anything even remotely close.
Having worked on 10m+ loc codebases, life gets very hard. Dynamic languages tend to be unworkable. Module control, strong interfaces, and a lot of division of responsibility becomes the order of the day. And just figuring out how to build the thing becomes it's own team.
For those who are just thinking "so, just slap together some JSON/web services and call it a day" - I invite you to research EDI and then report back here. Assuming you're still alive.
Taking on EDI projects are the the most difficult jobs I've had as a dev or as an admin. Interacting with someone else's database schema is like trying to put yourself in its designer's head, and it's on a good day when you have a standard protocol or database format. I've mostly worked in the auto-industry as end-user support and currently employed by a wholesaler - the connection between parts manufacturers overseas and dealerships, body shops and others. The business is more complicated than that, but the auto industry is one with many standards, lots of vendor lock in, and proprietary formats and coding schemes for everything.
With EDI you don't get to dictate standards, you can't demand JSON/web interfaces, you usually have to deal with flat text files (good luck) or schemaless XML (you're doomed) if that. Some vendors just provide XLS files to an FTP endpoint and call it a day. Some vendors send pricing information exclusively in PDF format over email. The manufacturers never maintain a format for more than a month, and downstream consumers of data can never decide how they want it.
Finally, you'd think that once you get all of that mess sorted out you're golden. The only problem is that the auto industry is the land that time and UPC codes forgot, and there are multiple inconsistent standards for labeling parts, and data services that claim to have a mapping between them are selling snake oil at prices several orders of magnitude higher than what you'd think fair.
Also no one in this industry believes in using software developed in the last ten years and I think I'll be near retirement age before my successors will first receive vendor data in JSON over HTTP/1.0.
Given the complexity of the system, the most shocking thing to me is that how little time they had to actually build it. And from what's been made public there was virtually no time for testing (load testing was certainly absent). It should been "turned on" months before the public go-live to iron out all of these issues.
That's one thing I never understood. The law was passed 3.5 years before they had to go live. Seems like they should have had plenty of time but it seems like almost all of the coding was done in the past year with little testing along the way.
If only there was a way to remove the hundreds of private insurance companies they have to interface with. I assume they each have their own flavor of EDI, greatly adding to the complexity of this thing. (I have had to write interop EDI stuff with some insurance companies a while back and iirc they were snowflakes)
A friend analyzed the page and noted that they didn't minify or combine anything.
"One obvious reason is that they're serving unminified and unconcatenated Javascript and CSS files. A single page load causes 92 requests to the server, and they're not using a CDN for any of that.
They're also not setting the Cache-Control and Expires headers, so the browser isn't caching many of those files, including on 80k file full of response strings.
For the non-technical, this means the site is essentially killing itself. An optimized and properly tuned production website would load one (1) HTML document, one (1) CSS document, and one (1!) JS document, they're loading 92. "
While that is an egregious amount of requests, it's not the reason the site is broken. Based on my experience as a user and poking around quickly in Firebug, the issues are largely server-side.
Yeah, people bitch about client-side because it's what they can see. But they are mostly just dumb inefficiencies that are easily fixed. (And if you were doing a sane rollout, you wouldn't be worrying about this stuff because you only have 10K users at once. Save the minifying for later.)
True, it is just an additional data point. You have to wonder, if they can't minify their JS or even run the pages through the Yahoo YSlow tool, what is the quality of the backend?
I seriously doubt that the static content performance of NGinx (or whatever) has anything to do with performance issues. They're probably spending more money on bandwidth than they need to, and offer a slightly degraded user experience, especially for people on slower connections, but otherwise...
Things may have changed since October 1st, but on second page load of the home-page, I'm seeing the majority of content as either cached or served by CDN, a relatively small functional payload compared to a lot of HN darlings, and an 79ms latency for the main request.
None of that is really alarming IMO.
Even so, half these assets are trackers and 3rd party resources that would probably be CDNed externally. If storage IO and bandwidth isn't your limit (and internally the URLs could easily be routed to different servers entirely as well as caching proxies), then it's pretty plausible that there's practically nil overall performance impact as far as service availability goes.
Also, I think the whole "you should only use one of each type of asset" philosophy is debatable at least.
It is, however more than just a smell: a front end of such an egregious turditude doesn't just uninspire confidence in any of the system - these poor practices demonstrate that whoever is in charge of the checks and balances hasn't done their job.
Sorry, but you obviously have no idea what you are talking about when you call it, "a website"
Or is google just a website? Is yahoo.com just a website? Did you look at the infograph?
How much do you think it will take to talk to 5 agencies and 300 insurance providers that have 4500+ health plans, and get them to agree/adhere, understand their EXISTING system and build a site?
I'm not justifying the expense, but let's not trivialize the task at hand and call it just a website.
Presumably the interfacing with insurance providers is done on the backend (not included in the $295M). It's possible that QSSI took all the content from the providers and standardized it for display in the front, near trivializing what you mention. Just a hunch, and not saying it isn't complex, just not $295M complex.
CGI Federal literally ripped off the government, screwed the American people badly, and "stole" hundreds of millions. They should never be allowed to work on another project again and should be required to refund the government. The project managers should be fired and publicly humiliated.
I STILL cannot even login to the website. The site has said "The system is down at the moment." for the past two weeks.
The only reasonable explanation that I can imagine is sabotage by someone who didn't want Obamacare to succeed, the same thing that happened in Congress recently.
> The only reasonable explanation I can imagine is sabotage by someone who didn't want Obamacare to succeed
It depends how you define "success". While most people assume success as a well-oiled healthcare machine that is affordable and cares about you then yes, they have failed. However, if you have done your homework on Obama and Democratic party (in biggest part), you will understand that they already greatly succeed - that is they shoved a huge tax (that over 10 years grow to 10% of your income, starting with 1% next year) in american's throat. While congress pass the law and Supreme Court upheld it, rest is just a statistic noise. Who care congress has 11% approval? Who care Obama has vanishing approval rate and his administration cannot catch up with any "phoney" scandals they getting into every month it seems? Its all irrelevant. After all, they did not hire 16,000 new physicians; they hired 16,000 new IRS workers that will be responsible for collecting what you lawfully owe. That's all! Its all about the taxes; its never been about you or your health. The only way Obama ACA would be more successful is that every single american throw a towel and say "screw it, I will pay the fine, this is too complicated!". That would be the best scenario for the Obama and the gov.
> the same thing that happened in Congress recently.
I think you mean the shutdown. It was so pathethic that it is funny. Imagine situation when you drive to work every day. And then every weekend you go out and drink and drive. The cop pulled you over twice but he let you go with a warning. You keep drinking and driving. Then one day he pulled you over, boom! DUI, license revoked. Then you go to your friends and family and you complain that because of this cop now you cannot do your work, because you drove to the office every day. If you agree with this reasoning then yes Reps were responsible for shutting down the Gov and not letting Congress do their work.
I think you misunderstand what goal government contractors, and the government officials that manage their contracts, are working toward. It's not a full-featured, working product. It's money. The contractors want more contract money. The government workers want visibility and job security. These things "work" in the contracting system by having contractors that play the same game the government workers play: which is needlessly and pointlessly extending contracts, mainly by fucking things up while fulfilling the technical letter of the contract.
Thats just the contract cost.... Numbers are estimated near $340M and the GAO just came out with current costs sitting at $320M but harder liners are coming out with numbers near $620M. By the time its done and fixed, its estimated at $1B.
10 web developers couldn't even remotely come close to building/implementing a system like this. This is not a web development project but an enterprise integration project. What you see as "the website" is just the tip of the ice berg. It's like saying that a car is just a steering wheel and four wheels because that's all you "see" but you don't know what's going on "under the hood".
However, I do agree that the costs for this are extravagant and there are certainly signs of "government milking" visible here.
Seriously ? Give me specifics where exactly do you think $300 million go? Look at the slides its not for the complete system but the "front end" of it. Most of the software they use is Open Source, Cloud services/hardware is dirt cheap and Software Developer salaries are not that high.
The problem isnt the part that you can see, the Front-End. The problem is the legacy systems on the Back-End that need to talk to each other. And if one of these requests time out, the whole request fails, doesnt matter if the 5 systems before responded it has to go through the process again.
The web part is just a tiy part of a major project. Extracting informations from 55 databases/unrelated systems in a coherent manner is a nightmare by definition.
The number of different 3rd party libraries that are required, the type incompatibilities, matching fields, accounting for changes in the schema, and talking to legacy systems (I am sure there are many of them in there...), all in 5 months? Good luck.
Add to this the overhead of talking to IT and management of insurance companies, the IRS and others, and just the fact that it does answer some queries is fairly impressive in my book.
Also I am sure that most of those systems are failry deep inside the corporate networks of these insurers, just getting access to them probably was not an easy task.
That said the authentication layers failures and lack of testing are far from impressive.
Where are the links to IRS and social security admin? During the sign up process they verify tax return status and ssns of dependents including children?
"I can tell that this system consists of a JBoss Application Server with data access components and RESTful web services developed using Java. Given my prior experience with JBoss and Java, although they are great for middleware development, they’re known to be a bit slow."
I always had this non founded impression that something running on the JVM would be slow ... this author also. Anyone with data to back this impression ?
It has nothing to do with "running on the JVM". The JVM is about as fast as you'll get any VM offering GC and running an IR like Java bytecode. It's only slow compared to compiled, native code (and even then nowhere near as relatively slow as it used to be), which no competing tech stack would be doing. It's much faster than Ruby, Python, etc.
That said, my experiences with JBoss haven't left me enamoured of it, and it certainly wouldn't be my first choice.
In techempower's web framework benchmarks[1], Java-base frameworks top most categories. One C++-based framework, CPoll, gets a few percentage points more in a few tests, but Java servlets are the highest performing across all tests. (As always with micro-benchmarks, take all this with a grain of salt, but at least it's some data.)
There are a ton of Java frameworks in the top performing benchmarks (See Lng:Jav). Whatever conclusion you've made from Java Swing applications running on top of WindowsXP machines in the early 00's are not applicable to modern web server performance.
I think the interesting lesson here is that even marketplaces should be competitively constructed - i.e., there should be significant price-competitive pressure to deliver goods and services; just letting the government to facilitate marketplaces removes price competition from the marketplace itself.
I'm not suggesting that the government create a meta-marketplace for health insurance marketplaces. Just that it isn't as efficient as some people think to introduce a powerful federal competitor with 'unfair advantages' like public funding into the health insurance selection market.
Realistically you would need to double that 100k to pay for other expenses (hardware, benefits, office, management). Even then, CGI Federal screwed the American people and should be fined for what they've done.
What makes you say that? Having built systems in this space with very large number of users using the .net platform, I wonder why you stated this.
I am a supporter of open and non-MS software but I don't think we can blame it on the technology this time.
In my experience so far a well engineered system, will perform regardless of the technology stack picked. Picking technology X or Y will not compensate for bad architecture and engineering practices.
stackoverflow and plentyoffish are windows stack. I just found out and read about this a few days ago, I'm very impressed. So I won't shake my fist at windows stack. With competent developers, windows/c#/MSQL, etc can holds it's on too. it's been demonstrated.
Amen to that. This thread and all others that mention the healthcare.gov debacle would go on forever talking about MS and ignoring the fundamental issues of bad architecture and government waste.
It's not just government waste. The private, for-profit companies that are awarded these contracts are part of the problem. It's not like "government waste" is even the biggest problem here. The biggest problem is the private companies involved in this. Not the government. Although all parties deserve some blame, for sure, because they have a "scratch my back, I'll scratch yours" approach.
I agree that all parties deserve some blame, but spreading the blame around does not excuse the amount of money the government spent on this and other projects such as recovery.gov. For profit private companies are in it to make money so I don’t blame them for taking the money; I blame them for bad design and architecture.
You should blame them for taking the money, because they "take the money" specifically by acting in bad-faith in order to extend, extend, extend and extend the contract money flow into their coffers. That's how the system works: private, for-profit companies fuck things up, intentionally, and then because they are the experts on the system they fucked up they (practically) automatically get the (almost as lucrative) follow-on extension/"fix it" contracts. The "bad design and architecture" these for-profit companies come up with are a feature. It is intentional, and it is done intentionally because these for-profit companies aren't in the business of providing products and services for the government--they're in the business of needling more contract cash out of it.
Or, to put it another way, you can't cry foul at the government's waste of money when it is the contractors' bids and bad-faith implementation of the contracts pushing up the costs. In return for their efforts, their liaisons at the government get to keep their visibility on "important" projects high, and extend the life of their own jobs.
Ok I get it: The government and the politicians are not at fault here; it is the private for-profit companies fault. Sorry, but I just don’t agree with your point of view. Like I said: there’s plenty of blame to go around but I don’t see how anyone can justify spending that much money for something that doesn’t work.
While it's easy for people to crap on things like js-minifying, cache control, etc - the hard reality is the stuff that is the hardest is so hard that most people have never done anything even remotely close.
Having worked on 10m+ loc codebases, life gets very hard. Dynamic languages tend to be unworkable. Module control, strong interfaces, and a lot of division of responsibility becomes the order of the day. And just figuring out how to build the thing becomes it's own team.
For those who are just thinking "so, just slap together some JSON/web services and call it a day" - I invite you to research EDI and then report back here. Assuming you're still alive.