Over the weekend I did an interview with blogger Stefan Molyneux. Like myself, Stefan worked in software development. So we got to talk shop.
After that interview, I thought that I might try to explain the why of H-1B. That requires a look at the state of the software industry.
There is a great disconnect between the perceived state of software engineering and its actual state. We talk about "high tech", but in reality software engineering is in its infancy. Software engineering, as we know it, is only about 60 years old. Compare that to civil engineering where there are thousands of years of experience.
Software engineering is as mature as the civil engineering of two thousand years ago. In fact, software engineering is probably less advanced than that.
Think about the age of the cathedral builders. They did not have the advantage of calculus and formal structural analysis. Their master masons designed cathedrals by instinct — and in most cases they were successful. But there were disasters.
Beauvais Cathedral collapsed. So did Ely Cathedral and Lincoln Cathedral. But for the most part the medieval cathedral builders were successful.
And the medieval cathedral builders were positively advanced compared to today's software engineers. Software disasters today are much more common than cathedral disasters in the Middle Ages.
Here are some reports on the rate of software engineering failure. I have tried to pick out reports that reflect the full range of such studies.
- "Why up to 75% of Software Projects Will Fail", Geneca.com, January 25, 2017
- "The failure rate is 15 to 20 percent for projects that have budgets of $10 million or more", IEEE Spectrum, September 2, 2005
- "Study: 68 percent of IT projects fail", ZDNet, January 14, 2009
- "More than half of IT projects still failing", CIO, May 11, 2016
- "Most Organizations Have a 70% Project Failure Rate", 4PM.com, September 27, 2015
- "Seven Reasons IT Projects Fail", IBMSystems Magazine, February 2012
- On the success side, the average is only 16.2 percent for software projects that are completed on-time and on-budget.
- "McKinsey revealed that 17 percent of IT projects budgeted at $15 million or higher go so badly as to threaten the company's existence, and more than 40 percent of them fail" - "The top 5 reasons why IT projects fail", Betanews.com
- "62 percent of IT projects fail. Why?", Cnet, March 21, 2008
- "Gartner: 75% of all Enterprise Resource Planning projects fail – But why?", The Office of Finance
- "70% of software projects fail" - "Leveraging Business Architecture to Improve Business Requirements Analysis", Business Architecture Guild, March 2014
- "34% of software projects are successes and 15% are total failures" - "IT Myth 5: Most IT projects fail", Infoworld, August 13, 2004
- "28% of large software projects are failures" - "Gartner Survey Shows Why Projects Fail", Thisiswhatgoodlookslike.com, June 10, 2012
- "Why 70% of Government IT Projects Fail – Quality Project Management for Education Agencies", ESP Solutions Group, 2009
- The International Project Leadership Academy compiled a list of analyses
- IT Cortex compiled another list
- "Case Study: A Computer Science Perspective on Bridge Design bridge construction to software development", Alfred Spector and David Gifford, Communications of the ACM, April 1986, Vol. 29, No. 4
The variation in the numbers here is the result of studies measuring different things; especially different measures of success or failure.
For any software project there is a range of outcomes. A project can be an unqualified success, an unqualified failure, or something in between. That in between includes projects that run way over budget, like the Obamacare website. Software projects that are completed but do not meet expectations or provide value also fit in this category
If we cut through the hype in these reports, we find that roughly a quarter of software projects are successful, a quarter are complete disasters, and about half fall somewhere in between.
To put this into perspective, this would be equivalent to a quarter of buildings erected falling down and another half requiring extensive reconstruction to be made usable. If just 1 percent of buildings collapsed there would be enormous pressure to take corrective action, but a quarter of software projects doing the same is accepted as normal. Any way you look at these figures, the current state of software engineering is primitive.
Whenever a disaster occurs, the human instinct is to cover it up. That usually is not possible in civil engineering where a building collapse produces a pile of debris.
When a software project fails, the only visible evidence is an accounting ledger that is much easier to conceal. When a software project fails, all involved, from the executives who cooked up the idea to the consulting companies that built it, try to cover up the disaster. Because of this ability to conceal, there has not been strong effort to prevent software disasters from occurring.
We can distill all these software engineering failures into a simple business problem: Software costs too much.
If you wanted to address that cost problem in a scientific way you would:
- Use development tools that maximize productivity;
- Set up physical work environments to maximize productivity;
- Ensure developers have the right equipment to do their jobs most efficiently;
- Put in place personnel policies to reward and retain the most productive developers; and
- Put in place software management policies to ensure systems are properly designed and to prevent scope creep.
That kind of change requires knowledge of the software development process. These are issues that your average accountant has no conceptual understanding of.
But accountants make high-level decisions in corporations.
When you put the software cost problem in the hand of accountants, they understand it as: American computer programmer cost $90 an hour and Indian computer programmers cost $80 an hour.
The obvious flaw in that accountant analysis is that in the best case it simply makes disasters cost less.
I shared with Stefan an experience I had before law school. I joined a project written in C++ that had serious performance problems. The original developers were not familiar with table joins in database queries. They wrote the system so that it used millions of nested queries where one query would suffice. I was in the process of replacing these millions of queries with single queries that obviously boosted performance tremendously.
While working on this issue, a senior accountant-type in the company (without consulting anyone familiar with the structure of the system) hired an Indian body shop to rewrite the entire system in Java to solve this problem. Their legion of H-1B programmers slavishly copied the C++ code into Java, including the parts that used millions of queries where one would suffice. The result was even slower than the original — another software disaster.
It is this fundamental problem of software cost that is driving the demand for H-1B workers. Unfortunately, cheap labor on H-1B became a distraction from fixing the underlying problems in software engineering. The H-1B opiate has thwarted the advancement of software engineering and undermines American leadership in this area.