Innovation Deficit: Why DC is Losing Silicon Valley
Originally posted March 1, 2016
The government is terrible at building and buying technology. I know it, you know it, and the government knows it.
Some of the #ThanksObama crowd will try to make this a partisan issue, but it’s not. Giving credit where credit is due, President Barack Obama noticed the problem when he assumed elected office. He has said, “Government has done technology and IT terribly over the last 30 years and fallen very much behind the private sector. And when I came into government, what surprised me most was that gap”.
Essentially, that means that our country’s IT department is worse than the guy on floor three who can’t get your printer to work for more than two days. But instead of printing PowerPoint decks, we’re fighting the Taliban, negotiating with Iran, monitoring borders, building planes, and sending satellites into space.
These failures represent literally trillions of dollars of waste. And you don’t have to look very hard to find them: the F-35 Joint Strike Fighter, FBI’s Virtual Case File, IRS tech modernization, the Army’s Distributed Common Ground System (DCGS), DHS’s Secure Border Initiative Network, Future Combat Systems, just to name a few.
Over the course of this administration, there have been a number of attempts to explain this failure. Lack of transparency. Ineffective managers. Poor standards. Lack of regulations. A slow acquisitions process. Those in power then say, ‘We’ll fix this,’ and add more regulations and programs to ensure that the problem is fixed, like the handful of acquisition reforms that made it into the National Defense Authorization Act, the US Digital Services Office, 18F, Presidential Innovation Fellows, and the Defense Innovation Unit Experimental (DIUX).
These may (or may not) be steps in the right direction, but they miss the real problem, which is that we have a technology acquisitions system built around concepts that make no sense and around companies who stopped being effective decades ago.
If this technology gap is going to be fixed, we need to be honest about what the real problems are before we can lay the tracks for a solution to them. First, we need to come to terms with the engineering talent shift out of large bureaucracies and into lean/agile companies. Second, we need to reform the requirements process to stem the flow of large program budgets into massive dysfunctional bureaucracies. And third, we need to change the incentive structure for budget owners and procurement officers to reward stewardship and effectiveness rather than risk avoidance and career planning.
US Secretary of Defense Ash Carter will be visiting Silicon Valley this week to remind us how important the region and its people are to the future of the nation. After submitting the DOD Budget Proposal to Congress last week, he posted an excellent defense of it here on Medium. In it, he recognized some of the problems referenced above, and said, “It’s no longer just a matter of what we buy; what also matters is how we buy things, how quickly we buy them, whom we buy them from, and how quickly and creatively we’re able to upgrade them and repurpose them to be used in different and innovative ways to stay ahead of future threats”. While I believe his heart is absolutely in the right place, he unfortunately stopped short of identifying where things have gone completely off the rails. If the US government truly wants to partner with Silicon Valley (which all outward indications suggest they do), then it’s time to stop rearranging the deck chairs on the Titanic and directly address the fundamental problems.
Engineers Are Not Commodities
Anyone who has played sports before is keenly aware of the natural distribution of talent. Superstars clearly have outsized performance, value, and compensation. For example, Lebron James played on a state and national championship basketball team at Akron’s St. Vincent — St. Mary High School. Despite that incredible achievement, none of his teammates made the NBA. Lebron has since gone on to be a 4x NBA Most Valuable Player, a 12x NBA All Star, and a 2x Olympic gold medalist. He makes more per year than 6 of the other 12 Cleveland Cavalier players combined, not including hugely lucrative endorsement deals with the likes of Nike, Samsung, Dunkin Donuts, Coca Cola, and Audemar Piguet. And this includes a bevy of 1st and 2nd round picks, college All-Americans, and fellow national high school basketball prodigies. One thing is incredibly clear: skills in basketball are not a commodity.
The distribution of talent in tech is not as dissimilar as you might expect. Bill Gates, Steve Wozniak, and Mark Zuckerberg are not incrementally more talented than an average engineer. They are every bit as exponentially different as an average basketball player is from Lebron James.
So if programming competency is no more of a commodity than professional athletics, why does the government treat it like it is when it comes to procurement? The average Request for Proposal (RFP) involving software development services requires estimates on count of full-time equivalents (FTEs) who will be engaged in the effort, a “basis of estimate” (BOE) modeling FTEs, their labor categories, and the FTE blend deployed against RFP requirements and delivery dates (e.g. requirement “x” needs 2.5 engineers for 6 months to complete), expected time to completion, and resumes for key personnel, with some sort of bonus edge for developers with a great deal of experience.
This process massively disadvantages efficient and more talented teams. Because the talent gap between average and excellent is so large, it would generally be better to have one Lebron-level coder than to have 100 average ones. The difference between a bad developer with no experience and a bad developer with a lot of experience is not much more than just the amount of years they’ve been bad. Remember the Healthcare.gov fiasco? Three 20-year-old programmers built a site that worked in a few days after the public failure.
The fact is, you can’t build a successful technology organization without making your company or mission aspirational for the most talented engineers. Working for/with the government hasn’t been aspirational for a really long time. I think it would be pretty challenging to argue that top computer science majors from the best schools are clamoring to work in a government cube farm rather than spending their time at Google, Facebook, Apple, or a myriad of other great tech startups with much higher upside, better access and more direct influence on the things they’re working on, and a short enough product horizon that they’ll actually be able to see the results of their work in a human lifetime. Too many people in the government are just plain wrong in their beliefs around this. If you are constantly surrounded by mediocrity, any above average performance will appear excellent. That doesn’t make it excellent, though. Ask anyone who has spent significant time in both of these environments. It’s not really even close. (Note: I realize there are exceptions to this, but the statement is directionally true)
When asked about the $500m California Public Employees’ Retirement System data infrastructure overhaul failure, California Lt. Gov. Gavin Newsom said, “How could this happen? It happened because no one on the government side had enough knowledge to properly assess what [was promised] and whether it made sense. It happened because the government will never be able to lure the kind of programming talent it needs.”
In Secretary Carter’s recent budget defense post, he said, “We in the Pentagon must try to make ourselves even better at attracting talent from new generations of Americans.” So maybe he gets this. At least, I hope he gets it.
Takeaway: Engineers are not a commodity and we need to stop treating them like they are. Empower them to take on ownership and risk and unshackle them from bureaucrats and program managers.
The Requirements Process
The Veteran’s Administration’s appointment scheduling system is a mess. A huge amount of vets have been experiencing more than 30-day waits before they can see a doctor. If you want to watch something that will make you laugh and cry simultaneously, check out this video of a vet trying to make an appointment using a VA automated phone system. For those of us used to booking almost every form of reservation online, this is mind-boggling. Don’t companies like ZocDoc already do this for literally tens of thousands of doctors’ offices already? Not according to the VA, which is why in August of last year, they launched into a >$600m contract with Lockheed Martin to build the whole system from scratch.
How does something like this end up happening? Unfortunately, the whole system (formal and informal) is set up to incentivize these types of behaviors.
It all starts with the requirements process. Let me explain.
When the government identifies a problem that they believe needs to be solved with technology, they build an opinion about what the solution should look like and then write up pages and pages of requirements around how a company should go about building it, usually in a vacuum, unaware of commercial best practices and available technologies. The companies bidding for that contract then compete with each other for how well they will be able to meet all of the requirements that have been defined by the issuing agency. The bigger the project, the higher the administrative burden heaped on the vendors (government auditing certifications, “earned value management systems”). This is an environment where only those companies adapted to this highly idiosyncratic and dysfunctional ecosystem are able to thrive at taxpayer expense while delivering garbage.
If that’s confusing to you, trust me, you aren’t alone. As the (likely apocryphal) Henry Ford quote goes, “If I had asked people what they wanted, they would have said faster horses.” It’s totally unclear to me what qualifies a government program or procurement officer to define what the future looks like, down to the color of buttons in a web app.
Unfortunately, when the government asks for something insane like that from an early stage software product company, they have to say “no”, otherwise face death by one thousand cuts. You see, to build a successful software business, you should generally try to sell your product to other customers as well, which means that it needs to be elegant and flexible for a variety of end uses. But services companies have no problem saying, “Yes, and don’t you want a red button to go along with it?”, because they’re billing by the hour and have no reason to not build every component completely custom. After all, they’re building this software for you (special you!) and would like to build the next similar thing for another group (special them!) at similar hourly rates over long predictable time periods.
There is also a risk allocation problem. These contracts are typically issued as “cost reimbursement (“buyer beware”) so the government bears the cost of inevitable delay (hooray!). On the other hand, commercial technology contracts are generally issued as “firm fixed price” such that the vendor bears the cost of delay (“seller beware”).
This works well on the government side as well. As things stand today, budgets are “use it or lose it”. You are incentivized to spend up to the last dollar, so that you can request more in the next fiscal year. The bigger the budget and the higher profile the project, the more of a career boost expected for a civil servant. The Fast Company profile of the US Digital Service noted that, “One of the first lessons Mikey Dickerson [the Director] learned about D.C. when he arrived was that the city traditionally conflates the importance of a task with its cost.” If you are a good steward of taxpayer money and spend less than originally projected, you are rewarded with a budget cut in the following year. That makes no sense.
As a result, if a government budget manager has a choice between building a program from scratch for an astronomical amount of money or buying that technology off the shelf for a fraction of the cost, they are strongly incentivized to do the former. Moneyed interests they’re surrounded by are urging them to this decision as well.
Takeaway: Engineering is an art. Stop telling us what the solution to your problem looks like. You have to give talented people room to solve problems in elegant and creative ways if you have any hope of building something memorable.
Structural Issues in the Procurement System
So who benefits in this system? A handful of large and well- connected companies, falling under a category generally called “systems integration (SI)”– whose very name suggests that they’ve forgotten how to build things. Their business model is structured around proposal writing and program management functions, spending days/weeks of non-billable overhead time building responses to RFPs, followed by years of building a bunch of one-off projects (from software, to trucks/tanks, fighter planes, rockets, etc).
To make matters worse, many of these SIs have a kind of “preferred vendor” relationship on the contract vehicles with the Federal Agencies. These “prime contractor” relationships are intended to make the procurement process more streamlined by consolidating overhead and making it possible for small and medium sized businesses to “sub-contract” as partners on bids. What it really ends up doing is (again) incentivizing the SIs to horizontally (rather than vertically) integrate, which over-complicates proposals and makes management nearly impossible.
It is common knowledge that most Requests for Proposal (RFP) are either written by or influenced heavily by companies with strong relationships inside government agencies. One of the key pieces of advice that I offer companies in the Founders Fund portfolio who are exploring work in the government is to avoid the temptation of responding to an RFP that has been sent to them or that they found while searching on one of the many bid posting sites, like fbo.gov. Most likely, posted RFPs are already awarded (or at least soft circled) and just running the countdown clock until they can close out the competitive bid window, quickly evaluate, and award to the expected party. Sure, there are exceptions, but chasing lottery odds usually shouldn’t be sufficient for allocating resources when you are a startup.
So how do you develop an inside relationship so that you can influence an RFP to be custom designed for you? That’s where the revolving door comes in. Large government contractors frequently hire government officials with procurement/budget authority, particularly after they’ve built strong relationships through previous contracts. The argument here is generally that these former government employees have relationships and skills that are in line with the needs of the contractors involved. You could also easily argue that the officials could be tempted to behave in ways that set them up for better paying jobs in the private sectors after they “retire” from government service. Need examples? Don’t worry, there are plenty. Here, here, here, here, and here, for starters.
Further, what repercussions is CGI Federal going to face due to huge failure with Healthcare.gov? What about SAIC on the FBI Virtual Case File System? What about Boeing on SBInet? Very likely, none. They have all turned into vast sinkholes for taxpayer money.
There are political challenges/pressures as well. Take the F-35 program, for instance. The F-35 is being built by literally hundreds of sub-contractors who collectively represent 127,000 jobs in 46 states. It’s a logistical nightmare and (simultaneously) a governmental affairs wet-dream. What member of congress in their right mind would shut down a program that employs hundreds if not thousands of people in their home districts?
Due to these structural issues, startups are forced to approach competition in a different way than they would if they were doing most of their business with the private sector. It’s not about evaluating whether or not other companies are offering similar products and/or services to the market. It’s about whether or not the government is going to try to build it themselves. And unfortunately, there’s not a lot of evidence to suggest that they won’t. Research and development pilots are admittedly becoming easier to work out (thanks to organizations like In-Q-Tel and the DIUX), but when it comes to going into production as a real line item on the budget, the systems integrators are always looming, ready to pounce and eat your lunch. Young tech companies should care less about how long and arduous the process is than that the correct decision is eventually made.
In his last address as President on January 17th, 1961, President Dwight D. Eisenhower said, “In the councils of government, we must guard against the acquisition of unwarranted influence, whether sought or unsought, by the military industrial complex. The potential for the disastrous rise of misplaced power exists and will persist.” This is just as true today as it was over 55 years ago.
Takeaway: The rules of procurement are designed to make things appear fair, but in doing so they condemn the whole process to failure. Systems integrator bias is killing our ability as a nation to engineer the future.
Conclusion
In truth, our government is incredibly advanced compared to others around the world. It’s really not even that close. But that’s not the point. We also have the best tech community, and should be doing more to harness the minds of the best and the brightest, which must ultimately be the standard if we are going to maintain our edge.
Whenever I hear about new government programs, I often find myself wondering if these things would be investable, even if success was measured by mission outcomes rather than financial ones. Would I invest in the F-35? DCGS-A? The planned B-21 (read Wired’s recent story, “America’s Next Great Boondoggle — Er, Bomber — Is on Its Way”)? Healthcare.gov? Forget the systems integrators who are being hired to build these things, would I invest in the program office? Not a chance.
Said another way:
In the 1890s and early 1900s, Smithsonian Institution Secretary Samuel Langley worked on a “flying machine” program he called the Aerodrome. After almost ten years of failed Aerodrome attempts, the Wright Flyer ended up beating Langley to the punch having reached some novel insights around aerodynamic stability and airframe stress re-distribution.
As David McCullough pointed out in his brilliant biography on the Wright Brothers, “The Langley project had cost nearly $70,000, the greater part of it public money, whereas the brothers’ total expenses for everything from 1900 to 1903, including materials and travel to and from Kitty Hawk, came to a little less than $1,000, a sum paid entirely from the modest profits of their bicycle business.”
Worse yet, with Smithsonian approval, modifications were made to the Aerodrome in order to attempt to invalidate the Wright Brothers’ aircraft patent, after which the Smithsonian put the Aerodrome on display as the first heavier-than-air manned, powered aircraft. Eventually, in 1942, the Smithsonian backtracked that decision, but their relationship with the Wrights was ruined. Sometimes, no amount of public money you throw at a problem will be able to beat a strong team, a novel approach, and the absence of bureaucracy.
So, while Silicon Valley certainly appreciates the overtures and back-patting coming from Washington, all of these gestures are really just opioids, not medicine, for what ails the nation. If you want a renewed engagement with the government from the region, then it’s time to get serious about addressing the core problems:
Engineers are not a commodity and we need to stop treating them like they are. Empower them to take on ownership and risk and unshackle them from bureaucrats and program managers.
Engineering is an art. Stop telling us what the solution to your problem looks like. You have to give talented people room to solve problems in elegant and creative ways if you have any hope of building something memorable.
The rules of procurement are designed to make things appear fair, but in doing so they condemn the whole process to failure. Systems integrator bias is killing our ability as a nation to engineer the future.
Said another way: Remember the Aerodrome
