My Take On "The Hard Thing About Software Development"
Jesse Watson wrote the most insightful post on software engineering. Permit me the honor of first summarising it, and then presenting my take on it.
Summary
The software industry has an atrocious failure rate: 40% of software projects fail. Half the projects end up taking double the estimated budget.
Why does this happen?
The #1 reason is building the wrong product.
How do you prevent this from happening?
By being an expert in the domain you’re working on.
This raises the question: how do you become an expert?
By immersing yourself for years in one domain. For example, Amazon has someone who worked in e-commerce warehouses and shipping for 10 years, and knows that capacity planning is more predictable for Black Friday than it is for Christmas because buying patterns are heavily impacted by day-of-the-week, and Christmas falls on different days, while Black Friday falls on Friday every year. This knowledge was hard-won over years, and resulted in a benefit to Amazon much more than his salary.
If you’re a techie, spending years becoming a domain expert in some domain like travel will let you develop deep context that will generate 10x more value than being a generalist.
Can’t we instead get a spec?
Instead of developers having deep context into the business, can the business people, or a business analyst, communicate requirements in the form of a spec, as this infographic shows?
No, for the following reasons:
First, it’s not possible to write a spec because you can’t enumerate the situations that will occur ahead of time. Software development is a journey of discovery, where you encounter a problem, like a boulder in the path of the road-laying crew. You must face each obstacle and either eradicate it or route around it, and soldier on, only to find another boulder around the next bend. You can’t foresee and list them all out in advance. Software development is exploratory, and a spec doesn’t work for that; a spec works for a predefined, closed-ended problem. The latter get built into a product, framework or web service, and there’s less business value in re-solving the same problem, just as we don’t rewrite a JSON parser.
Second, a business analyst would not understand the level of detail and explicitness that computers require.
Third, even with a spec, you’ll get some things wrong and have to go back to customers to iterate. A waterfall approach of “Here’s the spec, now go and build it” won’t work.
Fourth, the only complete spec is a program. A text document can’t describe all cases precisely.
Fifth, if an engineer were to verify every assumption with a business analyst, the project would grind to a halt.
For these reasons, the greatest misconception about software development is that it’s a separable discipline from deep analysis of the business problem.
Deep Context
We’ve seen that a spec doesn’t work. Neither does having an intermediary like a business analyst to insulate engineers from customers or business.
The only remaining solution is for engineers to develop deep context into the domain. Tough problems require a year with the customer, in person, forming a group mind, getting inside the customer’s head, checking each other’s assumptions, and ferreting out misunderstandings. Communication, internalisation, and synthesis of ideas is the hard part, not technology.
Years of a software project are spent in the grey area between business and tech, not in business, not in tech:
The only way human beings can learn such complexity is by traversing the same territory again and again, learning a bit each time. A sign this process is nearing completion is that you won’t have to think hard to recall a problem and solution. You come up with abstractions and give them a name. The name is like a handle: when you pull the handle, it opens a drawer full of context, which itself contains more drawers.
When a company looks back on years and millions of dollars spent every year and asks what they got, it’s having a team full of people with deep context.
My Take
First, Jesse’s post is the most insightful one on software development I ever read. Every engineering and product leader should study it.
Second, my experience has also been that it takes years to learn a domain. My startup Futurecam took me three years to build [1]. And a good part of that went into learning the domain of photography. Startup culture celebrates results but not learning. But that’s putting the cart before the horse. If you don’t respect and allocate time for learning, you’ll get only products that are hollow on the inside, or defective, or don’t really meet customers’ needs [2]. One well-intentioned but clueless person told me it should be done in weeks. That works for trivial software like a todo list app, but not for sophisticated software like Futurecam.
It takes years for our brains to learn extremely complex domains. Someday we may be able to load this knowledge into our brains:
But until our brains have a USB port, the only way to learn something complex is over years. That’s why a college degree takes four years and not four weeks.
As Jesse says, we invented our own abstractions and named them. Futurecam has features like long exposure that take a photo, but they internally take multiple photos and combine them using our algorithms. I came up with the term “frame” to refer to these intermediate photos, and “photo” to refer to the final photo that gets saved to the photo gallery.
Third, with Futurecam, I also faced the problem of both computers on one hand and users on the other being extremely picky, and entrepreneurs getting stuck in between.
Fourth, I also agree with Jesse that different functions in a team can’t be separated out to the extent where engineers can say “I don’t need to think about X; there’s a person with that job title in the team”:
Fifth, I agree with Jesse that the main return on investment for a company spending years and millions of dollars is a team with deep context into the domain. I’ve seen some businesspeople not understand this asset and lament that they didn’t get much in return. The return is not feature delivery, or productivity, or a big user base — these are just outcomes of the deep context.
Sixth, while deep context makes a 10x difference to companies, it doesn’t mean that you as an individual will be rewarded for it. I was talking to an engineer who invested years in the domain his company works in. As a result, he acquired deep context, but his company did not value that — they viewed him as disposable. This put him at a disadvantage compared to other engineers who invested in learning engineering skills as generalists, and were viewed as better engineers than him.
This is the opportunity cost of acquiring deep context. You’re giving up other opportunities for skill development as a generalist: you could learn to build intuitive apps. Or scale systems. Or develop soft skills like how to contradict people in such a way that it doesn’t become an argument. Deep context also closes down your options: if you’re an travel expert, you have to work in travel whether you like it or not. What if the sector faces a shock like Covid? Or a long-term downturn? I was talking to one person who felt stuck. Another who abandoned his domain after acquiring deep context, throwing away all his knowledge and starting over. So acquiring deep context has both pros and cons.
If you’re curious about what I do, here’s my consulting website.
[1] This was partly because of financial constraints forcing me to go slow, but even given infinite money, I would not be able to do it in a single year.
[2] The only way to get customers to use such shoddy products is to bribe them, by giving it away below cost, like BigBasket’s free 30-minute delivery.