The Swamp of Low-end Engineering

Ravi Iyer (disclosure: my professor and mentor at college) has an interesting article comparing engineering and scientific mindset in the context of software development. (I had an email discussion with him, where he corrected some of my mistakes, but we still disagreed at the end, so I'd like to put forward a different view here.)

His main point is that engineers operate under a need-to-know mindset, learning only as much as needed to get their present job done, and not needing or wanting to know how things work under the hood. They would use a vector class knowing only its interface, and not understanding how it actually works. On the other hand, scientists, in Ravi's telling, want to understand how everything works, what they are built out of, how those components work, etc. Ravi does qualify his comment about engineers by saying that it doesn't work for developing core building blocks like OSs, system libraries and programming languages — it works only for using existing components to build applications.

Unfortunately, this characterization still paints all engineers with too broad a brush — talking of an "engineering mindset" is misleading because high-end engineers perhaps have more in common with scientists than with their low-end brethren. So, a three-way categorization is more useful:

• low-end engineers

• high-end engineers

• scientists

High-end engineers typically work for the top tech companies in the world like Google (disclosure: my employer), Apple and Amazon, or startups that are equally competent, whereas low-end engineers typically work for out-scourcing companies like TCS or Infosys, where cost is the primary factor.

(The classification of engineers into high- and low- end may be uncomfortable for some people. My intent is to have an honest discussion, which requires calling a spade a spade, not to insult.)

These people are doing software only that's the best paying job they could get, not because they have a passion for it. If civil engineering, or movie-making, or marketing paid more and they could get a job, they'd be doing that instead. Low-end engineers are not the ones you'd find reading a paper on distributed systems, on the weekend, on their own time, for the love of it. Whereas both high-end engineers and scientists might, which is another way high-engineers are closer to scientists than they are to low-end engineers.

Another factor is scale — high-end engineers can build systems used by hundreds of millions or even a billion people. A related factor is technical complexity — are you building a system that can serve a petabyte of traffic at peak, or are you building a web site consisting of some Perl scripts running on an Apache server for Canara Bank? High-end engineers earn more, multiple times the salary of low-end engineers, in some cases, as a consequence of their greater abilities.

Both the high-end engineer and the scientist want a complete bottom-up understanding of the software stack, but their motivations are different — ultimately, the high-end engineer wants to learn so that he can build a better system, while the scientist may build a system so that he can learn more.

Ravi seems to mis-characterize this by saying:

Component-Based-Development delivers solutions quickly and at far lower cost than reinventing-the-wheel solutions

That's true by itself, of course, but almost a tautology — reinventing the wheel is by definition understood to be bad. More importantly, this statement is misleading — someone who wants a complete bottom-up understanding of the software stack isn't doing so because they want to reinvent the wheel. Just the opposite — if you know how things work at each layer, what is done at each layer, and why, how else we may build our software stack, and what the pros and cons of that alternate approach may be — if you understand all this, you are less likely to reinvent the wheel, because you'd know that you could reuse the existing component and that it would work for you. Similarly, a bottom-up understanding means that you are also less likely to blindly reuse a component when it wouldn't work for your use case, when the situation genuinely calls for a re-implementation. A person who doesn't understand why he's doing what he's doing is obviously going to produce better work than a person who doesn't.

More concretely, going with Ravi's example of understanding the vector's interface and not how it works, if all you know about the vector is its interface, if you think, "oh, I can invoke vector.add(4, object) and it will add the object at the fourth position [1]", and if you don't know about asymptotic complexity, you could insert things at the beginning of the vector when you could just as well insert them at the end. This wouldn't work when the vector got really large, and your program would then grind to a halt. That doesn't happen for every program, or for every vector in a program, of course, but the point is that someone who doesn't understand how the vector works is liable to building a broken program in his ignorance, like a civil engineer who builds a mall that collapses when it's packed, during the peak shopping season.

Time frame is another difference. Low-end engineers usually works at a shorter time-frame, and high-end engineering may work at either a shorter or a medium time frame, like a few years for Google (or, I assume, Apple). Science can go on for a long time frame, like a decade or longer. For example, the LHC took 14 years to go from proposal to when it was used for the first time. As another example, Knuth has been writing his books since 1962, and still hasn't finished.

Engineering, whether low- or high-end, is ultimately driven by solving people's problems, while science is driven by advancing the world's knowledge. So high-end engineering attracts people who get a kick of solving real-world problems, but who would otherwise have been scientists.

Both engineers and scientists can work in one of two directions: they can work backwards from the system that needs to be built, or forwards from the theory. Both directions are valid and have their place, and both high-end engineers and scientists work in both directions, but I think scientists work forward from theory more often than do engineers.

Science does run the risk of intellectual masturbation, doing theory for the sake of theory. I'm not talking about fundamental research like the properties of matter, or doing research with plausible benefits, but more about work like the kind one of my professors at IIT Bombay did. He'd say that if you have a struct:

struct Something {
    int a;
    char b;
};

and if you represent the set of int values by A, and the set of char values by B, then the set of values of this struct is A X B. Which is fine, but how does it help me write better programs? I remember arguing with him about this, saying that I want to do something useful, whether it's formal or not, but he seemed to want things that are formal, whether they are useful or not.

Paul Graham writes insightfully about this:

[...] Instead of doing what they really want to do, which is to design beautiful software, hackers in universities and research labs feel they ought to be writing research papers. 

In the best case, the papers are just a formality. Hackers write cool software, and then write a paper about it, and the paper becomes a proxy for the achievement represented by the software. But often this mismatch causes problems. It's easy to drift away from building beautiful things toward building ugly things that make more suitable subjects for research papers.

Unfortunately, beautiful things don't always make the best subjects for papers. Number one, research must be original-- and as anyone who has written a PhD dissertation knows, the way to be sure that you're exploring virgin territory is to to stake out a piece of ground that no one wants. Number two, research must be substantial-- and awkward systems yield meatier papers, because you can write about the obstacles you have to overcome in order to get things done. Nothing yields meaty problems like starting with the wrong assumptions. Most of AI is an example of this rule; if you assume that knowledge can be represented as a list of predicate logic expressions whose arguments represent abstract concepts, you'll have a lot of papers to write about how to make this work. As Ricky Ricardo used to say, "Lucy, you got a lot of explaining to do." 

The way to create something beautiful is often to make subtle tweaks to something that already exists, or to combine existing ideas in a slightly new way. This kind of work is hard to convey in a research paper.

Engineers have their value measured by the market — you won't get paid if you don't solve real problems. And you get paid more if you solve important problems. The market doesn't pay for intellectual masturbation. Of course, no one's arguing that the market is perfect, and indeed a lot of technological advancement has come from science, so we need both the market-driven engineering approach and the scientific one.

The real problem with low-end engineering is that it's often crappy — it does not really satisfy users. I remember the Indian Railways web site charging me money without issuing a ticket, and a dozen emails to their customer service department yielding no solution, beyond a "Please note that this transaction was cancelled." No, you doofuses, the transaction went through half-way — you charged me money.

Another fuck-up is the Airtel web site, where nothing works as you expect it to, where you can't find the thing you want, and even if you are able to find and change it, it doesn't reflect in the rest of their systems, forcing you to spend tons of time dealing with their customer-service incompetents. Airtel still emails me at my older email ID, more than a month after asking them to update their records, both on the web site and on phone, multiple times.

Ravi writes:

[...] The real world is driven by how society's needs are satisfied quickly and at acceptable cost.

Fair point, but in many cases, frustrating the user doesn't save money. If anything, as in the example above, it increases costs both for the user and the provider (who has to deal with support calls and negative reputation). A lot of things in the world suck not because of any intrinsic economic necessity, but for lack of imagination and empathy and a desire to make things better, even when such improvements have little or no cost.

These systems are aggravating and generate feelings of frustration, alienation and, at worst, rage, as opposed to the feeling of joy and satisfaction and competence people feel when they are able to accomplish a task. People use Airtel only because there may not be a better ISP. And that's a bad place to be in, both business wise and from a creative point of view. I used to happily tell friends what great service I used to receive from Airtel, but I no longer have any loyalty left for them. And I'm not the only one. Airtel leaves itself open to attack from any competitor who meets the low bar of merely not aggravating their paying customers.

Business aside, as a creative person, as a person who wants and hopes to make a positive contribution to the world over the course of his career, you're a failure if all you can do is build systems that at worst aggravate people (as with Airtel) or are often merely unpleasant to use (Indian Railways, SBI, Canara Bank...).

Low-end engineering is where ambition and creativity go to die, and where discontentment lives. In that sense, low-end software engineering is no different from the low-end of other industries. Taking transportation as an example, the low-end consists of traveling in a crushed bus, surrounded by other people and their sweat, for 10 rupees, while the high-end may be a spacious, air-conditioned chauffer-driven car. Why would expect the software industry to be any different, for a lot of low-end software engineering to not suck?

[1] Fifth, since it's zero-indexed.