Make Your Backend Layer As Thin As Possible

The best backend is no backend at all.

Say you’ve started a startup today.

You need to come up with an API between your backend (running in the cloud) and your frontend (running on users’ devices, whether in the form of a mobile or a web app).

We try to define such an API by identifying the domain objects in our system and the operations that will be performed on them, have the backend and frontend team agree on these nouns and verbs, have the backend team implement the API, and the frontend team consume them.

Instead, I propose the following approach:

In the three-tier frontend-backend-database architecture, try to have as thin a middle layer as possible. Or even eliminate the backend layer altogether by directly exposing the database to the frontend.

For example, if your API returns a Person object, just return the fields of the underlying table 1:1. Do a SELECT *, iterate over all the columns, and copy them to the JSON that will be sent back to the frontend. Such a backend layer won’t need changing when you add columns to your database. As opposed SELECT’ing only certain columns in your SQL query. Or copying only certain columns from the SQL response to the JSON like:

json[name] = cursor.name
json[email_id] = cursor.email_id
json[id] = cursor.id
json[birthday] = cursor.birthday

This code will require changing whenever you add a new database column. And when — not if — you forget, it result in needless confusion and debugging:

Frontend engineer: Why am I not getting the phone number field on the frontend? Did you not add the column to the database?
Backend engineer: Yes, I did.
Frontend engineer: Did you not populate it?
Backend engineer: Yes I did.
Frontend engineer: Then what’s going wrong?
Backend engineer: Are you making the right request?
Frontend engineer: Yes, look at this code…
Backend engineer: That looks correct, but maybe the parameters are wrong at runtime. Why don’t you show me the actual request that’s sent?
Frontend engineer: Here it is.
Backend engineer: Hmm… the request is correct. How can we be sure that the backend isn’t returning the phone number. Maybe it is, and it’s getting missed at some layer of the frontend code…
Frontend engineer: I don’t think so, because…
Backend engineer: Better test it using Postman, to be sure.
Frontend engineer: *goes off and tests* See, here’s the API request and response.
Backend engineer: Hmm, let me look at it, maybe in a few days, I’m busy with something else now. (After two days) Oh, the backend API implementation needs to be updated to add the new field.

You can bypass this whole inefficiency by making the backend a pass-through. That gives you one fewer thing to worry about, which any engineer will welcome.

In your API documentation, don’t list out fields that will be returned, and their types, one by one. Instead say “Returns the same fields as the database.” Again, for the same reason: It can — no, will — get out of date, and misleading documentation has a negative value to your company. Instead, ask the frontend engineers to directly refer to the database schema, which can’t get out of date, because it is the source of truth.

Neither should you do mapping like

json[full_name] = cursor.given_name + " " + cursor.family_name

That’s a layer of translation that doesn’t add value. And makes it harder for engineers to get an end to end view of the system, say if they’re curious whether names are represented as separated given and family name fields or as a single full name field. Not only that, but such translation layers will have bugs that need fixing.

You can take this one step further by directly exposing the database to the client app, such as by having the client app just send SQL queries to the server to execute. If you go this far, you wouldn’t even have a backend layer — instead of a three-tier frontend-backend-database architecture, you’d have a two-tier frontend-database architecture.

Now, that may not always be possible. Say for security — you don’t want a compromised client getting access to other clients’ data, or for a client to mark itself as being on the premium plan despite not paying. Or maybe some code requires access to resources not available on the client. Or is too demanding to run on the client. Or needs to be run at a certain time irrespective of whether the client device is online at that time.

So, intead of no backend, a thin backend may be the right choice for you. Maybe you can give clients read-only access to the database, but not write. Or maybe you can give clients read-write access to only their data. Ask yourself how you can be lazy, pushing as much work as possible down to the database. Modern backend frameworks use maybe 5% of the power of databases. This is unfortunate.

As an example of directly exposing only some data from the database, imagine you’re building a social network, where friend information is public. Say you model this as a friends table consisting of two columns for the user IDs of the two people who are friends. Now, the conventional approach would have you define a backend API for the frontend to get friends. Instead, you can define an API that takes an SQL query from the client, runs it, and returns the results. This query can be run under a database user account that has only read access, that too only to the friends table. This can accommodate unforeseen uses, like getting the number of friends rather than the actual list. Or if there’s a search box where the user can filter his friends, you can do a LIKE query. Or you can limit the number of friends returned to how many will be displayed in the UI. Or return only some fields, the ones you’ll use in your frontend. Or filter only friends who have a phone number, say if you’re building some kind of dialer UI. All this flexibility is now available without you having to foresee it, plan for it, and build it. Don’t make the backend layer a straitjacket between the database and the frontend, any more than you’d force participants in a meeting to communicate via morse code.

SQL is powerful, simple, declarative and most engineers know it — there’s nothing else with all these benefits. Certainly not your custom API. Greenspun’s tenth rule of programming says that any sufficiently complicated C or Fortran program contains an ad hoc, informally-specified, bug-ridden, slow implementation of half of Common Lisp. Similarly, your custom API contains an ad-hoc, informally-specified, bug-ridden, slow implementation of a fraction of SQL. Don’t define such an anaemic API.

This is a new way of thinking: build a backend API when it has something to do, not as a pass through to your database. This is the same reason you wouldn’t define an object that wraps another object and forwards all method calls to it, and the return values back, without doing anything else.

You’re anyway going to put in a lot of thought into designing your database schema. And if you’re not, you should, since that’s harder to change than either backend or frontend code. But once you do that, why design another schema, which is the responses returned by your backend?

Now, you may disagree with the above advice, by objecting that an ideal API should not change if the backend database changes. An ideal API should be a layer of abstraction. But you don’t need every possible layer of abstraction on day 1. Layers of abstraction take time to design and implement, and make it hard to refactor functionality and move it to another layer.

A layer of abstraction may even require hiring a separate backend team, which in turn imposes a coordination cost between the teams, which will never go away. Getting each additional person onboarded and fully productive takes a year. Having teams write a spec to agree on the interface between them, implement this spec, deal with bugs caused by the backend not implementing the spec or the frontend assuming things that the spec doesn’t guarantee, updating this layer when needs change, and so on, is more overhead.

Instead, put your frontend (which includes mobile) engineers in charge of the API. The backend engineers should be there to advise, assist, implement and point out things the frontend engineers overlooked, not to define the API. Instead of an equal partnership between the frontend and the backend teams, put the frontend team in charge of the API.

For example, suppose your frontend engineers propose that instead of having an API to get people, and a second API to get orders, and a third API to get something else, they’ll have an API to get all the data needed to render the home page in one go. Let them have their way. This, in fact, saves frontend engineers from having to make multiple calls to get the data they want to render the page and knit it together. Which in turn causes more complexity like how to handle partial failures, where one call succeeds and another fails. Also, if the calls be made one after another, as a naive implementation would, it increases latency. It’s better to make them in parallel. Unless the second API depends on the first. All this complexity goes away when you define APIs that do what you immediately need, like get data for home page. If you want a car, you don’t want to buy an engine, a steering wheel, brakes and other components and assemble them into a car yourself.

Whenever the home page UX changes, and different information is needed, let the API change. Good APIs aren’t supposed to change, but that doesn’t matter when you control both sides of the API [1]. An internal API should not be held to the same standard as a public API. If you try to, you’ve over-engineering it. Take shortcuts. You’re designing in a context, not in a vacuum, so it’s foolish not to take advantage of that context.

In fact, even using the same term API to refer to two completely different things — internal and public APIs — is questionable, because connotations from one don’t apply to the other. It’s like using the term vehicle to refer to both a bicycle and a nuclear submarine, and trying to apply some principles from designing the sub to the cycle.

If you take this idea of having as small a backend as possible to its logical conclusion, you’ll end up with a BaaS, like Firebase. There’s a lot of wisdom behind that architecture. If I were starting a new product, BaaS would be my first choice. I’d use others only if I’m sure BaaS wouldn’t work for me.

If BaaS doesn’t work for you, explore things like PostGraphile or Hasura that take in a database and autogenerate an API server, exposing the database contents via GraphQL. Or PostgREST, which is similar but autogenerates a REST server, if GraphQL is unnecessary for you.

Eventually you may find that you’ve outgrown this architecture, and need a heavier backend. That’s the time to build one, not before. That goes for a lot of things — you don’t need to have automated tests as a general practice. Or microservices. Or formalised HR policies. Or a lot of things. Don’t ape Google or Amazon. Do what’s right for you, given where you are.

[1] Assuming you do. Otherwise this entire blog post doesn’t apply.