Can filesystems adopt zero-overhead abstractions?
Bjarne Stroustrup explains the concept of zero-overhead abstractions: what you don’t use, you shouldn’t have to pay for. For example, why does C++ have both fixed-size arrays, and variable-size vectors? Because a vector incurs overhead: every access requires an indirection, creating a vector requires one more dynamic memory allocation (and eventual deallocation), and a vector allocates extra space for growth. When you use a vector, you pay all these costs, in exchange for its benefits. But when you don’t need its benefits, it doesn’t make sense to pay its costs. That’s why C++ offers both arrays and vectors.
Can the concept of zero-overhead abstractions be applied to the filesystem? For example, I’m told that high-performance databases bypass the filesystem and store data directly on the block device to avoid the overhead of the filesystem. But couldn’t we take that knowledge and go back and improve filesystems so that databases no longer have to bypass the filesystem?
I’d love to see a list of the features of modern filesystems along with which of them impose a cost when they’re not used.
For example, when you rename a file, if another file exists with the same name, it’s overwritten, and this is atomic — if the power fails, you’ll have either the old file or the new one, but you’ll never end up in a situation where both are lost. Or in which bits of one file are commingled with the other. This has feature been used by apps to save files safely: they save to a temporary file, and after the file has been written completely, atomically rename it to the desired name. That way, if the power fails, you’ll end up with either the old or the new document, but never lose both, and never end up with a corrupt document. But does supporting atomic renames force the filesystem to be implemented in a way that incurs cost when not used? In other words, if I asked you to implement a filesystem and told you renames need not be atomic, would you be able to implement a faster filesystem?
You can list other filesystem capabilities, and ask the same question for each of them:
You can delete a file while it’s open, and the filesystem deletes it for you when you close it. Would the filesystem be faster if you couldn’t delete an open file?
A file has a name that you can choose while creating it. Would the filesystem be faster if names were auto-assigned IDs you had no control over?
You can rename a file. Would the filesystem be faster if names were immutable?
A file has a persistent ID (inode number) that doesn’t change when it’s moved or renamed. Would the filesystem be faster if there was no way to track a file across moves?
Hard links let one file have multiple names, requiring indirection. Would the filesystem be faster if a file must have exactly one name, no more, no less?
Files can be organized in directories, which are first-class objects: if you delete all files from a directory, the directory still exists. It has permissions and metadata of its own. You can symlink to it. An alternative way of implementing directories is that they’re not a first-class object: they don’t have permissions or metadata of their own. You can’t symlink to it. A directory is merely the results of a query to list all files that have a common prefix that happens to include / characters. When the last file in a directory is deleted, the directory is deleted as well. Would this result in a faster filesystem?
I don’t have answers to these questions because I’m not a filesystem expert, but it would be good to evolve filesystems this way. Maybe someone could design a lite filesystem that doesn’t offer full POSIX semantics and is faster or less buggy for use cases that don’t require those semantics?
Filesystems have gained features over decades. Even when a new filesystem is designed, like AFPS, backward-compatibility with all existing apps forces the filesystem to continue to carry the accumulated baggage from decades ago. Could they go faster if they dropped the baggage when not needed?
Filesystem are like apps — they only gain features, never lose them, because someone would yell if they do. They become more and more bloated as the years go by. So just as new apps like Google Keep are created to replace bloat like OneNote, could we have a new filesystem that’s faster and less buggy?