Why Databases Instead of Filesystem?

Question

A database could be described crudely as a program for interfacing with data on disk. Most software is built to interface with databases although the software also has (often very capable) ways of interfacing with data on disk via filesystem interaction.My question is, what historically has driven the industry to focus on database-specific solutions, rather than on filesystem-specific solutions?This is not a rant against databases, but I do wonder why many major programming languages and frameworks (RoR Active Directory, C# EF, etc) have put much effort into making database interaction smooth from the perspective of the programming language, instead of putting effort into interacting with data on disk. Kind of an alternate reality sort of musing.

lgeorget · Accepted Answer

More of a sidenote than an answer but a database system can be faster than using the disk directly: https://sqlite.org/fasterthanfs.html#approx.It turns out having a defined abstraction like a database makes things faster than having to rely on a rawer interface like filesystems because you can then reduce the number of system calls and context switches necessary. If you wanted to optimize that in your own code rather than relying on a database, you'd end up reinventing a database system of sorts, when (probably) better solutions already exist.

codingdave · Answer

A database is a file system when you get down to it. The reason people use them is to abstract up a layer so you can query the data and get the results you want instead of having to iterate through direct reads of a disk, then having to read, parse, and filter what you want from those reads. You could always write code to help do those things direct from disk, but you know what you have just written if you do so? A database!

FrankWilhoit · Answer

It turns out to be quite unreasonably difficult to put something aside for a time -- from milliseconds to decades -- and then go back and find it again. The difficulty is all in the "find", because things that anyone might care about finding have various different aspects that different people may care about. There is no middle ground between the electronic equivalent of a pile of papers on one's desk and the full capabilities of the relational model. Filesystems are somewhere in between; but there is no useful place in between.

Someone1234 · Answer

Because you'll slowly start building the individual pieces of the database over the file system, until you've just recreated a database. Database didn't spawn out of nothing, people were writing to raw files on disk, and kept on solving the same issues over and over (data definitions, indexes, relations, cache/memory management, locks, et al).So your question is: Why does the industry focus on reusable solutions to hard problems, over piece-meal recreating it every project? And when phased in that way, the answer is self-evident. Productivity/cost/ease.

ksherlock · Answer

Historically, Unix (and many other operating systems) stored file names as an unsorted list so using the FS as KV store had O(N) lookup times whereas a single-file hashed database like dbm, ndbm, gdbm, bdb, etc gave you O(1) access.
If you're using a relational DB, like SQL, as a relational database, then it gives you a lot the FS doesn't give you. If you're using a relational database as a key-value store, SQLite is 35% than the filesystem [1]
Perhaps one of the biggest users of the filesystem as a KV store is git -- (not an llm, I just wanted to use --) .git/objects/xx/xxxxx maps the sha1 file hash to the compressed data, splayed by the first 2 bytes. However git also uses a database of sorts (.git/objects/pack/....). To sum up the git pack-objects man page, it's more efficient.
1. https://www.sqlite.org/fasterthanfs.html

stanfordkid · Answer

I think there are a lot of good answers here, but it really comes down to the type of content being stored and access patterns.
A database is a data structure with (generally) many small items that need to be precisely updated, read and manipulated.
A lot of files don't necessarily have this access pattern (for instance rendering a large video file) ... a filesystem has a generic access pattern and is a lower level primitive than a database.
For this same reason you even have different kinds of database for different types of access patterns and data types (e.g Elasticsearch for full text search, MongoDB for JSON, Postgres for SQL)
Filesystem is generic and low-level, database is a higher order abstraction.

codegeek · Answer

Just like any other abstraction that helps you do things more efficiently. Database is an abstraction over the more crude file system. It is similar to asking the question "why not write direct assembly code instead of a programming language". The answer is the same.

moomoo11 · Answer

I do that a little bit, for simple cache where I don't need it to be always up to date but close enough and on a single instance only.Some of my horizontally scaled services have like 500mb disk.