The Off-Key Bard
Technology

What Is a Database, and Why Not Just Store Everything in Files or Spreadsheets?

Or: why "just put it in Excel" eventually stops working. The spreadsheet didn't fail. The requirements changed.

Or: Why "Just Put It in Excel" Eventually Stops Working

Imagine running a small business with twenty customers. Keeping their names, phone numbers, and purchase history in a spreadsheet is fast, visual, and entirely manageable — add a few columns, sort by name, maybe use a filter, and everything holds together fine. Now imagine the business grows to 200,000 customers, thousands of orders a day, several employees entering information at the same time, a website checking inventory constantly, and customers editing their own accounts while all of that happens simultaneously. At that point the question stops being "where do we put the data?" and becomes "how do we organize, search, update, protect, and coordinate all of this without it turning into chaos?" That's the problem a database is built to solve.

More Than Just a Big File

At the simplest level, a database is an organized collection of information — but that definition is almost too broad to be useful, since a text file or a spreadsheet is technically organized too. What makes a real database system powerful isn't merely that it stores data. It's that it also provides rules and machinery for finding that data, updating it safely, managing relationships between pieces of it, handling many simultaneous users, and recovering cleanly when something goes wrong. The software that does all of this is called a database management system, or DBMS — PostgreSQL, MySQL, SQL Server, and SQLite are all examples. A relational database can look deceptively familiar at first: you have rows, columns, and values, same as a spreadsheet, and a table named Customers with columns for ID, name, and email looks exactly like a worksheet. The resemblance holds at small scale and breaks down fast after that, because a spreadsheet is designed for a person to work directly with a grid, while a database is designed for software and many people to reliably manage structured information at scale.

Why Identity and Relationships Need Rules

Suppose two customers happen to share the name John Smith. If a system used names as the only identifier, which one placed order 4728? Databases solve this with unique identifiers called primary keys — one John Smith might be customer 18472, the other 63391, and their database identities stay distinct even though their names are identical. That idea becomes essential the moment information starts connecting across tables. If Customer 18472 places three orders, you could store their name and address inside every order record, but then changing their phone number means hunting down and updating every copy. Instead, the Orders table just stores a CustomerID that points back to the Customers table — the customer's information exists in one place, and orders reference it rather than duplicating it. That reference is a relationship, and the column making it is called a foreign key. The database can enforce that relationship directly: try to create an order for a customer ID that doesn't exist, and the system can reject it outright, a guarantee known as referential integrity. The database isn't just storing numbers — it's enforcing rules about what those numbers are allowed to mean.

This is also why relational design leans on normalization — organizing information so each fact lives in one appropriate place rather than being copied everywhere. One giant spreadsheet containing every customer, product, and order detail in a single sheet means the same customer or warehouse information repeats across thousands of rows, and a single change can require updating all of them — duplication that creates real opportunities for inconsistency. That said, normalization isn't a religion: an order often deliberately stores the price a customer actually paid, rather than always referencing the product's current price, so that changing a product from $19.99 to $24.99 doesn't retroactively alter old receipts. Good database design is about understanding what each piece of data actually means, not eliminating duplication on principle.

Asking Questions With SQL

Relational databases are commonly queried with SQL, Structured Query Language, which lets an application or person ask for exactly what they need — every order placed by one customer, every product with fewer than ten units in stock, total sales by month — without manually scanning a file byte by byte. You describe what you want; the database figures out how to get it. Separating data into multiple tables raises an obvious follow-up question, though: how do you put it back together? A JOIN lets the database combine related rows from different tables using the relationship between their keys, so an order record referencing CustomerID 18472 can be paired back up with that customer's actual name from the Customers table on demand.

None of this works efficiently on huge tables without indexes — additional structures that let the database locate a specific value without scanning every row, the same way a book's index lets you jump straight to the pages about a specific topic rather than reading cover to cover. Indexes aren't free, though: every indexed column has to be updated whenever a row changes, so more indexes speed up certain reads while slowing down writes and consuming extra storage — a tradeoff rather than a universal improvement. For the same reason, a complex SQL query gets handed to a query optimizer inside the database engine, which estimates which execution plan — which table to start from, which index to use, in what order — will actually be cheapest given the data's size and shape, which is why two queries that look like they're asking the same question can perform very differently.

Handling Many Things at Once

Databases earn their keep most clearly when several things happen simultaneously. Imagine an online store has exactly one graphics card left, and two customers click "buy now" within the same instant; without coordination, both programs could read "quantity: 1," both conclude the item is available, and both create an order — now two units have sold from a stock of one. This is where transactions come in: a database can group related operations — subtract $100 from account A, add $100 to account B — into a single logical unit, so that either the entire transfer happens or none of it does, even if the server crashes exactly between the two steps.

The classic guarantee here is summarized as ACID — Atomicity, Consistency, Isolation, and Durability — and Wikipedia's overview of the concept lays out each piece plainly: atomicity means a transaction behaves as all-or-nothing, consistency means the database's own rules stay satisfied before and after, isolation governs how simultaneous transactions avoid corrupting each other's work, and durability means a committed transaction survives even an immediate crash afterward. PostgreSQL's own documentation on concurrency control describes one of the major modern approaches to isolation directly: rather than locking data and forcing every reader to wait behind every writer, PostgreSQL uses Multi-Version Concurrency Control, maintaining multiple snapshots of data so that reading never blocks writing and writing never blocks reading, letting many transactions proceed with less interference than a simple locking scheme would allow. Durability leans on similar engineering — a technique called write-ahead logging records enough information about an intended change in a durable log before the system considers the transaction fully committed, so recovery after a crash can replay exactly what was in progress rather than guessing.

Databases Expect Failure

This points to one of the biggest philosophical differences between a simple file and a serious database system: a database is designed around the assumption that things will eventually go wrong — programs crash, drives fail, networks disconnect, transactions collide, machines lose power — and it includes mechanisms specifically built to preserve consistency and recover from exactly those failures. That doesn't make backups unnecessary, though; transactional safety protects against a mid-operation crash, not against someone accidentally deleting an entire table or a bug issuing a perfectly valid but disastrous command. Important databases are also commonly replicated across multiple servers, so a copy can take over if the primary fails and read traffic can be spread out — though replication introduces its own wrinkle, since an update takes time to travel to a replica, meaning a query against a lagging copy can briefly appear to have lost a change that's actually just still in transit.

Beyond Tables: Other Database Models

Everything so far describes the relational model, but it's far from the only one. Collectively, many non-relational approaches get grouped under the label NoSQL — not because SQL is forbidden, but because the underlying structure doesn't rely solely on tables and foreign keys. A document database stores each record as something closer to a JSON object, with nested fields in one place rather than split across related tables, which fits naturally with applications already thinking in JSON. A key-value database works like a giant dictionary — give it a key, get back a value — and Redis's own documentation describes exactly this kind of data-structure server, built around extremely fast lookups for things like caching and session data rather than complex relationships. A graph database models data as nodes and relationships directly, which suits questions that are fundamentally about connections — how two people in a social network are linked, for instance — more naturally than a relational join chain would. None of these replace the relational model so much as specialize for a different access pattern; the right choice depends on the shape of the problem, not on which technology sounds newest.

Where the Danger Lives

Letting an application build SQL queries out of raw user input is one of the oldest and most damaging mistakes in software, known as SQL injection. OWASP's cheat sheet on preventing it describes the mechanism directly: when user-supplied text gets concatenated straight into a query string, an attacker can craft input that changes the query's actual meaning rather than simply supplying a value — turning a lookup for one username into an operation the developer never intended. The fix OWASP recommends is parameterized queries, or prepared statements, which separate the query's structure from the data filling it in, so the database always treats user input as a literal value no matter what it contains, rather than as part of the command itself. The broader lesson extends past SQL specifically: never treat untrusted input as though it might be executable instructions.

The Bard's Take

At first glance, a database can look like an unnecessarily complicated spreadsheet — rows, columns, values, so why not just use Excel? For small problems, sometimes you should. The difference shows up once the data becomes important enough that you need actual guarantees: several people and programs updating information at once without corrupting it, relationships between records staying valid, finding one record among millions without scanning everything, a bank transfer either finishing completely or not happening at all, and data surviving a crash after the system already told someone the transaction succeeded. That's when a database stops looking like a fancy table and starts looking like what it really is — an engine for managing shared truth.

Tables give information structure. Primary keys give records identity. Foreign keys connect related information and let the system reject nonsense before it's stored. Indexes make searches fast, at a real cost in storage and write speed. Transactions keep related changes together, and MVCC or locking lets many people work at once without standing in one line. Write-ahead logs help the whole thing survive a crash mid-sentence. And SQL gives applications a language for asking it all questions. Every time you check a bank balance, place an order, or open your contacts app, you're almost certainly asking a database something, somewhere behind the scenes — and a database is working very hard, invisibly, to make sure there's only one version of the truth waiting on the other side of that question.

Sources

← Back to the Desk — Technology