More capable coding agents will dramatically change the way we build and configure our databases, suggesting that long standing trade-offs may no longer apply. Given an ontology of a domain, can it now be possible for everyone to “3D print” their own custom DBMS that outperforms off-the-shelf solutions?
A database is a sociotechnical construct. Social, because its users enter into an evolving agreement on how they want to make sense of their particular slice of the world, recorded in the form of schemas, constraints, and access policies. Technical, because the implementing technology — the database management system — is what then enforces this shared understanding and powers its applications. Use of a database guarantees a base level of organizational alignment, but its implementation choices also limit the scope of the decisions an organization can make on top of it.
Take the canonical example of a SaaS provider calculating a churn KPI to (begrudgingly) present to their board. All their data on subscriptions, product analytics, and billing is in their Snowflake warehouse. The tradeoff is this: the centralized schema ensures that the KPI is computed based on an accurate, agreed upon notion of active customer, and translated into the correct dollar amount. However, analyzing the individual paths customers took through the product in order to explain the increased churn can’t be done; the level of granularity of this schema has limited the scope of decision making.
Every time an organization makes a database decision it starts with intent, “why do we care?”, that is followed by “what do we mean?”, and “how can we retrieve the necessary information?”†.
The story of databases is that of the increasing separation of the what from the how. First, by abstracting physical storage behind logical data representations, then by replacing procedural navigation code with query languages. All made practical by cost-based query optimization and techniques like query compilation, vectorized evaluationEverything You Always Wanted to Know About Compiled and Vectorized Queries But Were Afraid to AskThe query engines of most modern database systems are either based on vectorization or data-centric code generation. These two state-of-the-art query processing paradigms are fundamentally different in terms of system structure and query execution code. Both paradigms were used to build fast systems. However, until today it is not clear which paradigm yields faster query execution, as many implementation-specific choices obstruct a direct comparison of architectures. In this paper, we experimentally compare the two models by implementing both within the same test system. This allows us to use for both models the same query processing algorithms, the same data structures, and the same parallelization framework to ultimately create an apples-to-apples comparison. We find that both are efficient, but have different strengths and weaknesses. Vectorization is better at hiding cache miss latency, whereas data-centric compilation requires fewer CPU instructions, which benefits cacheresident workloads. Besides raw, single-threaded performance, we also investigate SIMD as well as multi-core parallelization and different hardware architectures. Finally, we analyze qualitative differences as a guide for system architects.VLDB proceedingsView paper, and hardware acceleration. Where this declarative approach fell short, databases grew tools and knobs that let users close the gap: batching, statically typed schemas, indices, optimizer hints, stored procedures, and many more.
This separation was an insurance policy against change. The real world is not a fixed target. Understanding evolves, priorities shift, and ambitions grow. A business can’t adapt successfully if every single secular change it experiences breaks all existing applications. Neither can it benefit from improvements at the hardware layer if applications hard-code a specific way of accessing data. So in essence, databases today try (not always successfully) to balance these competing goals of rigidity and flexibility to change.
The limits of my data model are the limits of my world
This sensitivity to change historically forced database designers to trade performance for flexibility, in an attempt to prevent their users from changing vendors every N years.
Flexibility determines how far down the road users can still change their mind about the data they’re dealing with, what it means, how it can change, and what questions they want to ask of it. Continuing with the example from earlier: many decisions start out by aggregating independent observations (time spent on a page, clicks of a button), before realizing that these observations really can’t be treated as independent at all. A “but why?” is all it takes for a clickstream problem to turn into a graph problem. And any one-off question of a static dataset will eventually experience the gravitational pull of a real time dashboard, should the answer prove valuable.
Performance is how much of this data users can have, and how many such questions they may ask.
For the vast majority of organizations (exactly those that buy rather than build databases) the trade is highly asymmetric in the same direction: inability to answer a new question the business has is a deal breaker, whereas, up to a minimal acceptable level, reduced performance is merely a nuisance and source of inefficiency.
Most modern database designs therefore converged on a similar sweet spot: lock-in a data model (relations, graphs, documents, key/value pairs) and a small set of appropriate data structures, but then offer flexible computational lego blocks on top. The operators of relational algebra, pattern matching and path finding in graph databases, or full-text search on documents are all examples of computational primitives that can be recombined to solve new use cases. Users can change how they view the world, as long as they stick to the same language with which to describe their view of the world.
Arguably most iconic and commercially successful among those database silhouettes are implementations of the relational model on B+ trees, behind a SQL interface. But for a more recent example look no further than turbopuffer: vectors arranged in a suitable data structure that supports approximate and exact similarity search.
Of course, users still do pay a tax wherever their domain does not fit the underlying data model exactly, and that tax is again asymmetric: storing a bunch of independent documents in a relational database is merely wasteful, while emulating a complex supply chain on a key-value store is a non-starter. As soon as performance is acceptable, more expressive data models become the rational choice (which explains why relational databases over time have won back most specialized use cases†).
By and large, this trade has served the industry well. Users find ways to squeeze their applications into general purpose data models running on general purpose systems, in exchange for alignment, reliable operation, good-enough performance, and not having to squander their best years on writing a database.
What has been much rarer is for a commercial database product to lock-in a domain model, i.e. to provide an interface that works with concrete business objects like Address, Sale, or Shipment instead of abstract structures like Document, Relation, or Graph. But in the rare cases where this is economically justifiable and done well, the results can be spectacular, showing us just how far below their speed-of-light most database workloads actually run.
TigerBeetle and the rewards of specialization
“Any customer can have a car painted any color that he wants so long as it is black”
- Henry Ford
TigerBeetle is an example of a database design that does lock-in a specific domain model: financial transactions. Paraphrasing Mr. Ford, TigerBeetle users are welcome to model the world any way they want, so long as that world is financial transactions, and the model is double-entry bookkeeping. In return, TigerBeetle provides significantly better performance (they claim 1000x) on this use case, compared to general purpose alternatives like Postgres.
Oversimplifying a bit, here is the world according to TigerBeetle:
- A ledger groups the accounts that can transact with each other.
- A transfer moves an amount from one account to another account on the same ledger.
- An account's balance is the difference between everything transferred out of it and everything transferred into it.
- An account can be marked so that transfers which would push its balance negative are rejected.
- Accounts and transfers are never changed or deleted; mistakes are corrected with new transfers.
It’s worth dwelling for a while on the benefits that this “reduction of scope” of the database offers.
Fixing the domain in this way lets the system use compact representations. In the context of the domain, a transfer requires only a handful of numbers to describe: (id, source, destination, amount) plus a little bit of metadata. TigerBeetle squeezes it all into just 128 bytes. There is no need for client requests to describe or reference a particular schema (there is only one), the maximum size of every value is known, and the system can lay out all values optimally in memory and in durable storage. Note also how there is no SQL to parse, and no arbitrary statements to capture in the WAL.
A general purpose database also has to describe processes in general terms like ”read row x”, “write row y”, “commit”, and “abort”. Business logic such as “What funds are required to accept a transfer?” is held in the application, invisible to the database. A transfer involving reading and writing from different rows would look something like this:
BEGIN
LOCK source row
LOCK destination row
READ source balance
READ destination balance
SEND balances to the application
RECEIVE decision on whether the transfer is valid
IF NOT VALID ABORT
WRITE new source balance
WRITE new destination balance
RELEASE source row
RELEASE destination row
COMMIT
ENDBecause only the application can validate a transfer, the system has to keep both rows locked while waiting for the business logic from across the network to run. In the common case of many transfers sharing the same counterparty (e.g. a bank), this effectively serializes the execution.
Thanks to its built-in knowledge of the rules and constraints governing transfers, TigerBeetle can process transfers without holding locks through a network roundtrip. Clients send a single TRANSFER(id, source, destination, amount) statement, which the system can process and validate entirely on its own, without any communication back to the application.
Finally, TigerBeetle doesn’t just process a single transfer at a time. Its clients automatically batch pending transfers locally, and can send an atomic chain of up to 8000 at a time. Full batches dramatically increase the amount of work that can be communicated in a single network hop, durably committed with a single WAL append, and replicated with a single consensus step.
This is not an afterthought, and not just a question of whether the database accepts a list of statements in a single request. Automatic, uncoordinated batching is again uniquely enabled by the fixed domain model! Postgres clients can mutate the database arbitrarily at any point in time, their requests are not homogeneous instances of the same operation. They often need to wait for a response before they can formulate the next request. The only way TigerBeetle clients interact with the system is by leaving little post-it notes with transfer details scribbled on them, that the system is free to process in bulk.
As far as commercially available databases go, TigerBeetle is arguably the canonical example of how rigidly scoping a database to a particular domain yields performance improvements and offers a fundamentally better experience for the user.
Towards the 3D printed database
When a database product ties itself firmly to one particular domain, it allows its designers to make much stronger assumptions at every layer of the architecture, unlocking deep efficiencies not readily available to general purpose solutions. The history of databases on the other hand teaches us to fear change, to avoid making strong assumptions that can’t be changed later…after all, many startups pivot dramatically!
Additionally, domains that are shallow and stable enough to warrant specialization are often adjacent to the core business, and thus get bought instead of built. No need to build a system, just offload the technical complexity to a vendor who specializes in precisely this domain (perhaps they will have their own domain-specialized database). Salesforce is a database for customer relationship management, just as Workday is a database for HR, Jira one for software issues, and so on. Few would argue that it’s worth building this “infrastructure” (if you can call it that) in house.
Until recently, the idea of building your own domain-specific database would have been prohibitively complex and expensive. TigerBeetle, for example, is the result of almost a decade of research and cracked systems engineering by a uniquely assembled and run team. And they’ve found a rare domain to play in: differentiated, business critical, and stable (as long as the world doesn’t spontaneously switch to triple-entry bookkeeping). In-house databases have historically been the purview of only the world’s largest companies†; and even then, many of them just use Oracle.
But today we have a new weapon, coding agents, attacking the cost of software development. Can they tip the balance in favour of extreme codesign? An assembly line for databases, churning out “3D-printed” TigerBeetle’s for every domain?
No matter how capable the agents or how cheap the tokens, the AI database factory will need a precise work order, covering (1) a description of the target domain, (2) a characterization of the expected workload, and (3) a set of service-level agreements.
A description of a domain is called an ontology. Think of it as a CAD model of a business, encoding the concepts, relationships, rules, and constraints the business operates with. Admittedly this is not as sexy as a CAD model of say a fighter jet, but it is just as useful. Earlier we saw a simplified version of the financial transaction ontology at the heart of TigerBeetle:
- A ledger groups the accounts that can transact with each other.
- A transfer moves an amount from one account to another account on the same ledger.
- An account's balance is the difference between everything transferred out of it and everything transferred into it.
- An account can be marked so that transfers which would push its balance negative are rejected.
- Accounts and transfers are never changed or deleted; mistakes are corrected with new transfers.
This ontology was provided in natural language. For more precision, modeling languages like OWL, RDF, or PyRel exist.
# A Transfer is an immutable record of a financial transaction between two accounts on the
# same ledger. It debits exactly one account and credits exactly one account.
Transfer = Concept("Transfer", identify_by={"id": Integer})
Transfer.debit_account = Property(f"{Transfer} debits {Account}")
Transfer.credit_account = Property(f"{Transfer} credits {Account}")
Transfer.amount = Property(f"{Transfer} moves {Integer}")
Transfer.ledger = Property(f"{Transfer} is on {Ledger}")
# Balance is derived depending on the account type.
Account.balance = Property(f"{Transfer} debits {Account}")
where(
Account.code.normal_balance == DEBIT
).define(Account.balance(Account.debits_posted - Account.credits_posted))
where(
Account.code.normal_balance == CREDIT
).define(Account.balance(Account.credits_posted - Account.debits_posted))
# Constraints rule out invalid states.
where(Transfer).require(Transfer.debit_account.ledger == Transfer.credit_account.ledger)
where(Transfer).require(Transfer.debit_account != Transfer.credit_account)
where(Transfer).require(Transfer.amount > 0)A sufficiently precise ontology provides the what:
- Types in the ontology that do not derive from others are precisely the inputs into the system. For TigerBeetle, those are Account and Transfer.
- Derivation rules specify the effect that input changes should have, e.g. how account balances are computed from credits and debits.
- Constraints rule out invalid state transitions, such as negative account balances.
Ontology in hand, we can now characterize the expected workload: how many do we expect to have of each type and how fast is the volume growing? How are relationships distributed? Which parts of the ontology are going to be read frequently? How will demand fluctuate over a given period of time? Of course, real telemetry can be used to improve such an operational profile over time. There is no industry standard for writing operational profiles yet, but inspiration can be taken from the field of workload synthesisRedbench: Workload Synthesis From Cloud TracesWorkload traces from cloud data warehouse providers reveal that standard benchmarks such as TPC-H and TPC-DS fail to capture key characteristics of real-world workloads, including query repetition and string-heavy queries. In this paper, we introduce Redbench, a novel benchmark featuring a workload generator that reproduces real-world workload characteristics derived from traces released by cloud providers. Redbench integrates multiple workload generation techniques to tailor workloads to specific objectives, transforming existing benchmarks into realistic query streams that preserve intrinsic workload characteristics. By focusing on inherent workload signals rather than execution-specific metrics, Redbench bridges the gap between synthetic and real workloads. Our evaluation shows that (1) Redbench produces more realistic and reproducible workloads for cloud data warehouse benchmarking, and (2) Redbench reveals the impact of system optimizations across four commercial data warehouse platforms. We believe that Redbench provides a crucial foundation for advancing research on optimization techniques for modern cloud data warehouses.arXiv:2511.13059v2View paper.
The third and final piece is an understanding of what the business values, captured as a set of service-level agreements under which an agent is free to make trade-offs. Systems like Google’s Napa demonstrate how even a simple utility model, that lets users define compute budgets and trade latency against freshness, is enough to effectively serve consumers with vastly different requirements. Search and other business-critical applications with high margins will gladly run with an unlimited compute budget, as long as it ensures extremely low latency and fresh results. Analytical or exploratory use cases can accept slightly stale results at much lower cost.
The primitive is the product
These three pieces — an ontology, an operational profile, and a set of SLAs — form the precise work order that an agentic database factory needs. The shape of the ontology suggests the data model to use, the SLAs and read/write patterns narrow the space of appropriate data structures and storage tiers. Derivation rules and constraints are baked into the transaction handling code.
However, databases are critical infrastructure that can’t just be one-shotted into production. Domain specialization can unlock incredible performance, but not to the point of reinventing foundational primitives like B+ trees, tiered buffer managers, sorting algorithms, and governance models from scratch over and over again, expending the energy to verify, tune, and secure them. As we’re increasingly finding out with agents, the primitive is the product.
More importantly, the mere existence of a database factory doesn’t change the fact that businesses will want to buy the assembled product, not oversee the factory floor.
Instead, the infrastructure layer should evolve a corresponding set of declarative APIs and services, that all speak in the same terms: ontologies, operational profiles, and associated SLAs. From “Provision three 64-CPU nodes running Postgres” to “Provision {1000} {Transfer}s / second at a p99 of {100ms}”.
And none of the above is specific to operational databases. Already today, projects like SynnoDB claim impressive speedups by putting a workload router and synthesis machinery in front of established OLAP databases.
The story of databases is one of separating the what from the how, of moving from procedural to declarative, all to protect applications against inevitable change.
Coding agents and the falling cost of software development should encourage us to push the declarative approach further, letting users pay for flexibility at deploy time, not at run time. For that, agents will need a description of the business (the ontology), a characterization of the workload, and an understanding of what the business values.
And they will want to deploy to an infrastructure layer that speaks in those terms as well.
