The Golden Data Era: Why Proprietary Data Became AI's Scarcest Asset

For most of the past decade, the raw material of artificial intelligence was treated as close to free. Large language models were trained on enormous scrapes of the public web, digitised books, forums and code, and the competitive race was framed almost entirely around compute: who could buy the most chips, build the largest clusters and run the longest training jobs. Data was the background condition, not the scarce input.

That framing is changing. As models have grown, the volume of text needed to train them efficiently has grown with them, and the public pool they draw on has not kept pace. The result is a quiet but important rebalancing of value across the AI supply chain. Compute is still expensive, but high-quality data, particularly data that is unique, well structured and legally clean, has moved from an afterthought to a strategic asset that laboratories now negotiate for, pay for and build companies around.

This shift matters well beyond the handful of frontier labs. It means that the archives, records and specialist knowledge sitting inside ordinary companies are increasingly an asset class in their own right. A publisher’s back catalogue, a platform’s years of user discussion, an engineering firm’s technical documentation or a clinic network’s de-identified records can now carry commercial value that did not exist five years ago.

The clearest quantitative picture comes from Epoch AI (opens in a new tab), an independent research group that tracks the inputs to AI progress. In its peer-reviewed study, Will we run out of data? (opens in a new tab), the team estimated the effective stock of quality, repetition-adjusted public human text at around 300 trillion tokens, with a 90% confidence range of 100 trillion to 1,000 trillion.

Set against that stock, Epoch modelled how quickly training datasets are growing. Its central finding is that if current trends continue, frontier models will be trained on datasets roughly equal in size to the entire available stock of public human text at some point between 2026 and 2032. The point estimate for a model trained in a compute-optimal way lands around 2028.

The timing moves earlier when developers “overtrain”, meaning they train smaller models on more data than strictly optimal to make them cheaper to run. Epoch notes that Meta’s Llama 3 70B was overtrained by a factor of about ten. At a 5x overtraining rate the stock would be fully used by around 2027, and at 100x as early as 2025. In other words, the commercial pressure to ship efficient models accelerates the arrival of the ceiling.

Epoch is careful to frame this as a constraint to be worked around rather than a wall. The same study points to several routes that extend the runway: better filtering of web data, training on the same data for multiple passes, synthetic data, and multimodal sources such as images and video. Each helps. None removes the underlying fact that genuinely new, high-quality human-generated data is now the input in shortest supply.

Epoch’s analysis also addresses the obvious next question: what about all the data that is not public? The team acknowledges that non-public data, such as private messages, internal documents and proprietary records, dwarfs the public web in volume. Its view is that this reserve is unlikely to be used at scale soon because of legal constraints and because it is fragmented across millions of separate owners.

Read from an owner’s side, that observation is the opportunity. Fragmentation is a barrier for a laboratory trying to source data one company at a time, but it is precisely what gives each individual owner leverage. A dataset that cannot be scraped, cannot be recreated and sits with a single rights holder is, by definition, scarce. The legal complexity that slows access is the same thing that rewards owners who arrive with documented rights and a clean chain of consent.

Epoch also makes a pointed observation about spending. Today, the money laboratories spend on compute dwarfs what they spend on data. The group notes that if data became the binding constraint, that ratio would be expected to reverse. The evidence from the past two years suggests the rebalancing has begun.

The first public evidence came from a company filing rather than a forecast. In its IPO registration statement (opens in a new tab) filed in February 2024, Reddit disclosed that in January of that year it had entered data licensing arrangements with an aggregate contract value of $203.0 million, running over two to three years, with at least $66.4 million expected to be recognised in 2024 alone. The same day, Google announced an expanded partnership (opens in a new tab) with Reddit that included access to its Data API.

Reddit was explicit about why this content commands a price. The filing describes more than a billion posts and over 16 billion comments accumulated through the end of 2023, across more than 100,000 active communities. It is a large, continually refreshed archive of real human conversation on almost every subject, and it cannot be recreated by anyone else.

Professional publishers followed the same path. Wiley (opens in a new tab) reported $44 million in AI content licensing revenue in its 2024 fiscal year, and Informa (opens in a new tab) told investors that AI partnerships for its Taylor & Francis academic business would generate more than $75 million in 2024. Shutterstock (opens in a new tab) reported $104 million in revenue from licensing its library for AI training in 2023 and set a target of $250 million by 2027.

Capital markets have moved even faster. In June 2025, Scale AI (opens in a new tab) announced a significant investment from Meta that valued the company at more than $29 billion; The New York Times reported (opens in a new tab) the investment at $14.3 billion for a 49% stake. Only a year earlier, Scale had raised $1 billion at a $14 billion valuation, according to CNBC (opens in a new tab). The step change reflects how central high-quality training data has become to the leading laboratories’ plans.

Scarcity alone does not make a dataset valuable. Laboratories pay for data that measurably improves a model, and that tends to come down to a handful of characteristics that owners can assess for themselves.

• UniquenessData that exists nowhere else, such as a proprietary archive or a closed community, cannot be substituted by scraping or synthetic generation.
• Depth of expertiseMaterial written or reviewed by specialists, such as clinicians, engineers, lawyers or analysts, teaches models reasoning they cannot learn from casual text.
• Structure and labelsRecords that are organised, consistently formatted and annotated reduce a buyer’s preparation cost and shorten the path to training.
• FreshnessContinually updated sources stay valuable over time, which supports recurring licences rather than one-off sales.
• Clean rightsDocumented ownership, consent and de-identification make data usable without legal risk, which is increasingly what laboratories pay a premium for.

The last of those traits is becoming the deciding one. As regulation and litigation have raised the cost of using data without permission, the value of a dataset now depends as much on what can be proven about it as on what it contains. A smaller archive with clear provenance can be worth more to a buyer than a larger one with open questions.

For most companies, data has historically been a cost centre: something to store, secure and occasionally analyse. The scarcity dynamic turns part of it into a potential revenue line. That does not mean every company should license its data, and many will rightly decide not to. It does mean that the decision is now a commercial one worth making deliberately, with a clear view of what the asset is and what it could earn.

The owners best placed to benefit share a pattern. They hold material that is genuinely theirs to license, they understand what is inside it, and they can describe it in terms a buyer can evaluate: volume, format, date range, subject coverage and the basis on which it was collected. Those that can also show how personal information is handled, and how consent was obtained, move to the front of the queue.

There is also a timing element. Epoch’s window of 2026 to 2032 describes the period in which demand for new, high-quality data is likely to be at its most intense. Licensing structures agreed in that window, particularly recurring arrangements tied to continually refreshed data, can create durable relationships with buyers who expect to keep training new models for years.

None of this requires an owner to give up control. The most valuable arrangements tend to be narrowly scoped: specific datasets, specific uses, defined terms and clear restrictions on onward use. Control, in a scarce market, is part of what is being priced.

A natural response to a shortage of human-written text is to have models write more of it. Synthetic data is now a standard part of how laboratories train and refine models, and Epoch’s research lists it among the main ways the industry can extend the life of the existing data stock. It is particularly useful for tasks where answers can be checked automatically, such as mathematics and code.

Synthetic data does not, however, replace the need for original human material. A model generating examples can only draw on what it has already learned, and the quality of what it produces depends on the quality of the human data used to train it and to check its output. In practice, that makes high-quality human data more important rather than less, because it becomes the reference point against which synthetic material is created, filtered and evaluated.

This is visible in where laboratories are spending. The fastest-growing suppliers in the market are not those generating synthetic text, but those supplying expert human work, the kind of careful, specialist judgement that synthetic pipelines need as an anchor. The more synthetic data the industry uses, the more it needs trustworthy human sources to keep that data grounded.

For owners of proprietary data, the implication is encouraging. A unique archive of real human work is valuable today as direct training material, and it remains valuable as synthetic methods mature, because it is exactly the kind of grounded, verifiable source those methods depend on.

The most visible buyers of training data are the frontier laboratories and the large technology companies that back them. Google’s partnership with Reddit, OpenAI’s agreements with publishers and Meta’s investment in Scale AI all sit in this group. Their needs are broad: large volumes of high-quality text, images, video and code, ideally refreshed over time.

Behind them is a growing second tier of buyers. Companies building models for specific industries need deep, narrow datasets in fields such as medicine, law, finance and engineering. Enterprise software providers adding AI features to their products need examples of how work is done in their customers’ domains. Each of these buyers values relevance over size, which opens the market to owners of focused, specialist collections that would not interest a frontier laboratory on volume alone.

This broadening of demand matters for pricing. A market with many buyers, each looking for something different, is one in which a well-described dataset can find the buyer that values it most. It also means that owners do not need to hold internet-scale archives to participate; they need data that is genuinely useful to someone building a model.

When a single licensing book can reach $203.0 million, as Reddit’s did, and when suppliers of training data are valued at tens of billions of dollars, the question of what a company’s data is worth stops being a technical matter. It becomes a strategic one, with implications for revenue, valuation, risk and competitive position.

Boards are increasingly asking three questions. What data do we hold that nobody else has? What would it be worth to the companies building AI models? And what would we need to do, in terms of rights, consent and preparation, to make it available on terms we are comfortable with? Companies that can answer those questions clearly are in a far stronger position than those discovering the opportunity only when a buyer calls.

Three trends look set to shape the market through the rest of the decade. First, laboratories will keep diversifying their sources, combining licensed human data with synthetic data and multimodal material. Synthetic data is useful, but its quality depends on the human data used to create and check it, which keeps original sources in demand.

Second, the market will become more organised. Specialist firms that source, prepare and verify data for laboratories have already become some of the fastest-growing companies in technology. As more owners enter the market, standard terms, pricing references and verification practices will develop, making transactions faster and more predictable for both sides.

Third, provenance will become a product feature. Regulations such as the European Union’s requirement for providers of general-purpose AI models to publish a summary of their training content push laboratories toward data they can describe and defend. Owners who can supply that documentation will find themselves selling not just data, but certainty.

The first era of AI was built on what was freely available. The next one is being built on what is owned. For companies holding unique, well-documented data, that is one of the more significant commercial openings of the decade.