How Do AI Companies Get Their Training Data?

AI companies obtain their training data through six main channels: crawling the open web, licensing content from publishers and platforms, commissioning human and expert data, buying or licensing records that companies already hold, building simulated environments and synthetic data, and utilizing their own users' data. Crawling the open web is the largest and most prevalent channel, with one monthly Common Crawl archive holding 2.17 billion web pages.

Licensing content from publishers and platforms is also growing, with Reddit signing a $203.0 million data license deal. Commissioning human and expert data is another rising channel, with companies paying experts and contractors an average of $60 million per month. Buying company records, such as email, chat, tickets, and CRM data, is gaining traction as well, with Spirit Airlines' archive recently sold for $5 million to $12.5 million.

Building environments and synthetic data for training models is the fastest-growing channel, with 89 vendors selling environments at varying price points. Lastly, using users' own data, such as posts, chats, and uploads on platforms like LinkedIn, Meta, and X, is expanding, although the cost is typically regulatory compliance. The mix of these channels is shifting, with licensing, commissioning expert data, buying company records, and synthetic data generation becoming more prominent, while crawling the open web faces challenges such as an increasing number of sites blocking AI crawlers.