Dear friends,
I’ve noticed a trend in how generative AI applications are built that might affect both big companies and developers: The gravity of data is decreasing.
Data gravity is the idea, proposed by IT engineer Dave McCrory in 2010, that data, or activity around data, attracts and creates more data. With traditional software workloads, data gravity is strong. If you have terabytes of data stored in a particular cloud, the cost to transmit it elsewhere for processing is high. So many teams pick a cloud such as AWS, Azure, or Google Cloud and build on it.
However, for many generative AI applications, the cost of processing is much greater than the cost of transmission. This weakens data gravity because data is more weakly bound to the cloud provider or data center where it’s stored, so it’s more practical to build systems that send packets to different servers all over the internet.
Let’s say transmitting 1GB of data costs $0.10. 1GB of text might correspond to about 250 million inputs tokens (if we average four characters per token), which costs about $125 to process using the relatively inexpensive gpt-3.5-turbo-0125 model. (With gpt-4-0125-preview, the cost would be 20x higher.) The cost of processing the data is significantly higher than the cost of transmission. Also, given the computationally intensive nature of using an LLM to read and generate tokens, the latency is high enough that sending your text or image tokens across the internet usually doesn’t add that much further latency.
This means that, even if we’re building software primarily on a particular cloud provider, it’s still quite feasible to transmit LLM prompts to OpenAI, Anthropic, Anyscale, or Together.ai — or, for that matter, AWS, Azure, or Google Cloud — to get a response. The incentive to build only on a single, monolithic cloud platform is lower than before.
This situation has implications for stakeholders:
- For developers, it means we’re increasingly assembling AI applications from lots of SaaS providers all across the internet, and stitching their services together.
- For CIOs, it’s creating headaches in terms of managing where their data goes and how to maintain lists of trusted vendors.
- For the big cloud companies, it’s changing the basis of competition, since the generative AI portions of their customer workloads look quite different from traditional software workloads.
- For new tool developers, it’s creating new opportunities for users to use their services, even if they aren’t bundled into one of cloud environments.
To be clear, many applications have large traditional software components (that serve up a websites, maintain databases, and so on) as well as new generative AI components (say, a chatbot built on top of the traditional infrastructure). My remarks here apply only to the generative AI portion, and the competitive dynamics of the traditional software components haven’t changed much.
Further, as new types of AI components emerge, I expect their gravity to evolve as well. For example, right now it appears reasonably easy to change LLM providers; if you’ve built a system on one LLM, it’s annoying but not impossible to switch to a different LLM provider. In comparison, shifting databases is much harder, and once you’ve stored a lot of data in one vector database, the complexity of migrating to a different one can be high.
The gravity of data has been a fundamental tenet of cloud computing, and a major factor of competition for many companies. Decreasing data gravity is decreasing is a complex, exciting trend that will affect many developers and businesses.
Keep learning!
Andrew
P.S. Our new short course “Knowledge Graphs for RAG” is now available, taught by Andreas Kollegger of Neo4j! Knowledge graphs are a data structure that’s great at capturing complex relationships among data of multiple types. They can improve the context you pass to the LLM and the performance of your RAG applications by enabling more sophisticated retrieval of text than similarity search alone. In this course, you’ll build a knowledge graph from scratch and see how it improves chat applications by providing both text and graph data to an LLM. Sign up here!