Dear friends,
There’s a lingering misconception that building with generative AI is expensive. It is indeed expensive to train cutting-edge foundation models, and a number of companies have spent billions of dollars doing this (and even released some of their models as open weights). But as a result, it’s now very inexpensive to build a wide range of AI applications.
The AI stack has several layers, shown in the diagram below. Here are the lower layers, from the bottom up:
- Semiconductors. Nvidia has been a huge benefactor in this space. AMD’s MI300 and forthcoming MI350 are also strong alternatives to the Nvidia H100 and the delayed Blackwell chips.
- Cloud. AWS (disclosure: I serve on Amazon’s board of directors), Google Cloud, and Microsoft Azure make it easy for developers to build.
- Foundation models. This includes both proprietary models such as OpenAI’s and Anthropic’s, and open weights models such as Meta’s Llama.
The foundation model layer frequently appears in headlines because foundation models cost so much to build. Some companies have made massive investments in training these models, and a few of those have added to the hype by pointing out that paying lots for compute and data would lead (probably) to predictably better performance following scaling laws.
This layer is also currently hyper-competitive, and switching costs for application developers to move from one model to another are fairly low (for example, requiring changes to just a few lines of code). Sequoia Capital’s thoughtful article on “AI's $600B Question” points out that, to justify massive capital investments in AI infrastructure (particularly GPU purchases and data center buildouts), generative AI needs to get around $600B of revenue. This has made investing at the foundation model layer challenging. It’s expensive, and this sector still needs to figure out how to deliver returns. (I’m cautiously optimistic it will work out!)
On top of this layer is an emerging orchestration layer, which provides software that helps coordinate multiple calls to LLMs and perhaps to other APIs. This layer is becoming increasingly agentic. For example, Langchain has helped many developers build LLM applications, and its evolution into LangGraph for building agents has been a great development. Other platforms such as Autogen, MemGPT, and CrewAI (disclosure: I made a personal investment in CrewAI) are also making it easier to build agentic workflows. Switching costs for this layer are much higher than for the foundation model layer, since, if you’ve built an agent on one of these frameworks, it’s a lot of work to switch to a different one. Still, competition in the orchestration layer, as in the foundation model layer, seems intense.
Finally, there’s the application layer. Almost by definition, this layer has to do better financially than all the layers below. In fact, for investments at the lower layers to make financial sense, the applications had better generate even more revenue, so the application vendors can afford to pay providers of infrastructure, cloud computing, foundation models, and orchestration. (This is why my team AI Fund focuses primarily on AI application companies, as I discussed in a talk.)
Fortunately, because of the massive investments in foundation models, it’s now incredibly inexpensive to experiment and build prototypes in the applications layer! Over Thanksgiving holiday, I spent about one and a half days prototyping different generative AI applications, and my bill for OpenAI API calls came out to about $3. On my personal AWS account, which I use for prototyping and experimentation, my most recent monthly bill was $35.30. I find it amazing how much fun you can have on these platforms for a small number of dollars!
By building on widely available AI tools, AI Fund now budgets $55,000 to get to a working prototype. And while that is quite a lot of money, it’s far less than the billions companies are raising to develop foundation models. Individuals and businesses can experiment and test important ideas at reasonable cost.
Keep learning!
Andrew