Can you build LLM infrastructure before LLM applications?
Kubernetes might have once been a new project, but it solved old problems. It was inspired by Borg, Google’s internal cluster manager, which had been in production for a decade. And the moment for Kubernetes would not have existed without the innovation from Docker. And before that, the applications motivated the design of cluster management and containerization.
The problem is amplified in MLOps — I reflected on my work in MLOp infrastructure in 2018 in lessons from the last AI cycle,
The hypothesis that the next generation of startups would mimic the machine learning stacks of Uber and Airbnb was false. Infrastructure takes time to build (and time to sell to enterprises). By the time it was productionized and SaaS-ified, it was too late, and the paradigms had changed.
Knowing the correct APIs for LLM infrastructure a priori is nearly impossible. Instead, I imagine we’ll have to see real LLM-enabled applications deployed to better understand the right horizontal infrastructure. The cycle goes something like this:
LLMs enable new applications, which in turn, require new infrastructure. That infrastructure allows for new applications. Planes, then airports. Cars, then roads. Lightbulbs, then electric grids.
That apps and infrastructure evolve in responsive, rather than distinct, cycles was a thesis put forth by the folks at USV in “Myth of the Infrastructure Phase.” While the initial post centered around crypto infrastructure, I think it might be even more important in AI infrastructure.
Even with the framework in mind, there are still many unanswered questions:
- What counts as an application, and what counts as infrastructure?
- Do the incumbent apps (e.g., Notion or Adobe) that add LLM features to their existing products count as new LLM applications?
- Do we have to wait for an LLM-native product (something that could only exist with LLMs) to motivate future LLM infrastructure?
- Are the underlying paradigms changing too quickly for the application/infrastructure virtuous cycle?
- Can open-source testbed of LLM infrastructure experimentation leapfrog this cycle?