In a now-taken-down blog post summarizing an event with Sam Altman, Altman revealed that he doesn’t believe that ChatGPT plugins have product-market fit (outside of the browsing plugin) and won’t be coming to the API soon. Why? A few hypotheses (not mutually exclusive).
Correct concept but not good enough yet:
- GPT-4 picks the wrong plugins or fails to chain together multiple calls reliably. This is the major problem with most agent or plugin frameworks — they don’t work. They might be able to initiate a call to an external API but are so brittle that they often break or misbehave quickly. Whether or not we need bigger models or more specific ones (i.e., fine-tuned), I’m not sure.
- The killer-app plugins have yet to be developed.
- Larger context windows mean more plugins can be called simultaneously, unlocking more powerful workflows.
The concept is not correct:
- Altman alludes to this in the post (paraphrased by the author) — a lot of people thought they wanted their apps to be inside ChatGPT, but what they really wanted was ChatGPT in their apps.
- LLMs will have “horizontal” extensions, such as connecting them to a web search or a database, but they will not call generic APIs through an App Store-like interface. Each use case will need a specific interface.
Correct concept, but not the right implementation:
- Chat is not the right UX for plugins. If you know what you want to do, it’s often easier to just do a few clicks on the website. If you don’t, just a chat interface makes it hard to steer the model toward your goal.
- Too expensive to serve at the current price — GPT-4 has a quota of 25 messages every 3 hours. This might not be enough for users to reach the “aha moment.”
- Not the right UX in some other way (e.g., having users choose plugins ahead of time, OpenAPI specification not the correct interface).
- Can’t aggregate enough demand with a plugin system that only works with a single model and needs broader adoption (potentially open-source). Building a successful app store is hard — and often doesn’t lead to the monopolies observed by Apple’s iOS App Store (see necessary conditions for an app store monopoly).