OpenAI may be synonymous with machine learning now and Google is doing its best to get itself off the ground, but soon both may face a new threat: the rapid multiplication of open source projects that push the state of the art and leave corporations deep-pocketed but difficult to handle in their dust. This zerg-like threat may not be existential, but it will certainly keep dominant players defensive.
The notion is by no means new – in the fast-moving AI community, expect to see this type of outage on a weekly basis – but the situation was put into perspective for a while. widely shared document supposedly originated within Google. “We have no moat, and neither does OpenAI,” the memo says.
I won’t burden the reader with a lengthy summary of this interesting and perfectly readable piece, but the bottom line is that while the GPT-4 and other proprietary models have garnered most of the attention, and indeed the revenue, the head start that they have obtained with financing and infrastructure looks scarcer every day.
While the pace of OpenAI releases may seem blistering by the standards of ordinary major software releases, GPT-3, ChatGPT, and GPT-4 certainly snapped on their heels when compared to iOS or Photoshop releases. But they are still occurring on the scale of months and years.
What the memo notes is that in March, a leaked base language model of Meta, called LLaMA, was leaked in a fairly rough fashion. Inside weeks, people playing on penny-a-minute laptops and servers had added core features like instruction wrapping, multiple modalities, and reinforcement learning from human feedback. OpenAI and Google were probably digging into the code as well, but they didn’t, couldn’t, replicate the level of collaboration and experimentation that was going on on subreddits and Discords.
Could it really be that the titanic computing problem that seemed to present an insurmountable obstacle, a moat, for challengers is already a relic of a different era of AI development?
Sam Altman already pointed out that we should expect diminishing returns when we throw parameters into the problem. Bigger isn’t always better, sure, but few would have guessed that smaller was instead.
GPT-4 is a Walmart, and nobody really likes Walmart
The business paradigm that OpenAI and others are pursuing right now is a direct descendant of the SaaS model. It has some high-value software or service and offers carefully controlled access via an API or something similar. It’s a simple and proven approach that makes a lot of sense when you’ve invested hundreds of millions to develop a single monolithic but versatile product as a large language model.
If GPT-4 generalizes well to answer questions about precedent in contract law, great; it doesn’t matter that a large part of his “intellect” is dedicated to being able to repeat the style of every author who has ever published a work in the English language. GPT-4 is like a Walmart. no one really wants to go there, so the company makes sure there is no other option.
But customers are starting to wonder, why am I walking down 50 junk aisles to buy a few apples? Why am I contracting for the services of the largest and most widely used AI model ever created if all I want to do is exercise some intelligence to compare the language of this contract with a couple hundred others? At the risk of torturing the metaphor (not to mention the reader), if GPT-4 is the Walmart you go to for apples, what happens when it opens a fruit stand in the parking lot?
It didn’t take long for the AI world to run a large language model, in heavily truncated form of course, on (appropriately) a Raspberry Pi. For a company like OpenAI, its jockey Microsoft, Google, or anyone else in the AI-as-a-service world, in fact, it trumps their entire business premise: that these systems are so hard to build and run that they have to do to you. In fact, it appears that these companies chose and designed a version of AI that fits their existing business model, not the other way around!
Once upon a time you had to offload the computation involved in word processing to a mainframe computer – your terminal was just a screen. Of course, that was a different era, and it was a long time ago that we were able to adapt the entire application on a personal computer. That process has happened many times since our devices repeatedly and exponentially increased their computing power. These days, when something has to be done on a supercomputer, everyone understands that it’s just a matter of timing and optimization.
For Google and OpenAI, the time has come much sooner than expected. And they weren’t the ones doing the optimization, and they may never do it at this rate.
Now, that doesn’t mean they’re just out of luck. Google didn’t get where it is by being the best, at least not for long. Being a Walmart has its benefits. Businesses don’t want to have to find a custom solution that gets the job done 30% faster if they can get a decent price from their current provider and not rock the boat too much. Never underestimate the value of momentum in business!
Sure, people are iterating on LLaMA so fast that they’re running out of camelids to name them. By the way, I would like to thank the developers for an excuse to see hundreds of images of cute, griffon vicunas instead of working. But few enterprise IT departments are going to cobble together an implementation of Stability’s ongoing open-source derivative of a quasi-legal leaked Meta model on top of OpenAI’s simple and effective API. They have a business to run!
But at the same time, I stopped using Photoshop years ago for image editing and creation because open source options like Gimp and Paint.net have gotten soooo good. At this point, the argument goes in the other direction. How much do you pay for Photoshop? No way, we have a business to run!
What clearly worries the anonymous authors at Google is that the distance between the first situation and the second is going to be much shorter than anyone thought, and there doesn’t seem to be anything anyone can do about it.
Except, the memo argues: accept it. Open, publish, collaborate, share, engage. As they conclude:
Google should establish itself as a leader in the open source community, taking the lead by cooperating with the broader conversation, rather than ignoring it. This will likely mean taking some cumbersome steps, like posting model weights for small ULM variants. This necessarily means giving up some control over our models. But this commitment is inevitable. We cannot hope so much to drive innovation as to control it.