The fast tempo of innovation and the proliferation of latest fashions have raised considerations about know-how lock-in. Lock-in occurs when companies turn into overly reliant on a particular mannequin with bespoke scaffolding that limits their skill to adapt to improvements. Upon its launch, GPT-4 was the identical price as GPT-3 regardless of being a superior mannequin with a lot increased efficiency. For the reason that GPT-4 launch in March 2023, OpenAI prices have fallen one other six occasions for enter information and 4 occasions for output information with GPT-4o, launched Could 13, 2024. In fact, an evaluation of this type assumes that technology is bought at price or a hard and fast revenue, which might be not true, and vital capital injections and destructive margins for capturing market share have possible backed a few of this. Nonetheless, we doubt these levers clarify all the advance features and value reductions. Even Gemini 1.5 Flash, launched Could 24, 2024, presents efficiency close to GPT-4, costing about 85 occasions much less for enter information and 57 occasions much less for output information than the unique GPT-4. Though eliminating know-how lock-in is probably not attainable, companies can cut back their grip on know-how adoption by utilizing industrial fashions within the quick run.
Avoiding lock-in dangers
In some respects, the bitter lesson is a part of this extra appreciable dialogue about lock-in dangers. We count on scaling to proceed, a minimum of for one more couple of interactions. Until you have got a specific use case with apparent industrial potential, or function inside a high-risk and extremely regulated trade, adopting the know-how earlier than the complete scaling potential is decided and exhausted could also be hasty.
In the end, coaching a language mannequin or adopting an open-source mannequin is like swapping a leash for a ball and chain. Both approach, you’re not strolling away with out leaving some pores and skin within the sport. It’s possible you’ll want to coach or tune a mannequin in a slender area with specialised language and tail data. Nonetheless, coaching language fashions includes substantial time, computational sources, and monetary funding. This will increase the chance for any technique. Coaching a language mannequin can price a whole bunch of hundreds to tens of millions of {dollars}, relying on the mannequin’s measurement and the quantity of coaching information. The financial burden is exacerbated by the nonlinear scaling legal guidelines of mannequin coaching, through which features in efficiency could require exponentially larger compute sources—highlighting the uncertainty and danger concerned in such endeavors. Bloomberg’s technique of together with a margin of error of 30 p.c of their computing price range underscores the unpredictable nature of coaching.