As chief info officers race to undertake and deploy synthetic intelligence, they ultimately encounter an uncomfortable fact: Their IT infrastructure is not prepared for AI. From widespread GPU shortages and latency-prone networks to quickly spiking energy demands, they encounter bottlenecks that undermine efficiency and enhance prices.
“An inefficient AI framework can tremendously diminish the worth of AI,” says Sid Nag, vp of analysis at Gartner. Provides Teresa Tung, international knowledge functionality lead at Accenture: “The shortage of high-end GPUs is a matter, however there are different elements — together with energy, cooling, and data center design and capacity — that impression outcomes.”
The takeaway? Demanding and resource-intensive AI workloads require IT leaders to rethink how they design networks, allocate sources and handle energy consumption. Those that ignore these challenges threat falling behind within the AI arms race — and undercutting enterprise efficiency.
Breaking Factors
Probably the most evident and extensively reported downside is a shortage of high-end GPUs required for inferencing and working AI fashions. For instance, extremely coveted Nvidia Blackwell GPUs, formally generally known as GB200 NVL-72, have been nearly impossible to find for months, as main corporations like Amazon, Google, Meta and Microsoft scoop them up. But, even when a enterprise can receive these items, the fee for a totally configured server can cost around $3 million. A cheaper model, the NVL36 server, runs about $1.8 million.
Whereas this will have an effect on an enterprise instantly, the scarcity of GPUs additionally impacts main cloud suppliers like AWS, Google, and Microsoft. They more and more ration sources and capability, Nag says. For companies, the repercussions are palpable. “Missing an sufficient {hardware} infrastructure that’s required to construct AI fashions, coaching a mannequin can develop into sluggish and unfeasible. It might additionally result in knowledge bottlenecks that undermine efficiency,” he notes.
GPU shortages are only a piece of the general puzzle, nonetheless. As organizations look to plug in AI instruments for specialised functions corresponding to pc imaginative and prescient, robotics, or chatbots they uncover that there’s a necessity for quick and environment friendly infrastructure optimized for AI, Tung explains.
Community latency can show significantly difficult. Even small delays in processing AI queries can journey up an initiative. GPU clusters require high-speed interconnects to speak at most pace. Many networks proceed to depend on legacy copper, which considerably slows knowledge transfers, in line with Terry Thorn, vp of business operations for Ayar Labs, a vendor that focuses on AI-optimized infrastructure.
Nonetheless one other potential downside is knowledge heart house and vitality consumption. AI workloads — significantly these operating on high-density GPU clusters — draw huge quantities of energy. As deployment scales, CIOs might scramble so as to add servers, {hardware} and superior applied sciences like liquid cooling. Inefficient {hardware}, community infrastructure and AI fashions exacerbate the issue, Nag says.
Making issues worse, upgrading energy and cooling infrastructure is difficult and time-consuming. Nag factors out that these upgrades might require a yr or longer to finish, thus creating extra short-term bottlenecks.
Scaling Good
Optimizing AI is inherently difficult as a result of the know-how impacts areas as various as knowledge administration, computational sources and person interfaces. Consequently, CIOs should resolve the way to strategy numerous AI tasks based mostly on the use case, AI mannequin and organizational necessities. This consists of balancing on-premises GPU clusters with totally different mixes of chips and cloud-based AI providers.
Organizations should think about how, when and the place cloud providers and specialty AI suppliers make sense, Tung says. If constructing a GPU cluster internally is both undesirable or out of attain, then it’s crucial to discover a appropriate service supplier. “It’s a must to perceive the seller’s relationships with GPU suppliers, what sorts of various chips they provide, and what precisely you’re getting access to,” she says.
In some circumstances, AWS, Google, or Microsoft might provide an answer by way of particular services and products. Nonetheless, an array of area of interest and specialty AI service corporations additionally exist, and a few consulting corporations — Accenture and Deloitte are two of them — have direct partnerships with Nvidia and different GPU distributors. “In some circumstances,” Tung says, “you will get knowledge flowing by way of these customized fashions and frameworks. You may lean into these relationships to get the GPUs you want.”
For these operating GPU clusters, maximizing community efficiency is paramount. As workloads scale, programs battle with knowledge switch limitations. One of many crucial choke factors is copper. Ayar Labs, for instance, replaces these interconnects with high-speed optical interconnects that scale back latency, energy consumption and warmth technology. The result’s higher GPU utilization but additionally extra environment friendly mannequin processing, significantly for large-scale deployments.
In actual fact, Ayar Labs claims a 10x decrease latency and as much as 10x extra bandwidth over conventional interconnects. There’s additionally a 4x to 8x discount in energy. Now not are chips “ready for knowledge moderately than computing,” Thorn states. The issue can develop into significantly extreme as organizations undertake advanced massive language fashions. “Growing the dimensions of the pipe boosts utilization and reduces CapEx,” he provides.
Nonetheless one other piece of the puzzle is mannequin effectivity and distillation processes. By particularly adapting a mannequin for a laptop computer or smartphone, for instance, it’s typically potential to make use of totally different mixtures of GPUs and CPUs. The end result could be a mannequin that runs sooner, higher and cheaper, Tung says.
Energy Performs
Addressing AI’s energy necessities can also be important. An overarching vitality technique might help keep away from short-term efficiency bottlenecks in addition to long-term chokepoints. “Vitality consumption goes to be an issue, if it isn’t already an issue for a lot of corporations,” Nag says. With out sufficient provide, energy can develop into a barrier to success. It can also undermine sustainability and enhance greenwashing accusations. He means that CIOs view AI in a broad and holistic manner, together with figuring out methods to cut back reliance on GPUs.
Establishing clear insurance policies and a governance framework round using AI can reduce the chance of non-technical enterprise customers misusing instruments or inadvertently creating bottlenecks. The danger is larger when these customers flip to hyperscalers like AWS, Google and Microsoft. “With out some steerage and route, it may be like strolling right into a sweet retailer and never realizing what to choose,” Nag factors out.
Ultimately, an enterprise AI framework should bridge each technique and IT infrastructure. The target, Tung explains, is “making certain your organization controls its future in an AI-driven world.”