Mark Zuckerberg revealed that Meta is training its Llama 4 models using an extensive cluster of over 100,000 H100 AI GPUs, emphasizing its advancements in capabilities. Unique to Llama 4 is its free access for researchers and companies, contrasting with proprietary models like GPT-4o. However, the immense power demand—potentially over 370GWh annually—raises concerns about sustainability and energy sourcing, prompting competitors like Google and Microsoft to explore nuclear energy solutions for their AI operations.
—
Mark Zuckerberg a révélé que Meta forme ses modèles Llama 4 en utilisant un vaste cluster de plus de 100 000 GPU AI H100, soulignant ses avancées en capacités. Llama 4 se distingue par son accès gratuit pour les chercheurs et les entreprises, contrairement aux modèles propriétaires comme GPT-4o. Cependant, la demande d’énergie immense—potentiellement plus de 370 GWh par an—soulève des inquiétudes quant à la durabilité et à l’approvisionnement énergétique, poussant des concurrents comme Google et Microsoft à explorer des solutions nucléaires pour leurs opérations d’IA.
During a recent earnings call, Mark Zuckerberg revealed that Meta is utilizing a training cluster that exceeds 100,000 H100 AI GPUs for its Llama 4 models, claiming it is larger than any reported by competitors. Although Zuckerberg did not disclose specific capabilities of Llama 4, he described it as featuring “new modalities,” “stronger reasoning,” and operating “much faster.” This advancement is significant as Meta strives to keep pace with industry giants like Microsoft, Google, and Musk’s xAI in the race to create next-gen AI large language models (LLMs).
Meta is not alone in operating an AI training cluster of this magnitude; Elon Musk also launched a comparable cluster in late July, dubbing it a “Gigafactory of Compute” with ambitions to expand it to 200,000 AI GPUs. Nonetheless, Meta has indicated that it anticipates having over half a million H100-equivalent AI GPUs by the close of 2024, suggesting a robust current capacity for Llama 4 training.
In a distinctive strategy, Meta makes its Llama models available for free, promoting collaboration among researchers, organizations, and companies. This contrasts sharply with other offerings like OpenAI’s GPT-4o and Google’s Gemini, which require API access for use. However, Meta has imposed certain restrictions on Llama’s usage, especially concerning commercial applications, and has not publicly disclosed its training methodologies. Regardless, the “open source” status of Llama could position it favorably in the evolving AI landscape, mirroring the success seen with open-source Chinese AI models which rivaled GPT-4o and Llama-3 in various benchmarks.
Concerns over power consumption
The immense computational power necessary for these developments translates to a significant power requirement, with each modern AI GPU potentially consuming up to 3.7MWh of energy annually. A cluster with 100,000 GPUs could, therefore, use a staggering 370GWh each year, sufficient to power over 34 million average U.S. households. This raises pressing questions about how companies will procure such vast energy supplies, especially since new energy sources take time to be integrated. Zuckerberg himself acknowledged that power constraints could hinder AI advancements.
For instance, Elon Musk utilized large mobile generators to supply power for his 100,000 GPU compute setup in Memphis. In contrast, Google has seen an increase in greenhouse gas emissions by 48% since 2019, falling short of its carbon goals. The former CEO of Google even suggested abandoning climate targets to allow AI companies to accelerate their development, intending to leverage AI solutions for environmental challenges later on.
Nevertheless, Meta executives were evasive when questioned about their strategy for powering such an extensive computing cluster. Meanwhile, competitors like Microsoft, Google, Oracle, and Amazon are turning towards nuclear energy, exploring investments in small modular reactors or refurbishing old nuclear facilities to ensure a stable energy supply for their future operations.
While these nuclear solutions will take time to implement, equipping AI data centers with small nuclear reactors could significantly alleviate the power demands placed on the national grid by these energy-intensive clusters.
Considérations sur la consommation d’énergie
La puissance computationnelle immense nécessaire pour ces développements se traduit par une demande d’énergie significative, chaque GPU AI moderne pouvant consommer jusqu’à 3,7 MWh d’énergie par an. Un cluster de 100 000 GPU pourrait donc utiliser une quantité stupéfiante de 370 GWh chaque année, assez pour alimenter plus de 34 millions de ménages américains à la moyenne. Cela soulève des questions pressantes sur la façon dont les entreprises vont se procurer de telles quantités d’énergie, surtout que de nouvelles sources d’énergie prennent du temps à être intégrées. Zuckerberg lui-même a reconnu que des contraintes énergétiques pourraient entraver les avancées de l’IA.
Par exemple, Elon Musk a utilisé de grands générateurs mobiles pour alimenter son système de calcul de 100 000 GPU à Memphis. En revanche, Google a vu ses émissions de gaz à effet de serre augmenter de 48% depuis 2019, ne respectant pas ses objectifs en matière de carbone. L’ancien PDG de Google a même suggéré d’abandonner les objectifs climatiques pour permettre aux entreprises d’IA d’accélérer leur développement, prévoyant d’utiliser les solutions d’IA développées pour relever plus tard les défis environnementaux.
Néanmoins, les dirigeants de Meta ont éludé la question de leur stratégie pour alimenter un cluster de calcul aussi vaste. Pendant ce temps, des concurrents comme Microsoft, Google, Oracle et Amazon se tournent vers l’énergie nucléaire, explorant des investissements dans des petits réacteurs modulaires ou rénovant