Google Cloud’s Post

#latest announcements from Amazon Web Services (AWS) and Nvidia at the ReInvent conference in Las Vegas. It covers the following topics: Project Ceiba: A new Nvidia-built supercomputer that delivers 65 exaflops of AI performance and is the world’s first ARM-based AI supercomputer in the cloud. It will be used by Nvidia’s own research and engineering team to develop new AIs for various domains1. New VMs with Nvidia GPUs: AWS announced new virtual machines with Nvidia’s new H200, L40S, and L4 GPUs, which offer high-performance computing and AI capabilities for different workloads and budgets2. Graviton4: AWS’s latest CPU, which is based on Arm’s Neoverse V2 design and offers 50% more cores, 30% more speed, and 75% more memory bandwidth than Graviton3. It also supports AWS’s Elastic Fabric Adapter and Trainium2, its next-generation training chip. Trainium2: AWS’s new chip for AI training is four times faster than Trainium and can be deployed in EC2 UltraClusters of up to 100,000 chips3. It enables faster and more efficient training of large-scale models such as foundation models and large language models.

Semiconductor expert and former $AMD employee shares some valuable insights about the GPU market, $NVDA, and AI: - The expert thinks that even if the newer generations of GPUs improve performance by only 10-25% from your base compute, that is still enormous for many of the companies building AI models and a big reason to upgrade to the newest chip. - In terms of $META, he believes they will continue to buy the newest and highest-performing GPU for research. Still, for running some of the recommendation models in the background, the H100 and A100 might be enough, as you don't need the most cutting-edge GPUs to run some of the models. - The expert thinks the cycle for upgrading the GPUs is 3-4 years, so the back-of-the-napkin math of the AI accelerator market being around $250B per year would make sense. - When it comes to the demand for the $NVDA H100 in the hyperscallers, the experts said that the demand is very high and that AWS might not give you access if you only want to rent it for a week or less over the guys who are renting it for a month or two months. - Even though $AMZN, $GOOGL, $META, and $MSFT are all developing their own chips, the expert still thinks they have a few years to catch up, as many of these chips are still immature products. - He gives an interesting example of $GOOGL, where, for internal research, they use TPUs but still buy $NVDA GPUs to offer them on their public-facing cloud to entice customers and keep them happy on the cloud. - $NVDA's CUDA is one of the main reasons why $NVDA is so dominant. - Down the road, the expert still thinks that the hyperscalers and $META will be able to save a lot of money by bringing these chips in-house, but this will not mature overnight, as the software for programming these chips is not that simple.

Cloudflare will make it possible for any organization globally to start deploying AI models — powered by NVIDIA GPUs, networking and inference software — without having to worry about managing, scaling, optimizing, or securing deployment. Read more in our press release. https://lnkd.in/eAFGkTCn

0420 13032024 RAINB0W CORP Rainbow🌈 Networks Hyperscalar Cloud Services for Defense META ACCELERATES By the end of 2024, the company is aiming to grow its infrastructure build-out to include 350,000 Nvidia H100s as part of a portfolio that will feature compute power equivalent to almost 600,000 H100s. While the number of GPUs is the same, the two clusters differ in network infrastructure. Both solutions interconnect 400 Gbps endpoints, but Meta has built one with a remote direct memory access (RDMA) over converged Ethernet (RoCE) network fabric solution based on the Arista 7800 with Wedge400 and Minipack2 OCP rack switches. The other cluster features an Nvidia Quantum2 InfiniBand fabric. Additionally, both clusters have been built using Meta’s in-house open GPU hardware platform, Grand Teton, the company's GPU-based hardware platform to support large AI workloads. The follow-up of the Zion-EX platform, it contains 4x the host-to-GPU bandwidth, 2x the compute and data network bandwidth, and 2x the power envelope when compared to its predecessor. ❤️ 🧡 💛 💚 💙 💜 💓

Only a select set of companies have a large share of NVIDIA's H100 chip orders. H100 is the ninth generation of Nvidia’s data center GPU. The device is equipped with more Tensor and CUDA cores, and at higher clock speeds, than the A100. There’s 50MB of Level 2 cache and 80GB of familiar HBM3 memory, but at twice the bandwidth of the predecessor – reaching 3 TB/sec. H100 is the premier AI chip in the market and it is highly coveted for its performance: it promises up to 9x faster AI training and up to 30x faster AI inference in popular machine learning models over the previous generation’s A100, released just two years ago. The lead time of H100-based servers is from 36 to 52 weeks. Omdia, a market tracking company, believes that Meta and Microsoft are the largest purchasers of Nvidia's H100 GPUs. They procured as many as 150,000 H100 GPUs each, considerably more than the number of H100 processors purchased by Google, Amazon, Oracle, and Tencent (50,000 each). It is noteworthy that the majority of server GPUs are supplied to hyperscale cloud service providers. Server OEMs (Dell Technologies, Lenovo, Hewlett Packard Enterprise) cannot get enough AI and HPC GPUs to fulfill their server orders yet. That being said, most hyperscalers are now creating their own chips: custom silicon for AI, High Performance Compute, and video workloads. The customized server market is estimated to be $196B by 2027. This inclination towards custom, application-optimized server configurations is set to become the norm as the cost-efficiency of building specialized processors is realized, with media and AI being the current front runners and other sectors like database management and web services expected to join the movement. What are your thoughts on companies' appetite for AI chips like H100? #Nvidia #AIInfrastructure #ServerInnovation #ProprietaryTechnology #FutureOfComputing #h100 #generativeai Data: Omdia, Data Center Knowledge, Yahoo Finance.