How Nvidia Came to Own AI Compute: From CUDA to the H100 Shortage

Data-center revenue of $10.32 billion, up 171 percent in a year

Second-quarter fiscal 2024 data-center revenue more than doubled from the prior quarter to a record $10.32 billion, up 141 percent sequentially and 171 percent from a year earlier, as H100 systems shipped to cloud providers and consumer-internet companies. It was the quarter the demand shift first showed in Nvidia's numbers.

• NVIDIA Newsroom · Aug 23, 2023 · Second-quarter revenue was a record $10.32 billion, up 141% from the previous quarter and up 171% from a year ago.

result of NVIDIA begins shipping Hopper-based H100 GPUs on Nvidia Data Center GPUs: Hopper, Blackwell and Vera Rubin.

Nvidia reports "surging demand" and says it is significantly increasing supply

With first-quarter fiscal 2024 data-center revenue at a record $4.28 billion, Jensen Huang said the computer industry was going through two transitions at once, accelerated computing and generative AI, and that Nvidia was significantly increasing supply to meet surging demand. The guidance that followed became the clearest public marker of the H100 shortage.

• NVIDIA Newsroom · May 24, 2023 · “The computer industry is going through two simultaneous transitions — accelerated computing and generative AI,” said Jensen Huang, founder and CEO of NVIDIA.

A100 in full production: Ampere unifies training and inference, adopted by every major cloud

Nvidia said the first Ampere GPU, the A100, was in full production and shipping worldwide, with up to 20 times the performance of its predecessors and adoption by the top cloud providers and server makers. The A100 was the chip the large language models of 2020 to 2022 were trained on, which is why the H100 launched into an installed base that already ran on Nvidia.

• NVIDIA Newsroom · May 14, 2020 · NVIDIA today announced that the first GPU based on the NVIDIA® Ampere architecture, the NVIDIA A100, is in full production and shipping to customers worldwide.

led to Nvidia announces the Hopper architecture and H100 GPU at GTC 2022 on Nvidia Data Center GPUs: Hopper, Blackwell and Vera Rubin.

AlexNet: a deep network trained on two GPUs wins ImageNet by a wide margin

Krizhevsky, Sutskever and Hinton reported a 60-million-parameter convolutional network that cut ImageNet's top-5 error to 18.9 percent, far below the previous state of the art, and credited "a very efficient GPU implementation" for making the training feasible. The result is the moment deep learning and Nvidia GPUs became tied together in research practice.

• NeurIPS 2012 proceedings · Dec 3, 2012 · On the test data, we achieved top-1 and top-5 error rates of 39.7% and 18.9% which is considerably better than the previous state-of-the-art results. The neural network, which has 60 million parameters and 500,000 neurons, consists of five convolutional layers, some of which are followed by max-pooling layers, and two globally connected layers with a final 1000-way softmax. To make training faster, we used non-saturating neurons and a very efficient GPU implementation of convolutional nets.