Graphics Processing Units in AI Training

A graphics processing unit (GPU) is a processor designed to perform many similar numerical operations simultaneously. AI training repeatedly applies matrix and tensor operations during forward computation and backpropagation; GPUs accelerate this work through massive parallelism, high memory throughput and specialised arithmetic units.

Why AI training suits GPUs

Large neural networks contain many parameters and process large batches of examples. Their calculations can be divided into numerous largely independent operations, making them suitable for data-parallel computation.

• Matrix multiplication and convolution consist of repeated multiply-and-accumulate operations that can run concurrently across many GPU processing units.
• During backpropagation, GPUs calculate gradients for many parameters and training examples in parallel.
• Unlike CPUs, which optimise low-latency execution of varied tasks, GPUs devote more hardware to parallel numerical throughput.

Hardware features that provide acceleration

GPU architecture reduces the time required for each training iteration by combining parallel execution with rapid movement and specialised processing of tensor data.

• The SIMT model, or single instruction, multiple threads, allows many threads to execute similar instructions on different data elements.
• High memory bandwidth feeds model parameters, activations and gradients to processing units faster than typical general-purpose processors.
• Modern AI-oriented GPUs include specialised matrix units and support lower-precision formats, enabling more operations and lower memory use through mixed-precision training.

Scaling to very large models

When one GPU cannot hold or efficiently process a model, multiple GPUs are connected and trained cooperatively.

• In data parallelism, each GPU processes a different batch subset and the resulting gradients are synchronised.
• In model parallelism, parameters, layers or tensor operations are divided across GPUs.
• Actual speed gains are limited by memory capacity, communication overhead, synchronisation and the proportion of work that cannot be parallelised.

How UPSC asks this