Amazon Web Services (AWS Machine Learning Blog) · GPU Partitioning

SageMaker Inference GPU partitioning (MIG)

AWS has added GPU partitioning to Amazon SageMaker Inference, using NVIDIA Multi-Instance GPU to split one physical GPU into hardware-isolated partitions with fixed compute and memory shares. It works on endpoints that use inference components, so several models can run on one GPU, and a partition type can be chosen per model. The example given is an ml.p4d.24xlarge, where eight A100s divided into three partition types yield 24 partitions. The feature is configured through two optional API fields, and endpoints that omit them behave as before.