🥶 Take 5: GSP errors
GPU System Processor, это отдельный чип отвечающий за инициализацию и управление заданиями.
У него есть отдельная прошивка, и эта штука может отдельно ломаться.

❗Как правило, пробелемы с GSP приводят карту в inoperable state или hung / freeze state

GSP errors can be caused by either GSP firmware bugs [32] or
demanding workload. For example, Delta SREs observed that these
errors were highly correlated with heavy ML benchmarks, and
they suggested that GSP errors are high-impact errors whose recov-
ery requires manual node draining and reboots. Our propagation
analysis confirm that the GSP is a single point of failure on both
A100 GPUs in part because of their spontaneous nature and high
downstream impact (e.g., GPU hangs) on the GPU.

In fact, AWS recom-
mends disabling GSP for stability over performance benefits [4].

🚌 Take 6: GPU fallen Off the Bus
Это скорее типичный износ и перегрев.

GPU Fallen Off the Bus errors were
logged when the GPU driver could not reach the GPU over the
system bus. This error is an integration error often caused by a
loose GPU-motherboard connection or contact failure because of
thermal cycles [49]. Over 99% of the errors of this type lead to
similar errors in close successions and eventually put the GPU into
an error state.