While building my first private LLM inference platform on Kubernetes powered by an NVIDIA RTX 3090, I discovered that managing a GPU requires different troubleshooting habits compared to managing a traditional server.

When an NVIDIA GPU does not behave as expected, the first questions are usually the same:

is Linux detecting the hardware correctly?
is the NVIDIA driver loaded?
is the GPU responding properly?
which application is currently using the resources?
are there any errors reported by the Linux kernel?

The goal of this article is not to cover every existing NVIDIA command, but to share the essential commands I use to check, monitor, and troubleshoot an NVIDIA GPU on Linux.

Prerequisites

To use the commands presented in this article, you need:

a Linux machine ;
one or more NVIDIA GPUs installed ;
an NVIDIA driver installed to use nvidia-smi ;
terminal access.

Results may vary depending on the GPU model, the NVIDIA driver version, the Linux distribution, and the server hardware configuration.

1. Verify that Linux detects the GPU

Before checking the NVIDIA driver, the first step is to verify that the operating system can properly detect the hardware.

lspci -nn | grep -i nvidia

NVIDIA GPUs generally use the PCI Express (PCIe) interface to communicate with the system. The -nn option also displays the PCI hardware identifiers.

This command is particularly useful:

after installing a new GPU card;
after hardware maintenance;
when a GPU is no longer visible;
when an application cannot detect the GPU.

2. Check the general GPU status

nvidia-smi

nvidia-smi is the main tool provided by NVIDIA to monitor and diagnose GPUs. It is usually the first command executed when an issue occurs. It provides a global overview of the GPU:

GPU model;
NVIDIA driver version;
temperature;
GPU utilization;
memory usage;
power consumption;
processes using the GPU.

3. List available GPUs

On a server equipped with multiple GPUs, it is often useful to quickly retrieve the list of available devices.

nvidia-smi -L

This command displays a simplified list of GPUs visible to the NVIDIA driver. Each GPU has:

a local identifier;
a UUID (Universally Unique Identifier).

The UUID allows a GPU to be uniquely identified, especially in multi-GPU environments.

This command is particularly useful for:

multi-GPU servers;
virtualized environments;
hardware inventory operations.

4. Retrieve detailed GPU information

When nvidia-smi is not enough, it is possible to retrieve a much more detailed report.

nvidia-smi -q

This command displays a detailed report about the GPU. The -q (query mode) option displays available detailed information.It includes:

hardware characteristics;
memory information;
power information;
temperature;
PCIe information;
driver status;
detected events.

For example, to display only memory-related information:

nvidia-smi -q -d MEMORY

5. Monitor GPU activity in real time

A single command execution only provides a snapshot of the GPU at a specific point in time.

nvidia-smi -l 1

The -l (loop) option enables automatic refresh mode. The command refreshes the output every second. It allows monitoring:

GPU utilization changes;
memory consumption;
temperature variations;
power consumption changes.

6. Extract only the required metrics

The complete output of nvidia-smi contains a lot of information. For automation purposes or when feeding scripts, it is often better to select only the required metrics.

nvidia-smi \
--query-gpu=name,utilization.gpu,memory.used,memory.total,temperature.gpu,power.draw \
--format=csv

This command returns structured output containing:

GPU name;
GPU utilization;
used memory;
total memory;
temperature;
power consumption.

7. Quickly observe GPU activity with dmon

This command is particularly useful on servers with multiple GPUs when you need a quick overview of GPU activity.

nvidia-smi dmon

dmon (Device Monitor) provides a compact view of GPU metrics. It allows monitoring:

GPU utilization;
memory usage;
power consumption;
temperature.

8. Identify processes using the GPU

nvidia-smi pmon

pmon (Process Monitor) allows monitoring processes using GPUs. It helps identify:

the process PID;
the GPU being used;
associated activity.

Depending on the NVIDIA driver version, some features may vary.

9. Check loaded NVIDIA modules

A GPU can be physically present but unusable if the required kernel modules are not properly loaded.

lsmod | grep nvidia

lsmod displays the modules currently loaded in the Linux kernel. A kernel module is a software component dynamically loaded by Linux to extend kernel functionality. For NVIDIA, common modules include:

nvidia
nvidia_uvm
nvidia_drm
nvidia_modeset

This command is useful after a kernel update, an NVIDIA driver update, or when nvidia-smi returns an error.

10. Analyze NVIDIA errors in Linux logs

When the issue is not obvious, system logs become essential.

journalctl -k -b | grep -iE "nvidia|nvrm|xid"

Options used:

-k: display only kernel messages;
-b: limit messages to the current boot session.

This command searches for NVIDIA-related events. It can reveal:

driver errors;
loading issues;
GPU resets;
hardware-related events.

Understanding Xid events

Xid events are generated by the NVIDIA driver when a GPU-related error is detected. They can be related to:

a software issue;
communication problems;
unexpected hardware behavior.

They should always be analyzed in the context of the overall system.

Conclusion

Managing an NVIDIA GPU on Linux does not require dozens of different tools.With a few essential commands, it is possible to:

verify that Linux correctly detects the hardware;
validate that the NVIDIA driver is working properly;
monitor resource usage;
identify active processes;
analyze system errors.

nvidia-smi remains the central tool, but it becomes much more effective when combined with standard Linux utilities such as lspci, lsmod, and journalctl.

These commands provide a solid foundation for anyone responsible for maintaining Linux servers or workstations equipped with NVIDIA GPUs.