Slurm GPU Widget: A macOS Desktop Monitor for HPC GPU Queue Status

Before a GPU job is submitted, the number of idle GPUs in each partition must be confirmed. The conventional procedure consists of connecting to the VPN, logging in via SSH, running scontrol and squeue, and tallying the output by hand. Slurm GPU Widget encapsulates this procedure in a background collection script and keeps the occupancy of each partition permanently on display in two forms, an Übersicht desktop card and a SwiftBar menu-bar item; a single refresh takes approximately 0.7–1.3 s, and the default refresh period is 15 s. The project is open-sourced under the MIT License: Yu-Xiang-Yang/slurm-gpu-widget.

1 Background and Motivation

GPU clusters typically organise resources by partition, and partitions differ substantially in GPU model, node count and load. Before submitting a job, the following three questions must be answered:

• which partitions still have idle GPUs;
• the maximum number of idle GPUs on a single node, which determines whether a multi-GPU job can start immediately;
• the elapsed run time, time limit and queue status of the user’s own jobs.

For clusters accessed through a VPN, the conventional procedure for obtaining this information is inconvenient in three respects:

• Lengthy procedure: the three steps of VPN connection, SSH login and command execution must be completed in sequence and repeated for every query;
• Raw output lacks aggregate values: scontrol show node prints one line per node, with the total GPU count in the Gres field and the allocated count in the AllocTRES field, which must be parsed and summed;
• Node states must be filtered: nodes in states such as DOWN and DRAIN cannot be scheduled, and counting them in the total overestimates the available resources.

The aim of Slurm GPU Widget is to automate this procedure and present the results in a persistent interface, so that the resource state can be read without logging in to the cluster.

2 Feature Overview

Figure 1 Übersicht desktop card interface (simulated data, rendered from the repository’s mock/ data and front-end source with SHOW_MY_JOBS=1 and JOB_GPU_USAGE=1 enabled). The upper part shows, for each partition, the numbers of running, pending and own jobs, GPU and memory occupancy bars, and the maximum number of idle GPUs on a single node; the lower part, “My jobs”, lists the job ID, name, elapsed time / time limit, and the memory usage and utilisation of each GPU.

The main features are as follows:

3 System Architecture

The project consists of one collection script and two front ends; the data flow is as follows:

Tunnelblick (VPN state detection / automatic connection)
        │
        ▼
cluster-status.sh ── ssh (BatchMode, ControlMaster) ──▶ login node: scontrol / squeue
        │  local awk aggregation
        ├──── JSON ─────────────▶ Übersicht desktop card
        └──── --swiftbar-stream ▶ SwiftBar menu bar

Both front ends call the same collection script, cluster-status.sh, and they can be installed individually or together. The collection script supports four invocation modes:

The JSON structure is the interface contract between the collection layer and the presentation layer; replacing either side does not affect the other.

4 Data Collection Method

The collection procedure comprises five steps:

1. VPN state: osascript -e 'tell application "Tunnelblick" to get state of first configuration where name = "…"'.
2. Data retrieval: ssh -o BatchMode=yes $SSH_HOST "scontrol show node -o; squeue -h -o '%P|%T|%u'"; the two commands share a single SSH round trip.
3. GPU details (optional, GPU_DETAIL=1): the login node runs nvidia-smi --query-gpu=index,name,memory.used,memory.total,utilization.gpu on each GPU node in parallel, with the per-node timeout bounded by GPU_DETAIL_TIMEOUT. On clusters with pam_slurm_adopt enabled, only nodes on which the user has running jobs accept connections; the remaining nodes fall back to the statistics reported by Slurm.
4. Own jobs: squeue -u $USER -t R provides the job ID, name, elapsed time, time limit, nodes and GPUs per node.
5. Local aggregation: performed by awk, according to the rules below.

Aggregation rules:

• CPU usage is taken from CPUAlloc/CPUTot;
• The total GPU count is parsed from the Gres field, supporting formats such as gpu:a100:8(S:0-1), gpu:4 and gpu:2g.20gb:12,gpu:1g.20gb:4; the allocated count is taken from gres/gpu=N in AllocTRES (the field is omitted when the count is zero);
• The Partitions field is split on commas; a node belonging to multiple partitions is counted separately in each partition but only once in the global GPU total;
• Nodes in the DOWN, DRAIN, FAIL, MAINT or NOT_RESPONDING states are excluded from the available capacity;
• Host memory usage is computed as RealMemory − FreeMem, because AllocMem is always 0 on clusters that do not schedule memory.

4.1 Maximum Idle GPUs per Node

The gpu_max_free_node field in the output is the maximum number of idle GPUs (or MIG instances) on a single node; this value gives the upper bound on the size of a single-node job that can start immediately. In Figure 1, the gpu partition has 10/16 idle, but at most 8 idle GPUs on a single node; the gpu-v100 partition has 0/4 idle and at most 0 idle GPUs on a single node. The global idle count and the per-node idle count must be interpreted together: the former reflects the total, whereas the latter determines whether a multi-GPU job can obtain the required node at once.

4.2 GPU Usage of Jobs

With JOB_GPU_USAGE=1, srun --jobid=<id> --overlap -N1 -n1 nvidia-smi … is executed for each running job (up to 8) within its existing resource allocation. On clusters with ConstrainDevices=yes, this command sees only the GPUs held by the job, so no additional index mapping is required and no new resources are requested. This path is not restricted by pam_slurm_adopt, but each query is recorded in sacct as a job step (<jobid>.0, <jobid>.1, etc.); it is therefore disabled by default, and its query frequency is limited by JOB_GPU_INTERVAL (default 30 s), which keeps the number of job steps generated by the widget bounded.

The lower part of Figure 1 shows the display of this feature: job 123456 (train_llm, 2 GPUs) has been running for 1:23:45 with a time limit of 2 days; GPU0 uses 59.8/80 GB of memory at 97% utilisation, and GPU1 uses 39.3/80 GB at 88% utilisation. The memory bars are coloured by proportion (green below 60%, yellow for 60%–90%, red at 90% and above), and the elapsed time is incremented locally every second between two cluster queries.

5 Refresh Mechanism and Performance

Three measures are taken with respect to performance:

• Connection reuse: SSH enables ControlMaster=auto and ControlPersist=300; reusing the connection eliminates the handshake overhead, which the source comments quantify as a reduction from approximately 1 s to approximately 0.3 s; a complete refresh takes approximately 0.7–1.3 s.
• Query merging: node and queue information are retrieved together within the same session.
• Overlap suppression: the collection script uses a directory lock (mkdir) to prevent concurrent runs; if the previous collection has not yet finished (for example, because of a slow VPN or SSH response), the previous result is returned directly. A lock that has persisted for more than 90 s is regarded as the result of an abnormal process exit and is removed.

6 Installation and Configuration

6.1 Prerequisites

1. Password-less SSH: the widget runs in the background and cannot respond to interactive prompts, so key-based authentication is mandatory. If the key has a passphrase, UseKeychain yes and AddKeysToAgent yes must be added to ~/.ssh/config; a jump host can be declared via ProxyJump.
2. Non-interactive VPN: the Tunnelblick credentials must be stored in the macOS Keychain. VPNs that require a one-time password cannot be connected automatically; in that case AUTO_CONNECT=0 should be set.
3. Automation permission: on the first run, macOS asks whether Übersicht or SwiftBar may control Tunnelblick; this permission must be granted for automatic reconnection.

6.2 Installation Steps

Install either front end and launch it once (to create its plugin directory), then run:

git clone https://github.com/Yu-Xiang-Yang/slurm-gpu-widget.git
cd slurm-gpu-widget
bash install.sh
open -e ~/.config/cluster-widget/config

After editing the configuration, verify the collection script; JSON output indicates that the configuration is valid:

~/.config/cluster-widget/cluster-status.sh
~/.config/cluster-widget/cluster-status.sh --swiftbar

Running install.sh again updates the scripts and widgets but does not overwrite an existing configuration file.

6.3 Configuration Options

The configuration file is located at ~/.config/cluster-widget/config (template config.example); environment variables given on the command line take precedence over the file settings. The commonly used options are as follows:

To view all partitions and node details temporarily, the configuration can be overridden on the command line without affecting the configuration file:

GPU_ONLY=0 SHOW_NODES=1 PARTITIONS=gpu,compute ~/.config/cluster-widget/cluster-status.sh --swiftbar

For large partitions (approximately 100 nodes), SHOW_NODES=1 should be combined with PARTITIONS to limit the card height.

7 Output Format

The JSON object output in the default mode has the following structure (excerpt):

{
  "lang": "en", "cluster": "Demo", "time": "13:58:32",
  "vpn": "CONNECTED", "error": "",
  "gpu_used": 10, "gpu_total": 20,
  "partitions": [{
    "name": "gpu", "nodes": 3, "idle_nodes": 1, "down_nodes": 1,
    "cpu_used": 48, "cpu_total": 128, "gpu_used": 6, "gpu_total": 16,
    "gpu_max_free_node": 8, "running": 2, "pending": 1, "mine": 0, "mig": false,
    "model": "NVIDIA A100-SXM4-80GB", "mem_used": 1236992, "mem_total": 3072000
  }]
}

The field conventions are as follows:

• Memory values are in MiB;
• node_list appears only with SHOW_NODES=1;
• The gpus array of each node is non-empty only with GPU_DETAIL=1 and when the node accepts the query;
• With SHOW_MY_JOBS=1, a my_jobs array is appended to the top-level object, with each job containing elapsed_s and limit_s (UNLIMITED is recorded as −1); the top-level epoch records the collection time for local timekeeping.

8 Testing Without a Cluster

The repository’s mock/ directory provides synthetic scontrol and squeue output, allowing collection and front-end rendering to be verified in an environment without a cluster:

MOCK_DIR=mock bash cluster-status.sh
MOCK_DIR=mock SHOW_NODES=1 CLUSTER_NAME=Demo bash cluster-status.sh --swiftbar

Figure 1 was rendered from simulated data: the collection script was run with MOCK_DIR=mock, USER=testuser, SHOW_MY_JOBS=1, JOB_GPU_USAGE=1 and PARTITIONS=gpu,gpu-v100, and the resulting JSON was rendered by the Übersicht front-end source in the repository. The job names, memory figures and times in the figure are therefore synthetic values from the mock/ directory and do not correspond to a real cluster.

9 Design Trade-offs

10 Frequently Asked Questions

11 Extension to Other Schedulers

Only the remote commands and the awk parsing in the collection script are specific to Slurm. The PBS counterparts are pbsnodes -a and qstat, and the LSF counterparts are bhosts and bqueues. Once these two parts are replaced, neither the JSON structure nor the two front ends require any change.

12 Project Repository

GitHub: https://github.com/Yu-Xiang-Yang/slurm-gpu-widget (MIT License, with Chinese and English READMEs).