Upper portion of the GDSight accepted manuscript, showing its title, authors, and abstract.

GDSight: Per-Operation Cross-Layer Attribution for GPUDirect Storage

Izzet Yildirim1   Xian-He Sun1   Anthony Kougkas1

1Illinois Institute of Technology

PDSW 2026 · Accepted

Paper coming soonCodeArtifact DOIBibTeX

Abstract

GPUDirect Storage moves data directly between NVMe and GPU memory, bypassing the host-I/O path that normally identifies which application operation caused device work. GDSight reconstructs that link for external tracing.

It traces cuFile, nvidia-fs, and NVMe with eBPF, then composes a time basis with an address basis to attribute device commands and bytes to individual operations, including asynchronous submissions for which temporal identity is absent or unsafe.

How it works

GDSight follows one read across the path, then attributes each device command using the strongest valid evidence.

cuFile operationoffset, size, correlation id
nvidia-fsGDS kernel path
NVMe commandsector and device bytes
Offline attributiontime plus address basis

The time basis is exact when a synchronous key is valid. The address basis maps file ranges through FIEMAP to handle asynchronous work without guessing.

What it finds

Across unmodified GDS software stacks, familiar observers can report health while per-operation attribution identifies the responsible operation class and a measured action.

91%

KvikIO bypass

Reads silently routed through POSIX rather than GDS cost 2.67× device bytes until the GDS path is restored.

4.00×

NIXL KV reads

A 4 KiB filesystem grid rounds 2048 B reads; layout coalescing reduces device traffic to 1.00×.

27%

HDF5 cache path

The raw-data chunk cache host-stages reads and consumes more host CPU than true GDS.

22×

Tail contention

Small-read p99 inflates under overlapping large device commands while median latency changes only 9%.

Reproducibility

The artifact contains the tracer, offline analyzer, real-engine drivers, committed measurement outputs, and figure-data pipeline. It is pinned through submodules and runs on a reusable Chameleon Cloud GDS environment.

git clone --recurse-submodules https://github.com/izzet/gdsight.git
cd gdsight
/opt/gds-tools/mount_gds_nvme.sh
gdscheck -p

Frozen artifact: doi:10.5281/zenodo.22719795.

View code on GitHub

BibTeX

@inproceedings{yildirim2026gdsight,
  title = {GDSight: Per-Operation Cross-Layer Attribution for GPUDirect Storage},
  author = {Yildirim, Izzet and Sun, Xian-He and Kougkas, Anthony},
  booktitle = {International Parallel Data Systems Workshop},
  year = {2026}
}