ggmlR (version 0.8.1)

ggml_vulkan_p2p_selftest: Opaque-fd device-to-device P2P self-test

Description

Exercises the VK_KHR_external_memory_fd transport used by Vulkan tensor parallelism to move data between GPUs. A byte pattern is written on the source device, its memory is exported as an opaque fd and imported on the destination device, copied into a local buffer there, then read back and compared.

Usage

ggml_vulkan_p2p_selftest(
  src_device,
  dst_device,
  bytes = 64L * 1024L * 1024L,
  iters = 50L,
  transport = c("host-staging", "opaque-fd", "device-group")
)

Value

A named list: status (integer, 0 = data verified, <0 = failure),

gbps (numeric, measured cross-device bandwidth; 0 for loopback or on failure) and report (character diagnostic, incl. the NVLink-vs-PCIe inference).

Arguments

src_device

Source GPU index (0-based).

dst_device

Destination GPU index (0-based). Equal to src_device for a loopback sanity check.

bytes

Transfer size in bytes (default 64 MiB).

iters

Number of copies to time for the bandwidth measurement (default 50).

transport

Cross-device transport to exercise: "host-staging" (default, portable device->host->device copy — correct on every driver, PCIe + RAM bounded), "opaque-fd" (VK_KHR_external_memory_fd P2P; works on AMD/RADV but does NOT alias memory cross-device on the NVIDIA proprietary driver — reads back as zeros), or "device-group" (experimental NVIDIA LDA). host-staging is the transport Stage E3 uses.

Details

When src_device == dst_device the test runs in loopback mode (export and import on the same GPU) — a sanity check of the fd mechanism that touches no inter-device link. When the devices differ it runs cross-device: after verifying correctness it times iters device-to-device copies and reports the achieved bandwidth.

Interpreting bandwidth: a measured rate above the PCIe 3.0 x16 ceiling (~16 GB/s) is empirical evidence that a faster physical link (e.g. NVLink) carried the bytes. It is not a claim that Vulkan used an NVLink API — Vulkan exposes no call to query the route, so the conclusion is inferred from the rate alone, never asserted from the API.

Examples

# \donttest{
if (ggml_vulkan_available()) {
  # loopback sanity check on device 0
  r <- ggml_vulkan_p2p_selftest(0L, 0L)
  cat(r$report)
  # cross-device P2P (requires >= 2 GPUs)
  if (ggml_vulkan_device_count() >= 2) {
    r <- ggml_vulkan_p2p_selftest(0L, 1L)
    cat(r$report)
  }
}
# }