znvme

A userspace NVMe driver implemented in Zig, leveraging VFIO for direct hardware access.

Features

• Direct access to NVMe device registers and features from user space.
• Secure operation without sudo by using VFIO.
• Easy integration with Zig applications.

Requirements

• Zig 0.14.1 or later
• Linux kernel with vfio-pci support
  • IOMMU enabled in the kernel (Intel VT-d or AMD-Vi)
• NVMe devices connected to the system

Installation

To use this library, you need to clone the repository and build the Zig package.

git clone https://github.com/wipeseals/znvme.git
cd znvme
zig build test

Usage

Basic Example

TODO

Advanced Example

TODO

IOMMU Setup

Ensure the kernel command line includes intel_iommu=on or amd_iommu=on depending on your CPU architecture.

cat /proc/cmdline
sudo dmesg | grep -e DMAR -e IOMMU

user in 🌐 nbg9 in znvme on  develop [$✘?] via ↯ v0.14.1 via ❄️  impure (nix-shell-env) 
❯ sudo dmesg | grep -e DMAR -e IOMMU
(snip)
[    0.005995] ACPI: DMAR 0x00000000725CE000 000088 (v02 INTEL  EDK2     00000002      01000013)
[    0.006037] ACPI: Reserving DMAR table memory at [mem 0xXXXXXXXX-0xXXXXXXXX]
[    0.027230] DMAR: IOMMU enabled
[    0.079635] DMAR: Host address width 39

For Intel CPUs

Add intel_iommu=on to the kernel command line in your bootloader configuration (e.g., GRUB):

sudo vim /etc/default/grub

GRUB_CMDLINE_LINUX_DEFAULT="quiet splash intel_iommu=on"

Then update GRUB

sudo update-grub

VFIO Setup

To access NVMe devices from user space, you need to switch the kernel driver from nvme to vfio-pci.

Driver switching scripts are available in the misc/ directory.

Listing NVMe devices

First, check the NVMe devices connected to your system.
The following command lists devices currently bound to the nvme driver:

user in 🌐 nbg9 in znvme on  master [!?] via ↯ v0.14.1 via ❄️  impure (nix-shell-env) 
❯ sudo ./misc/list.sh
# Listing NVMe devices
Node                  Generic               SN                   Model                                    Namespace  Usage                      Format           FW Rev  
--------------------- --------------------- -------------------- ---------------------------------------- ---------- -------------------------- ---------------- --------
/dev/nvme0n1          /dev/ng0n1            Z4HF716NF88S         KIOXIA-EXCERIA with Heatsink SSD         0x1          1.02  TB /   1.02  TB    512   B +  0 B   AJRA4101
/dev/nvme1n1          /dev/ng1n1            201008800819         WDC WD BLACK SDBPNTY-512G-1106           0x1        512.11  GB / 512.11  GB    512   B +  0 B   HPS2    
/dev/nvme2n1          /dev/ng2n1            TTSMA2513X09608      TWSC TSC3AN512-F8T40S                    0x1        512.11  GB / 512.11  GB    512   B +  0 B   SN13126 

# Listing NVMe devices with PCI addresses, VID, PID, and manufacturer/model info
PCI Address: 01:00.0, VID: 15b7, PID: 5006, Info: Sandisk Corp SanDisk Extreme Pro / WD Black SN750 / PC SN730 / Red SN700 NVMe SSD
PCI Address: 06:00.0, VID: 1e0f, PID: 000d, Info: KIOXIA Corporation NVMe SSD Controller XG7
PCI Address: 07:00.0, VID: 1e4b, PID: 1202, Info: MAXIO Technology (Hangzhou) Ltd. NVMe SSD Controller MAP1202 (DRAM-less) (rev 01)

You can also check each device's PCI address, vendor ID, and product ID.
This information is required for driver switching.

Binding to the vfio-pci driver

Unbind the NVMe device from the nvme driver and bind it to the vfio-pci driver.
Example for switching the device at PCI address 0000:06:00.0:

user in 🌐 nbg9 in znvme on  master [!?] via ↯ v0.14.1 via ❄️  impure (nix-shell-env) 
❯ sudo ./misc/bind_vfio.sh 0000:06:00.0 1e0f 000d
Unbound 0000:06:00.0 from nvme driver.
grep: /sys/bus/pci/drivers/vfio-pci/new_id: Permission denied
Bound 0000:06:00.0 to vfio-pci driver.
Bound 0000:06:00.0 to vfio-pci driver.
Successfully bound 0000:06:00.0 to vfio-pci driver.
06:00.0 Non-Volatile memory controller: KIOXIA Corporation NVMe SSD Controller XG7
        Subsystem: KIOXIA Corporation NVMe SSD Controller XG7
        Kernel driver in use: vfio-pci
        Kernel modules: nvme

If you see Permission denied for /sys/bus/pci/drivers/vfio-pci/new_id, make sure you are running as root.
After binding, confirm Kernel driver in use: vfio-pci.

Unbinding from the vfio-pci driver and rebinding to the nvme driver

After your work, you can return the device to the original nvme driver:

user in 🌐 nbg9 in znvme on  master [!?] via ↯ v0.14.1 via ❄️  impure (nix-shell-env) 
❯ sudo ./misc/unbind_vfio.sh 0000:06:00.0
Unbound 0000:06:00.0 from vfio-pci driver.
Bound 0000:06:00.0 to nvme driver.
Successfully bound 0000:06:00.0 to nvme driver.
06:00.0 Non-Volatile memory controller: KIOXIA Corporation NVMe SSD Controller XG7
        Subsystem: KIOXIA Corporation NVMe SSD Controller XG7
        Kernel driver in use: nvme
        Kernel modules: nvme

References

• NVMe Specification
  • NVM Express Base Specification Revision 2.1
  • NVM Express PCI Express Transport Specification Revision 1.1

License

This project is licensed under the MIT License. See the LICENSE file for details.