Virtio-network is fast until the workload makes it relative, and the tier where ten-gigabit line rate, microseconds of latency and zero host-CPU cost are the requirements rather than the aspirations needs a different mechanism entirely. SR-IOV, Single Root I/O Virtualization, is that mechanism: a capable network card splits itself into multiple virtual functions, each one a real PCI device that the hypervisor assigns directly to a guest, the guest drives its own slice of the hardware with its own driver, and the host's network stack steps out of the data path. This article builds the setup on a modern server with VT-d support: the hardware prerequisites and the BIOS ceremony, creating virtual functions on the host, assigning them to guests through libvirt, the guest-side reality, the migration tradeoff that every SR-IOV deployment must plan around, and the monitoring that keeps the whole arrangement observable.
The model, physical functions, virtual functions and the data path
An SR-IOV-capable network card presents two kinds of identities on the PCI bus. The physical function, the PF, is the card itself, the full device the host owns and configures. The virtual functions, the VFs, are lightweight siblings the card firmware creates on demand, each with its own PCI address, its own queues, its own interrupt vectors, and its own MAC address. A guest assigned a VF talks to the card directly: the packet leaves the guest's driver, enters the VF's hardware queue, and moves on the wire, with the host's kernel involved in setup and teardown but absent from the per-packet path.
The contrast with virtio-net is architectural rather than incremental. Virtio is paravirtual, the guest and host cooperating through shared memory and a backend service, the host's vhost code touching every packet even if only to move it. SR-IOV is direct assignment, the guest owns a hardware resource the same way bare metal does, and the per-packet cost inside the host drops to zero. The measured difference on the right workload is the difference between line rate with headroom and line rate with the CPU already busy, and the tier that buys ten-gigabit and twenty-five-gigabit cards is the tier that wants the headroom.
The trade that SR-IOV makes for the speed is the loss of the host's mediation, and the consequences thread through everything below: the host's iptables OUTPUT policy from this series does not see the guest's packets, the host's tcpdump on the physical interface sees fragments of what passes, and the live-migration capability from the previous article stops working for the assigned VF because a PCI device physically lives in one machine. The architecture is the choice to give a specific guest the network as directly as possible, and everything else in this article is the discipline that keeps that choice manageable.
Hardware prerequisites, IOMMU and the BIOS ceremony
Direct device assignment requires the hardware to guarantee that a device's DMA cannot touch memory outside its assignment, the job of the IOMMU, called VT-d by Intel and AMD-Vi by the AMD side, and the checklist starts in the firmware settings. The server's BIOS or UEFI carries the virtualization flags under different names on every vendor, Intel VT-d, IOMMU, AMD IOMMU, sometimes nested under chipset settings, and the flag must be enabled before the operating system sees anything.
The kernel side is a boot parameter, intel_iommu=on for the Intel spelling and amd_iommu=on for AMD, though current AMD kernels enable the IOMMU by default where the firmware does. The parameter lives in the bootloader configuration, GRUB_CMDLINE_LINUX in /etc/default/grub on the standard stack, and after the rebuild and reboot the confirmation is one command:
dmesg | grep -i -e DMAR -e IOMMU
whose output names the detected units and the enabled remapping, the first checkpoint that the hardware ceremony worked. The second checkpoint is the IOMMU grouping, the kernel's partitioning of devices into groups that must be assignable as a whole, readable through the sysfs tree under /sys/kernel/iommu_groups/, with each group's members symlinked beneath their group number. A network card that shares an IOMMU group with its PCI bridge is assignable cleanly, a card that shares a group with an unrelated device needs the ACS-enabled bridges that serious server hardware carries, and the group listing is the preflight check that prevents the assignment failure with the cryptic message on the first evening.
Creating virtual functions on the host
The card exposes its SR-IOV capabilities through sysfs, and the control file is the VF count, the number of virtual functions the PF creates. The interface names the card, and the count is a single write:
echo 4 > /sys/class/net/ens1f0/device/sriov_numvfs
after which the card carries four VFs, each appearing as a new PCI device, visible through lspci:
lspci | grep -i ethernet 01:00.0 Ethernet controller: Intel Corporation 82599ES (rev 01) 01:00.1 Ethernet controller: Intel Corporation 82599ES (rev 01) 01:00.2 Ethernet controller: Intel Corporation 82599ES (rev 01) 01:00.3 Ethernet controller: Intel Corporation 82599ES (rev 01) 01:00.4 Ethernet controller: Intel Corporation 82599ES (rev 01)
The .1 through .4 entries are the new VFs, real devices with their own drivers, and their count persists only until reboot unless the creation is made permanent, either through the driver module parameter, the ixgbe family's max_vfs option on the classic Intel cards, or through a systemd unit that performs the sysfs write at boot. The count chosen is a capacity decision: each VF is a real hardware queue set, and the card's documented maximum, varying by model from eight to one hundred twenty-eight, is the ceiling the pool plans against.
The per-VF properties are set on the host through the PF's link device, the MAC address and the VLAN tag being the two that production uses:
ip link set dev ens1f0 vf 1 mac 52:54:00:2a:1b:0c vlan 21
which programs the card's hardware with the guest's identity, the same properties the bridge and NAT models set in software, set here in silicon.
Assigning the VF to a guest through libvirt
The assignment is a hostdev declaration in the machine's XML, naming the PCI address directly:
<interface type='hostdev' managed='yes'>
<driver name='vfio'/>
<source>
<address type='pci' domain='0x0000' bus='0x01' slot='0x00' function='0x1'/>
</source>
<mac address='52:54:00:2a:1b:0c'/>
</interface>
The managed flag tells libvirt to handle the device's detach from the host driver and its reattach on shutdown, and the vfio driver is the kernel's assignment machinery, the layer that holds the IOMMU guarantees. The guest boots and sees a real network card, its own model detected by its own driver, the ixgbevf for the Intel family or the sibling drivers for the Mellanox and Broadcom lines, and from that moment the guest configures its network exactly as a physical machine does.
The MAC in the XML must match what the host programmed into the card's VF, or the mismatch produces the classic first-evening symptom, the guest's packets leaving but the return traffic landing in the wrong queue, and the ip link verification from the previous section is the thirty-second check that separates a configuration error from a debugging session.
The migration tradeoff and the two-tier fleet pattern
The VF is physically in the host, and a guest holding a physical device cannot migrate, the hard limitation that shapes every SR-IOV deployment. The previous article's live migration works precisely because the moved state is software, memory and registers, and the network attachment is virtual on both ends, while a VF is a hardware claim that the destination host cannot honor, a different card, different PCI addresses, no continuation.
fleets live with the limitation through the two-tier pattern, and the design reads cleanly once stated. The latency tier, the machines whose network demands earned them the VFs, runs SR-IOV without migration, and their availability plan is the rapid-provisioning path, templates, shared storage, a replacement guest built in minutes. The elastic tier runs virtio-network with full live-migration, and workloads graduate between tiers on measured need rather than on enthusiasm. The hybrid pattern exists for the machines that need both, a virtio interface for the management and migration plane alongside the assigned VF for the data plane, and the guest routes its two roles across the two interfaces deliberately.
Monitoring and the troubleshooting file
The monitoring story with SR-IOV splits along the same line as the data path. The host's interface counters on the PF reflect the card's aggregate, and the per-VF statistics are the guest's own, read inside the guest with ethtool and readable through the guest agent's domifaddr family for the inventory side. The host still owns the card's health, the PF's dmesg entries for link events and firmware messages, and the vendor tools, ethtool -i for the driver versions and the vendor-specific diagnostics, belong in the runbook for the hardware tier the same way the MegaCli commands did for the RAID tier in this series.
The troubleshooting file is short because the failure modes are few. An assignment error naming the IOMMU group points at the grouping preflight from the second section. A guest with an interface that sends but never receives points at the MAC or VLAN mismatch between the XML, the ip link programming and the guest's expectation. A VF that vanishes after a host reboot points at the persistence gap in the sysfs creation. And the throughput that lands below expectation points at the interrupt coalescing and queue settings, the ethtool tuning inside the guest that the virtio article's multi-queue story parallels, because a directly-assigned card still has the same tunables a physical one does.
The closing frame
SR-IOV is the last stop on the virtualization performance train, the point where the guest and the hardware meet with nothing between them, and its discipline is knowing exactly which workloads belong at that stop. The setup path, BIOS flags, kernel parameters, sysfs VFs, libvirt assignment, is an afternoon on compatible hardware, and the operational pattern, the two-tier fleet with deliberate graduation between them, keeps the speed honest. The guest that needed line rate gets line rate, the fleet that needed mobility keeps it, and the choice between them is written down as architecture rather than discovered as a limitation on the night it matters.
## Guest-side reality and the VLAN patterns
The guest's experience of its VF is deliberately unremarkable, which is the design working: the machine boots, the kernel loads the vendor's VF driver, the interface appears as an ordinary Ethernet device, and every tool from the rest of this series applies to it unchanged. The guest configures the interface with its own network stack, static addressing or DHCP as the site prefers, the ethtool tuning for queues and coalescing lands directly on real hardware, and the monitoring agents see exactly what a physical machine's agents would see.
The VLAN question deserves its explicit paragraph because it is where host and guest responsibilities meet. The per-VF VLAN programmed on the host, the ip link set vf command from the earlier section, tags the traffic in hardware before it reaches the physical port, the pattern for the fleet where the guest must not be trusted with the tag. The alternative runs the tagged interface inside the guest, the 802.1q module and a vlan subinterface in the guest's own configuration, which suits the trusted tenant that manages its own switching. The two patterns are mutually exclusive per VF, and the choice is a policy statement, the untrusted tier gets its tags from the host's silicon, the trusted tier wears its own.
A worked sizing example
A host with a dual-port ten-gigabit card and a pool of six guests on the latency tier sizes its VFs against the card's queue capacity rather than its port count: eight VFs on the first PF, one per guest plus a spare, the second port cabled to the second switch of the redundant pair, and the six guests each carrying one VF with its MAC and VLAN programmed in the first boot script. The pool plans one guest per VF and documents the ceiling, because the card's queues are the exhaustible resource, and the seventh guest that arrives unannounced finds the answer in the runbook rather than in a surprised evening. The example's moral is the article's: SR-IOV is capacity planning with hardware vocabulary, and the deployment that names its ceilings in writing is the one whose speed does not come with surprises attached.