Single Root IOV
Hypervisors are being used in servers to provide CPU, memory, and I/O virtualization. SR IOV implements I/O virtualization in the hardware for improved performance and to make use of hardware-based security and QoS features in a single physical server. SR-IOV also allows the sharing of an I/O device by multiple guest OSes running on the same server. This scales the number of VMs per physical server, scales the I/O devices, and scales the performance of the I/O devices with VM scaling. This has a profound effect on the I/O performance and I/O-centric loads on the devices.

As seen in Figure 3, the NICs and HBAs have virtual functions defined by the SR-IOV specification. SR-IOV performs better than VNICs and VHBAs, where virtualization is performed in software. However, this architecture doesn’t solve the power and cooling issues faced at the server unit because each processor complex still requires its own physical NIC and HBA.

Multi-Root IOV
MR IOV lets the I/O devices be removed from the processor complexes and then connects to them through an I/O converging server fabric. The MR IOV specification defines the use of PCIe as an I/O converging fabric and extends the definition of an SR IOV device to one that can be shared across multiple physical servers.

The reasons I/O devices are out of the servers include:

1. Server means CPU and Memory along with the sharing of IO devices across multiple physical servers on a switch card. This has the potential to provide more than 70% savings in power and total cost of ownership.
2. The architecture does not require access switches of networking and storage, which is a big plus point.
3. This architecture does not keep states of applications on the servers.

I/O virtualization provides a one-to-one mapping of the I/O devices to the servers and provides network consolidation of I/O devices onto a single fabric. The early de-multiplexing on these devices (aka multiple queues per virtual resource) provides traffic classification and prioritization that was being implemented in the access layer network switches. These features, along with QoS, enable mission-critical and non-mission-critical applications to be run simultaneously on the same server.

As shown in Figure 4, the I/O devices are connected on the opposite side of the MR-IOV switches from the CPU complexes. The MR IOV switches act as a Virtual Patch Panel to connect servers that share the downstream ports where MR-IOV devices are connected.

The Next-Generation Data Center
The next-generation data center includes the servers as CPU complexes with I/O devices, separated and connected via an MR IOV switch. This architecture has multiple advantages. It allows the vision of stateless computing to be implemented, with application migration and high server utilization. It minimizes cost by minimizing the number of I/O devices and fabrics required in the system. This architecture can be easily extended to combine FC (storage area network connection) and Ethernet connections like iSCSI or FCOE (see Figure 5).

Conclusion
MR-IOV switches with iSCSI/FCOE and virtualized resources provide the optimal architecture for next-generation data centers. The MR-IOV switch provides Virtual Patch Panel capability along with hardware-based security and QoS features (versus single-wire Ethernet with virtualized functions), providing scalability and performance of highly reliable Ethernet in next-generation data centers.

The salient features of this architecture are:

• Server means CPU + memory (stateless computing vision)
  
• Sharing of I/O devices across multiple servers
  a. Power, cable, TCO and OPEX, and power savings
  
• I/O devices can be assigned on-the-fly with the applications’ IP and WWN maintained.
  
• Applications running over the server can be migrated to other servers without its MAC and WWN numbers being changed (provided MAC and WWN are being maintained by MR-IOV NICs and HBAs).
  
• Hypervisors can run the applications on virtual servers with its own virtual machines.
  
• I/O devices running on single-wire (Ethernet, protocol can be iSCSI/FCOE)
  a. Reduction in CAPEX, OPEX
  b. Merged network with networking and storage

It does not require access protocol switches like Ethernet and Fiber channel. The IO devices feed directly into core switches that removes inter-operability of switches from different vendors.

• This realizes the stateless computing vision where servers do not keep any states that allow the applications floating on fluid infrastructure as shown in Figure 5.