Fujitsu Adds Light To PCI Express with Intel Silicon Photonics
Fujitsu has worked with Intel to build and demonstrate the first Intel Optical PCIe Express (OPCIe) based server, enabled by Intel Silicon Photonics technology.
OPCIe powered servers offer advantages over non OPCIe based servers. Rack based servers, especially 1u and 2u servers are space and power constrained. Sometimes OEMs and end users want to add additional capabilities such as more storage and CPUs to these servers but are limited because there is simply not enough space for these components or because packing too many components too close to each other increases the heat density and prevents the system from being able to cool the components.
Fujitsu found a way to fix these limitations. The solution to the power and space density problems is to locate the storage and compute components on a remote blade or tray in a way that they appear to the CPU to be on the main motherboard. The other way to do this is to have a pool of hard drives managed by a second server - but this approach requires messages be sent between the two servers and this adds latency. It is possible to do this with copper cables; however the distance the copper cables can span is limited due to electro-magnetic interference (EMI). One could use amplifiers and signal conditioners but these obviously add power and cost. Additionally PCI Express cables can be heavy and bulky.
Fujitsu took two standard Primergy RX200 servers and added an Intel Silicon Photonics module into each along with an Intel designed FPGA. The FPGA did the necessary signal conditioning to make PCI Express "optical friendly". Using Intel Silicon Photonics they were able to send PCI Express protocol optically through an MXC connector to an expansion box.
In this expansion box was several solid state disks (SSD) and Xeon Phi co-processors and there was a Silicon Photonics module along with the FPGA to make PCI Express optical friendly. With this configuration, SSD?s and Xeon Phi's appeared to the RX200 server as if they were on the mother board. With photons traveling at 186,000 miles per second the extra latency of travelling down a few meters of cable cannot reliably be measured (it can be calculated to be ~5ns/meter or 5 billionths of a second).
This approach allowed Fujitsu to increase the storage capacity of the server because they now were able to utilize the additional disk drives in the expansion box. The number of drives is determined by the physical size of the box.
Fujitsu was also able to increase the effective CPU capacity of the Xeon E5's in the RX200 server because the Xeon E5's could now utilize the CPU capacity of the Xeon Phi co-processors. In a standard 1u rack it would be hard if not impossible to incorporate Xeon Phi's.
The Fujitsu approach also solves the SSD expansion problem, the CPU expansion problem and the total cooling and cooling density problems.
Besides Intel Silicon Photonics the Fujitsu demo also included Xeon E5 microprocessors and Xeon Phi co-processors.
Photonic signaling (aka fiber optics) has two fundamental advantages over copper signaling. First, when electric signals go down a wire or PCB trace they emit electromagnetic radiation (EMI) and when this EMI from one wire or trace couples into an adjacent wire it causes noise, which limits the bandwidth distance product. For example, 10G Ethernet copper cables have a practical limit of 10 meters. Of course, you can put amplifies or signal conditioners on the cables and make an "active copper cable" but these add power and cost. Active copper cables are made for 10G Ethernet and they have a practical limit of 20 meters.
Photons don't emit EMI like electrons do thus fiber based cables can go much longer. For example with the lower cost lasers used in data centers today at 10G you can build 500 meter cables. You can go as far as 80km if you used a more expensive laser, but these are only needed a fraction of the time in the data center (usually when you are connecting the data center to the outside world.)
The other benefit of optical communication is lighter cables. Optical fibers are thin, typically 120 microns and light.
But optical communications is not used more in the data center today due to high costs. Optical devices made for data centers are expensive. They are made out of expensive materials like Lithium-Niobate or Gallium-Arsenide, which are typically difficult to manufacture. The state of the art for these exotic materials is 3 inch wafers with very low yields. They are designed inside of gold lined cans and sometimes manual assembly is required as technicians "light up" the lasers and align them to the thin fibers. A special index matching epoxy is used that sometimes can cost as much as gold per ounce. Bottom line is that while optical communications can go further and uses light fiber cables it costs a lot more.
According to Victor Krutul is the director of marketing for the Silicon Photonics Operation at Intel, Silicon Photonics is th answer. Silicon Photonics is the science of making Photonic devices out of Silicon in a CMOS fab. Silicon is the most common element in the Earth's crust, so it's not expensive.
Fujitsu found a way to fix these limitations. The solution to the power and space density problems is to locate the storage and compute components on a remote blade or tray in a way that they appear to the CPU to be on the main motherboard. The other way to do this is to have a pool of hard drives managed by a second server - but this approach requires messages be sent between the two servers and this adds latency. It is possible to do this with copper cables; however the distance the copper cables can span is limited due to electro-magnetic interference (EMI). One could use amplifiers and signal conditioners but these obviously add power and cost. Additionally PCI Express cables can be heavy and bulky.
Fujitsu took two standard Primergy RX200 servers and added an Intel Silicon Photonics module into each along with an Intel designed FPGA. The FPGA did the necessary signal conditioning to make PCI Express "optical friendly". Using Intel Silicon Photonics they were able to send PCI Express protocol optically through an MXC connector to an expansion box.
In this expansion box was several solid state disks (SSD) and Xeon Phi co-processors and there was a Silicon Photonics module along with the FPGA to make PCI Express optical friendly. With this configuration, SSD?s and Xeon Phi's appeared to the RX200 server as if they were on the mother board. With photons traveling at 186,000 miles per second the extra latency of travelling down a few meters of cable cannot reliably be measured (it can be calculated to be ~5ns/meter or 5 billionths of a second).
This approach allowed Fujitsu to increase the storage capacity of the server because they now were able to utilize the additional disk drives in the expansion box. The number of drives is determined by the physical size of the box.
Fujitsu was also able to increase the effective CPU capacity of the Xeon E5's in the RX200 server because the Xeon E5's could now utilize the CPU capacity of the Xeon Phi co-processors. In a standard 1u rack it would be hard if not impossible to incorporate Xeon Phi's.
The Fujitsu approach also solves the SSD expansion problem, the CPU expansion problem and the total cooling and cooling density problems.
Besides Intel Silicon Photonics the Fujitsu demo also included Xeon E5 microprocessors and Xeon Phi co-processors.
Photonic signaling (aka fiber optics) has two fundamental advantages over copper signaling. First, when electric signals go down a wire or PCB trace they emit electromagnetic radiation (EMI) and when this EMI from one wire or trace couples into an adjacent wire it causes noise, which limits the bandwidth distance product. For example, 10G Ethernet copper cables have a practical limit of 10 meters. Of course, you can put amplifies or signal conditioners on the cables and make an "active copper cable" but these add power and cost. Active copper cables are made for 10G Ethernet and they have a practical limit of 20 meters.
Photons don't emit EMI like electrons do thus fiber based cables can go much longer. For example with the lower cost lasers used in data centers today at 10G you can build 500 meter cables. You can go as far as 80km if you used a more expensive laser, but these are only needed a fraction of the time in the data center (usually when you are connecting the data center to the outside world.)
The other benefit of optical communication is lighter cables. Optical fibers are thin, typically 120 microns and light.
But optical communications is not used more in the data center today due to high costs. Optical devices made for data centers are expensive. They are made out of expensive materials like Lithium-Niobate or Gallium-Arsenide, which are typically difficult to manufacture. The state of the art for these exotic materials is 3 inch wafers with very low yields. They are designed inside of gold lined cans and sometimes manual assembly is required as technicians "light up" the lasers and align them to the thin fibers. A special index matching epoxy is used that sometimes can cost as much as gold per ounce. Bottom line is that while optical communications can go further and uses light fiber cables it costs a lot more.
According to Victor Krutul is the director of marketing for the Silicon Photonics Operation at Intel, Silicon Photonics is th answer. Silicon Photonics is the science of making Photonic devices out of Silicon in a CMOS fab. Silicon is the most common element in the Earth's crust, so it's not expensive.
