Co-Packaged Optics and an Open Ecosystem

Some technology transitions are easy to spot and their adoption is inevitable. The only question is when the transition happens and how quickly will it be adopted.

Co-packaged optics (CPO), or in-package optics (IPO) depending on your terminology, is one of those technologies. Bringing optics and switch silicon together in the same package creates a synergy between once disjoint and independent technologies thereby saving significant power.

The industry acknowledges this important evolution is coming but hasn’t been able to predict its arrival. Experts previously claimed that an ASIC’s electrical SerDes I/O would be unable to pass 10 Gbit/sec. When we pushed past this barrier, predictions were that 56 Gbit/sec SerDes would be impossible and therefore the entire 12.8 Tbps switch silicon generation would be based on CPO. Today we are in the 112 Gbit/sec SerDes generation with 25.6 Tbps silicon and we have yet to see the arrival of this CPO technology in any meaningful way, so what has changed and why are we talking about it more seriously now?

Before we get into when this critical transition will happen, I think it’s important to analyze why I say the technology transition is inevitable. The answer lies in two simple assumptions:

1. Analyzing historical growth trends provide a good indicator of future requirements.

2. We are at an inflection point in the industry where power has now become the ultimate limiting factor.

Analyzing Historical Trends – Switch Silicon

Analyzing the historical trends of switch silicon highlights two long-running trends.

1. Approximately once every two years the bandwidth of the switch silicon doubles tracking well to the notion of Moore’s Law which states that the number of transistors in a piece of silicon doubles every two years.

2. To support the increase in total switch silicon bandwidth both the speed and number of SerDes increased. The SerDes speed increased from 10 Gbit/sec to 112 Gbit/sec and the number of SerDes around the chip increased from 64-lanes to a projected 512-lanes in the 51.2 Tbps generation.

Unfortunately, Moore’s Law governs the number of transistors which more closely tracks to digital logic rather than the SerDes which includes analog portions in their design.

If we further analyze the data, we find that to achieve the 80x switch silicon bandwidth increase from 640 Gbps to 51.2 Tbps the total power of the switch silicon increased by 9.5x. Or said another way, although the power efficiency increased with each new advanced CMOS node the total power still increases generation after generation.

Further breaking this down we can see that the silicon core power has increased by 7.4x, while the per-SerDes power increased by 2.84x. Coupled to the increasing number of SerDes, the total SerDes power in the switch silicon increased by 22.7x causing the ratio of power spent on SerDes to increase dramatically over-time.

From this historical context, we can extrapolate that a 51.2 Tbps switch device will arrive in 2022 and 102.4 Tbps device will arrive in 2024 and that the power associated with the SerDes interconnect will continue to increase as a percentage of the total switch power and consume more of the system budget, ultimately dominating the total power consumption of the switch.

Analyzing Historical Trends – Copper to Optical

The next piece of historical context has to do with how devices are connected. When global communication infrastructure was first deployed it used copper cables. Today in the Service Provider and Web-Scale networks most links outside of the rack are optical while wiring within the rack is copper. As speeds increase the longest copper links need to move to optical. Eventually, all the links leaving a silicon package will be optical rather than electrical.

Power – The Ultimate Limiting Factor

 In my How Cisco Silicon One Can Help You Save Millions, I go into some of the reasons why power is so impactful for our customers and the environment, but taking a step back and thinking about the broader picture I believe power is the ultimate limiting factor because:

◉ Power limits what systems we can build, creating a technology imperative

◉ Power limits what our customers can deploy, creating a business imperative

And most importantly,

◉ Power limits what our planet can sustain, creating a moral imperative

These three imperatives create a perfect environment for us to drive innovation.

Because SerDes power is such a large portion of the total system power today, and it’s reducing slower compared to system bandwidth increases, it’s an area that we must address with architectural innovations.

These trends and limits are why solving how to implement CPO today are so important.

Minimizing Interconnect Power

From “Through the Looking Glass – The 2018 Edition: Trends in Solid-State Circuits from the 65th ISSCC” we can see the strong relationship between power efficiency and the insertion loss of the channel the SerDes is designed to drive.

As the distance, or more precisely the channel insertion loss, decreases the SerDes can be simplified saving significant power. This means the closer two devices are to each other the lower power it takes to send a signal between them. Taking this concept to the extreme, bringing the optical engine directly into the switch silicon package creates the shortest possible electrical traces thereby saving significant power.

This is the advantage of co-packaged optics.

Why 51.2T?

At this point, we have done enough analysis to show that a 51.2 Tbps based Ethernet switch can be built supporting 64 x QSFP-DD800 pluggable modules so we aren’t forced to build CPO to ship the product. However, our power analysis shows that a CPO-based switch design is significantly more power-efficient than a traditional 51.2 Tbps design with pluggable optics.

It is also clear that the 102.4 Tbps generation based on 224 Gbps SerDes will be a power-hungry and challenging system design, while the 204.8 Tbps generation will further challenge our traditional design techniques.

Architecting, designing, deploying, and operationalizing systems with CPO is an incredibly difficult task and therefore it is critical as an industry that we start before it’s too late. Therefore, I believe that the 51.2 Tbps switch silicon generation is the correct time to introduce CPO.

Cisco is in a unique position in the industry where we have industry-leading silicon, pluggable optics, on-board optics, silicon photonics, and system design in-house and we are working hard in conjunction with our customers to bring these technologies together to enable this important transition.

Creating an Ecosystem

Despite having our own extensive in-house experience and capabilities, Cisco believes that any such disruptive technology can only succeed when the right ecosystem is in place. The industry has a long history of standardization efforts such as the OIF, IEEE, and the MSAs which have defined the standards for pluggable optical modules. These standardization efforts have resulted in interoperable products being available from a wide variety of suppliers that customers can be confident will work together, providing customers with choice, the security of supply, and shorter time to market. These collaborative efforts are the foundational bedrock that our industry needs in order to progress at the technological and commercial pace that is required.

As a precursor to a broader standards effort, today I am pleased to announce a collaboration between Cisco and Inphi to cooperate on the definition of a CPO-based switch/optics solution to drive the industry forward and ensuring interoperability between silicon and optical engines from multiple different companies.

This collaboration will help our customers to enjoy a diverse and open ecosystem and interoperable best-of-breed technologies from a variety of suppliers.

Source: cisco.com