Welcome!

Containers Expo Blog Authors: Elizabeth White, Liz McMillan, Pat Romanski, Jyoti Bansal, Yeshim Deniz

Related Topics: SDN Journal, Java IoT, Linux Containers, Containers Expo Blog

SDN Journal: Blog Post

There Is Cost Per Port and Then There Is Cost Per Port…

I let you do the math on the cost of that infrastructure. And the installation. And the maintenance

Earlier this week, Ethan Banks wrote a very nice article about Mellanox’s dual spine and leaf network in support of a large amount 10GbE access ports. After describing the scaled up network design, he reviews 8 observations about the design, not to point out good or bad, but merely to point out specific points to consider. Fully coincidental (Ethan lives close to us, but I am pretty sure he cannot peek through our windows) we had gone through a similar exercise this week, documenting the choices and limitations of spine and leaf networks. And as always, the conclusions are not ones of right or wrong, more of awareness of choices and consequences.

The Mellanox design Ethan describes employes an extra spine layer, we have seen and heard the same from Arista and others, some calling it a spine-spine or similar. Nitpicking perhaps, but adding a spine layer to a spine and leaf network and still calling it a spine and leaf network is like adding a docking station, screen and keyboard to a phone and still calling it a phone. It’s a computer that can make calls. And a spine and leaf network with an extra spine is a fat tree.

Ethan points out in his first few points that the sheer amount of cables and optics is astonishing. Let me try and put some numbers around that statement. If you build an approximate 3:1 oversubscribed spine and leaf network out of generic switches, I would probably use a Trident2 based ToR switch with 48 SFP+/10GbE ports and 6 QSFP/40GbE ports. The 48 ports should serve most rack deployments with single homed servers, only really dense or heavily multi homed servers would need more ports and I will use 4 of the QSFP ports to connect to my spine. That leaves 2 QSFPs on each leaf to be used for extra access ports, at the cost of some oversubscription. Or if I wanted I could use these as extra spine connections, lowering my oversubscription to 2:1. As a spine, and staying away from chassis based systems, I would pick a 32xQSFP/40GbE switch.

The largest spine and leaf I could build out of this combination is one that contains 16 spine switches and 96 leaf switches. In a spine and leaf I need to connect each leaf to each spine and I have the equivalent of 16x10GbE to use out of each leaf, which I can connect to at best 16 spines. With each spine receiving 1x10GbE from each leaf, I can build out to 96 leafs to fill out my 32xQSFP or 96x10GbE equivalent spine switch capacity. 96 Leaf switches give me 4608 10GbE access ports at 3:1 oversubscription, 5376 at a slightly worse oversubscription if I use the extra QSFP ports on my leaf switches. To support these 5376 access ports, I need 3584 10GbE fabric ports: 16 each from 96 leaf switches plus 32×4 from each spine. That means 1792 switch interconnect cables. And 3584 10GbE short reach optics, because the vast majority of connections between spines and leafs is likely to be at a distance that DAC cables cannot cover (not considering the fact that most high density 10GbE switches are designed without PHYs, limiting the use of passive cables to usually 5m or less).

I let you do the math on the cost of that infrastructure. And the installation. And the maintenance. And the sheer complexity of running a 10GbE from each leaf to each spine. If you want to reduce some of this complexity, you can switch to using 40GbE/QSFPs between the leafs and spines, but by doing so you have reduced the maximum size of the network. Each leaf will now contribute 1 QSFP worth of interconnect to each spine, to a 32xQSFP spine can only support 32 leaf switches, or 1536 10GbE access ports (1792 if you use the 2 extra leaf QSFP ports). And you probably noticed that I left no room on the spine switches to actually leave this spine and leaf network to the rest of the network infrastructure. Taking a few QSFPs for that will reduce the size of the network more.

Even in the spine, spine and leaf network (or spine, leaf and ToR in Mellanox terminology) the amount of cabling between the ToR (someone explain why that is not the leaf of the network?) and the aggregation leaf may be reduced, the cabling between their aggregation leaf and spine still follows the same model as above.

We often focus on the cost of switches and the derived cost per port. Of course the cost per port is important, but don’t fool yourself by taking the cost of a switch and dividing it by the number of ports it supports. It makes for great quotes in press releases, but the actual cost for that port is way higher the moment you add in the overhead required to connect that port to the rest of the network. And for spine and leaf networks, the cost of the rest of the network has a very large portion of cables and optics. Even for reasonably priced optics, that piece of the infrastructure is likely going to cost more than the cost of the spine switches together. Even when you create one of these multispined animals…

[Today's fun fact: The human spine contains 120 muscles and approximately 220 individual ligaments.]

The post There is cost per port and then there is cost per port… appeared first on Plexxi.

Read the original blog entry...

More Stories By Marten Terpstra

Marten Terpstra is a Product Management Director at Plexxi Inc. Marten has extensive knowledge of the architecture, design, deployment and management of enterprise and carrier networks.

@ThingsExpo Stories
SYS-CON Events announced today that DivvyCloud will exhibit at SYS-CON's 20th International Cloud Expo®, which will take place on June 6-8, 2017, at the Javits Center in New York City, NY. DivvyCloud software enables organizations to achieve their cloud computing goals by simplifying and automating security, compliance and cost optimization of public and private cloud infrastructure. Using DivvyCloud, customers can leverage programmatic Bots to identify and remediate common cloud problems in rea...
In his opening keynote at 20th Cloud Expo, Michael Maximilien, Research Scientist, Architect, and Engineer at IBM, will motivate why realizing the full potential of the cloud and social data requires artificial intelligence. By mixing Cloud Foundry and the rich set of Watson services, IBM's Bluemix is the best cloud operating system for enterprises today, providing rapid development and deployment of applications that can take advantage of the rich catalog of Watson services to help drive insigh...
SYS-CON Events announced today that Tintri, Inc, a leading provider of enterprise cloud infrastructure, will exhibit at SYS-CON's 20th International Cloud Expo®, which will take place on June 6-8, 2017, at the Javits Center in New York City, NY. Tintri offers an enterprise cloud platform built with public cloud-like web services and RESTful APIs. Organizations use Tintri all-flash storage with scale-out and automation as a foundation for their own clouds – to build agile development environments...
A strange thing is happening along the way to the Internet of Things, namely far too many devices to work with and manage. It has become clear that we'll need much higher efficiency user experiences that can allow us to more easily and scalably work with the thousands of devices that will soon be in each of our lives. Enter the conversational interface revolution, combining bots we can literally talk with, gesture to, and even direct with our thoughts, with embedded artificial intelligence, whic...
SYS-CON Events announced today that Cloudistics, an on-premises cloud computing company, has been named “Bronze Sponsor” of SYS-CON's 20th International Cloud Expo®, which will take place on June 6-8, 2017, at the Javits Center in New York City, NY. Cloudistics delivers a complete public cloud experience with composable on-premises infrastructures to medium and large enterprises. Its software-defined technology natively converges network, storage, compute, virtualization, and management into a ...
Every successful software product evolves from an idea to an enterprise system. Notably, the same way is passed by the product owner's company. In his session at 20th Cloud Expo, Oleg Lola, CEO of MobiDev, will provide a generalized overview of the evolution of a software product, the product owner, the needs that arise at various stages of this process, and the value brought by a software development partner to the product owner as a response to these needs.
SYS-CON Events announced today that delaPlex will exhibit at SYS-CON's @ThingsExpo, which will take place on June 6-8, 2017, at the Javits Center in New York City, NY. delaPlex pioneered Software Development as a Service (SDaaS), which provides scalable resources to build, test, and deploy software. It’s a fast and more reliable way to develop a new product or expand your in-house team.
SYS-CON Events announced today that Technologic Systems Inc., an embedded systems solutions company, will exhibit at SYS-CON's @ThingsExpo, which will take place on June 6-8, 2017, at the Javits Center in New York City, NY. Technologic Systems is an embedded systems company with headquarters in Fountain Hills, Arizona. They have been in business for 32 years, helping more than 8,000 OEM customers and building over a hundred COTS products that have never been discontinued. Technologic Systems’ pr...
With major technology companies and startups seriously embracing Cloud strategies, now is the perfect time to attend @CloudExpo | @ThingsExpo, June 6-8, 2017, at the Javits Center in New York City, NY and October 31 - November 2, 2017, Santa Clara Convention Center, CA. Learn what is going on, contribute to the discussions, and ensure that your enterprise is on the right path to Digital Transformation.
SYS-CON Events announced today that Tappest will exhibit MooseFS at SYS-CON's 20th International Cloud Expo®, which will take place on June 6-8, 2017, at the Javits Center in New York City, NY. MooseFS is a breakthrough concept in the storage industry. It allows you to secure stored data with either duplication or erasure coding using any server. The newest – 4.0 version of the software enables users to maintain the redundancy level with even 50% less hard drive space required. The software func...
SYS-CON Events announced today that Outscale will exhibit at SYS-CON's 20th International Cloud Expo®, which will take place on June 6-8, 2017, at the Javits Center in New York City, NY. Outscale's technology makes an automated and adaptable Cloud available to businesses, supporting them in the most complex IT projects while controlling their operational aspects. You boost your IT infrastructure's reactivity, with request responses that only take a few seconds.
New competitors, disruptive technologies, and growing expectations are pushing every business to both adopt and deliver new digital services. This ‘Digital Transformation’ demands rapid delivery and continuous iteration of new competitive services via multiple channels, which in turn demands new service delivery techniques – including DevOps. In this power panel at @DevOpsSummit 20th Cloud Expo, moderated by DevOps Conference Co-Chair Andi Mann, panelists will examine how DevOps helps to meet th...
SYS-CON Events announced today that EARP will exhibit at SYS-CON's 20th International Cloud Expo®, which will take place on June 6-8, 2017, at the Javits Center in New York City, NY. "We are a software house, so we perfectly understand challenges that other software houses face in their projects. We can augment a team, that will work with the same standards and processes as our partners' internal teams. Our teams will deliver the same quality within the required time and budget just as our partn...
Most technology leaders, contemporary and from the hardware era, are reshaping their businesses to do software in the hope of capturing value in IoT. Although IoT is relatively new in the market, it has already gone through many promotional terms such as IoE, IoX, SDX, Edge/Fog, Mist Compute, etc. Ultimately, irrespective of the name, it is about deriving value from independent software assets participating in an ecosystem as one comprehensive solution.
In his keynote at @ThingsExpo, Chris Matthieu, Director of IoT Engineering at Citrix and co-founder and CTO of Octoblu, focused on building an IoT platform and company. He provided a behind-the-scenes look at Octoblu’s platform, business, and pivots along the way (including the Citrix acquisition of Octoblu).
SYS-CON Events announced today that A&I Solutions has been named “Bronze Sponsor” of SYS-CON's 20th International Cloud Expo®, which will take place on June 6-8, 2017, at the Javits Center in New York City, NY. Founded in 1999, A&I Solutions is a leading information technology (IT) software and services provider focusing on best-in-class enterprise solutions. By partnering with industry leaders in technology, A&I assures customers high performance levels across all IT environments including: mai...
Five years ago development was seen as a dead-end career, now it’s anything but – with an explosion in mobile and IoT initiatives increasing the demand for skilled engineers. But apart from having a ready supply of great coders, what constitutes true ‘DevOps Royalty’? It’ll be the ability to craft resilient architectures, supportability, security everywhere across the software lifecycle. In his keynote at @DevOpsSummit at 20th Cloud Expo, Jeffrey Scheaffer, GM and SVP, Continuous Delivery Busine...
SYS-CON Events announced today that Progress, a global leader in application development, has been named “Bronze Sponsor” of SYS-CON's 20th International Cloud Expo®, which will take place on June 6-8, 2017, at the Javits Center in New York City, NY. Enterprises today are rapidly adopting the cloud, while continuing to retain business-critical/sensitive data inside the firewall. This is creating two separate data silos – one inside the firewall and the other outside the firewall. Cloud ISVs ofte...
DevOps at Cloud Expo – being held October 31 - November 2, 2017, at the Santa Clara Convention Center in Santa Clara, CA – announces that its Call for Papers is open. Born out of proven success in agile development, cloud computing, and process automation, DevOps is a macro trend you cannot afford to miss. From showcase success stories from early adopters and web-scale businesses, DevOps is expanding to organizations of all sizes, including the world's largest enterprises – and delivering real r...
SYS-CON Events announced today that Systena America will exhibit at SYS-CON's 20th International Cloud Expo®, which will take place on June 6-8, 2017, at the Javits Center in New York City, NY. Systena Group has been in business for various software development and verification in Japan, US, ASEAN, and China by utilizing the knowledge we gained from all types of device development for various industries including smartphones (Android/iOS), wireless communication, security technology and IoT serv...