A lot of datacenter networks are run by absolute clowns. Not Amazon's

Last month I sat in a networking lab with AWS VP of Global Network Engineering Matt Rehder and cracked a joke about cage nuts cutting your hands to ribbons. I wasn't prepared for the blank stare I got in response, but I really should have been. Mind you, not because the joke wasn't funny (my jokes are hilarious), but rather because the world the joke was referencing no longer exists for them. Racks show up as assembled units, and apparently nobody screws hardware into anything in an AWS datacenter facility these days. It was a peek behind a curtain into a world that powers everything we do in cloud, but that remarkably few of us know exists. My Reg colleague Thomas Claburn toured the lab previously and gave us a deep dive into AWS's paper on this. In summary, "a flat, single-tier network wired in a deliberately near-random pattern leads to a way more resilient network that costs far less to run" and now you're up to speed. The part that I felt didn't get enough attention (and is why I invited myself to tour an AWS facility, much to AWS's surprise) is the economic part and how it impacts AWS customers. In short, their new networking approach can be up to 40 percent more energy efficient, and it's the default for most new datacenter builds (and dear reader, they are building a lot of those these days), and it's been completely invisible both inside and outside of the walls of the world's largest bookstore. The savings and cost efficiencies are real — so where do they go? Cost efficiencies abound Seventeen years ago, Amazon SVP James Hamilton got on stage and pointed out [PDF] that network vendors ran businesses whose margins resembled those of mainframe vendors, who themselves in turn resembled starving pigs at a trough (colorful metaphor mine, but also... not wrong). His big issue with this distilled down to "they're in my way," and he plus his team set out to do something about it. It turns out that "what they did about it" was nothing less than rebuilding the entire networking stack themselves atop commodity hardware. This brings us to a few years ago. Their networking cost efficiencies were already far lower than they would have been running atop traditional networking vendors, and then their Resilient Network Graph shufflebox play took another axe to the cost of networking. So... where have those savings gone over these past few years? I asked point blank whether it was fair to say that they kept the margin improvement rather than passing it back to customers. Their answer was unequivocal: "From a cost perspective? Effectively, yes." AWS cloud economics It's easy to see that answer as damning, but I humbly suggest that if that's your reaction, you haven't been paying attention to broader industry trends these last few years. Unlike its competitors, AWS has not raised prices on existing SKUs. (Yes, the cost of GPU capacity blocks has been raised on a quarterly basis, but that's apparently how those were designed to work, much like a slow version of Spot Instances. And yes, they started charging per public IPv4 address a couple of years ago.) You can go out and spin up the same instance with 64GB of RAM that you could in 2020 and pay the same price today; that's not even increasing a price to keep up with inflation! And while there's a 9 percent increase to go from a Graviton4-based c8g.2xlarge to its equivalent c9g.2xlarge Graviton5 instance, nothing is making you upgrade. Frankly, given the component cost hikes, I'm more than a little astonished that they were able to hold the increase to 9 percent. Further, despite the madness-inducing level of granularity you see in an AWS bill with its millions of SKUs, it's worth noting that each SKU itself obscures dizzying levels of complexity. An EC2 instance charges you per hour (yes, it's metered by the second; do not email me), but that one charge covers the CPU, the RAM, the backplane, the power, the building's physical security, the IAM security bits that keep it from resembling a public computer/something from Microsoft Azure, the employees it takes to build and run these systems, and a stupendous amount of networking magic. A sea change in approach One of those aspects of networking magic is that while it can be usuriously expensive to move traffic between Availability Zones (in major regions each gigabyte has a list price of a penny in and a penny out, so "two cents per gigabyte," which adds up), moving data inside of an AZ remains free. That's impressive when you start to realize how many different datacenter facilities can comprise a single AZ. It's damned impressive when you realize that AZs are expanding, data volumes are increasing, and that expansion comes with costs that thus far AWS has chosen to bear rather than pass on. It's significantly more expensive to move data out of AWS. Their now-defunct Snowball family of devices let you ship data back and forth to AWS. Sending one in cost a set fee, while sending one out cost that fee plus a per-GB charge. Annoying, right? That changed over the last year with the advent of AWS Interconnect - multicloud, which charges you nothing per GB, just an hourly port charge. There's a free tier of one 500 Mbps port per provider, which reduces to zero the AWS-side cost of sending that traffic to GCP, Oracle, and soon Azure. For comparison's sake, it'd be roughly $12,000 to send a month's saturation of that port across the open internet. Stated more plainly, this has never been done before on the AWS side, and it's fascinating. While it's not the primary area of focus for the folks I talked to, I nonetheless asked them about it for an on-the-record answer: "We know customers do not like rate-based network charges because it’s hard to predict their cost, which is why we are moving towards flat-rate pricing for new network products." Well slap me naked and hide my clothes; I'd be less surprised if Matt Garman gave his next re:Invent keynote in iambic pentameter. This bears watching; it's more interesting to me that it comes at a time when prices are rising instead of falling. How hard can it be? Very hard The problem is that a lot of what AWS does is invisible. For example, why am I telling you this instead of AWS? AWS concedes that they're generally bad at telling their own story, but they clearly want to tell it. Historically, they've punted to customers to tell their own stories, on the belief that the benefit will accrue to them. Here's my favorite example: the hard part of a near-random network isn't the cabling so much as the routing. Every protocol you've heard of (BGP, OSPF, a few Cisco-specific protocols that nobody sane uses, etc.) computes shortest paths, and "shortest path" stops meaning much when there are roughly thousands of equivalent routes between any two points. AWS solved that with a protocol called SIDR (Scalable Intent-Driven Routing, pronounced "cider," like the CIDR method for allocating IP addresses because these people are malevolent at naming things) that runs the control plane, while Spraypoint picks up the actual forwarding path. They presented SIDR publicly at re:Invent in 2023, then covered it again at the "Monday Night Live" re:Invent keynote in 2024. Then they published the RNG paper, which doesn't mention SIDR at all. As best I can tell, nobody outside the company has ever connected the protocol to the network it makes possible. I had to ask just in case I was missing something. I'm right. It isn't a secret! They just never got around to mentioning how important it was to making the whole thing go. Unfortunately, their lack of storytelling prowess about these things isn't working for them. The perception runs the other way: the market believes that the neoclouds and the Nvidia reference designs are better. They aren't. You can build a neocloud without knowing much about networking, and some outfits clearly have. A lot of datacenter networks are run by absolute clowns. I know this; I used to be one of said clowns before I got religion / discovered I could pay a cloud provider to make this stuff Matt Rehder's problem. AWS has gotten so good at making network arcana invisible that we don't really stop and think about just how wild it is that you can get damned near full line-rate between any two points in AWS and it just works. In the datacenters you or I build, you get to worry about fun things like "the switch at the top of the rack can only push so much traffic, so not every node can talk at full speed all the time." I have never encountered a real-world scenario of that being a concern in AWS. As a result, when folks set out to build their own datacenters, they don't know what they don't know after a generation of that complexity being hidden away from them, and they approach it with a sense of "how hard could it be?" It is extremely hard, and I have little confidence that the neoclouds spinning up new DC facilities overnight are paying attention to this. I asked Rehder directly whether the "ML datacenter facilities" AWS was launching in places like Mississippi were taking shortcuts since GPU workloads might have different requirements. He looked at me as if I were nuts; every datacenter they build is to the same spec. But customers won't care about that until right after they really should have cared about that; I have faith that a datacenter failure will be sufficiently public, embarrassing, and impactful so as to remind them. The tree of uptime must be periodically refreshed with the blood of massive outages. The humble mumble The folks I spoke to at AWS about this have built something that most of their own customers will never know exists, and they seem generally untroubled by that fact. I found that more admirable than I would have expected to; it's the very definition of thankless work, and AWS shines here. The reason the work is invisible is simply because it succeeds. And they're so humble about it! They politely chuckled at my joke about cage nuts once they got the reference, and I politely chuckled at their joke about how each year's switch is painted the Pantone "Color of the Year," which it turns out was in fact not even slightly a joke. AWS gets a lot wrong. They have ridiculous marketing campaigns, they build five services that do mostly the same thing and then name them like malevolent toddlers, and they've never found a partner they couldn't find a way to compete with. But when it comes to the "chop wood, carry water" type of work that makes the entire cloud possible, the kind that would put most of a keynote audience to sleep, they're the finest in the world. They're just bad at telling the world about it. ®