The Colocation Model


Back to Taxonomy of Data Centers

The U.S. vs. European Internet Exchange Point Models


The European IXP Model

The European model IXP is modeled in Figure 12-19.

The European IXP Model has the follow characteristics:

European IXP Model

Figure 12-19. The European IXP Model.

Notes from the field.

European Model IXP Blessing and Curse

This association structure for IXPs has proven to be both a blessing and a curse. It helps get the IXP launched and builds a population; since the founding association members share in the benefits and the costs equally, they all can exert some pressure to get additional ISPs into their IXP. However, the “same pricing for all structure” model also limits the use of discriminatory pricing that could otherwise be used to lure in strategically important peers. This ability to drop the costs of peering for the right key peers is especially important during the early stages before an IXP reaches critical mass, and the value of peering there is less than the cost of peering there.

The U.S. IXP Model

IXPs in the U.S. are largely for-profit commercial operations run by the colocation companies that house them (Figure 12-20).

The U.S. IXP model has several characteristics.

For example, when Equinix Ashburn was launched and there were no tenants, public debt filings show that WorldCom was incentivized to come in with stock warrants. In Los Angeles the Any2 IXP was free for tenants in 1 Wilshire, and in Miami the NOTA IXP offered free peering ports to all tenants. The ability to incentivize the right ISPs into the IXP is a tool that the U.S. IXPs have and use, whereas the European IXPs have the same posted prices for all.

U.S. Model IXP

Figure 12-20. The U.S. IXP model.

The differences that people point to the most often are shown in Table 12-1.

Table 12-1. The differences between U.S. and European model IXPs.

European vs. U.S. IXP

Notes from the field.

European Operations Meetings are Dainty

I found it interesting that even the way the Internet Operations meetings around the world “feel” is different.

The RIPE meetings, for example, feel more community-based. They focus a lot on sharing information and comparing notes in public. A few “working groups” focus on common problems, and they publish them to share the wisdom beyond the meetings. The U.S. counterpart (NANOG) feels more commercial and conflict-filled, with a greater focus on sponsorship money and the politics of the organization than on sharing experiences.

The coffee breaks feel a bit formal in European meetings. Coffee and tea service has a “queue.” The wait staff ask people in turn if they want coffee or tea. They may add cream and sugar for you. The coffee and hot water for tea are served from an elegant urn.

In the U.S. the coffee breaks involve wheeling in a few tables of 20-gallon self-serve cafeteria-style urns. A plastic container houses the sugar packets and used plastic stirrers – the meeting has a different feel to it.

taxonomy of Internet Data Centers

Figure 12-21. The Taxonomy of Data Centers.

A Couple of Notes about Real Estate for the IXP Operator

Location is critically important as it directly affects the business case for peering, and therefore the likelihood of success as an IXP. At the highest level, IXP operators said that they first decide on the Internet Regions of interest. In the U.S., one of the first thoughts is to build in the “NFL” cities. The thinking is that if this city can support a major football team, then there is likely enough traffic to support an IXP.

Another critically important location issue is the existence of other competing IXPs. I surveyed several dozen ISPs and asked them how many IXPs they would like to have per region.

Half of them answered that they wanted to see only one IXP per region. That way, they could pick up all of the traffic they could there, and wouldn’t have to pay to reach the others that went into a different IXP. Keep the costs of peering down was their attitude. Some also pointed out that they preferred to handle the redundancy themselves through more interconnect points, diverse routing, etc.

The other half of the group said that they wanted to see exactly two IXPs per region for redundancy reasons. They wanted to see two different IXP operators, each hiring a different security guard company, a different hardware vendor for its public switch, a different fuel delivery company that would travel on different paths to refuel the diesel generators, etc.

And nobody wanted to see any more than two IXPs per region. That would “splinter the population” they said, “increasing the total cost of peering for the region.” If there were three or more IXPs in a region, each would probably have some distinct participants. To pick up all regional traffic in such a region, one would need to build into all IXPs at potentially great expense.

The point is, the competitive landscape for IXP operators is a critical factor for real estate selection.

High-Margin Cross-Connects

Another point must be made: To the colocation provider, cross-connects are wonderful high-value and high-margin products.

U.S. colocation providers charge customers for (typically fiber) cross-connects between two parties within their U.S. data centers. Within a cage, customers can of course run their own wires. It is when the wires cross the customer boundary that the colocation provider owns the interconnect and manages the interconnection process. Interconnects have a monthly recurring fee in the U.S.

These cross-connects are estimated to cost about $40 to run. This amount includes the cost of the fiber, the cost of the labor to run the fiber, and even a small cost allocation for the indirect costs of a database in which to store the customer information. There are lots of assumptions here, of course.

But the cost of the cross-connect is immaterial compared to the $300 per month that the U.S. colocation providers charge for cross-connects. The important question is: What is the value of the cross-connect to the participant? How high could the price get?

Let’s compare free peering using a cross-connect against the next-best alternative, purchasing Internet Transit. If one could freely peer 5Gbps over a cross-connect, and the alternative was to purchase 5Gbps of transit at $1/Mbps, colocation providers could charge as much as $5000/month for that cross-connect!

Based on this analysis, the $300/month cross-connect is quite a bargain.

This analysis has a lot of assumptions, but these assumptions are the broad strokes. In both cases (transit and peering) we assume peers and transit providers participate at the colocation center, no increase in equipment costs is incurred, and no incremental labor costs of peering, etc. are incurred. The point is that the value derived from the cross-connect is calculably higher than the price of the cross-connect.

The other comparison that is often made is the cost of a cross-connect compared to the cost of a circuit. Again, the cross-connect is a cost-effective alternative.

The profit on cross-connects is nearly 100%. A colocation provider with 5000 cross-connects generating $250/month yields $1.25M every month. Not a bad business.

It is interesting to observe that in Europe, cross-connects within data centers have always been run either by the tenants of the colocation center or by the colocation providers for a one-time fee of about $200. So when the European ISPs came to the U.S., they were surprised that there were monthly recurring charges for a piece of fiber run between participants.

Some of the European colocation operators I spoke with see Europe adopting the U.S. monthly recurring cost cross-connect model. They thought their customers might revolt. Some European colocation centers have already started owning the cross-connect process, so they seem to be inching along this path towards the U.S. cross-connect model.

Notes from the field.

Bypassing U.S. Monthly Recurring Cross Connect Fees

Since there is a monthly recurring cost for a cross-connect in the U.S., ISPs are incentivized to “cheat” and bypass the system. How do they bypass the system? We learned several methods in the field.

In one example, an ISP realized that its peer was on the floor directly below it. During a midnight installation, the ISP drilled a hole through the concrete floor to run the cable to its peer.

In another example, an ISP paid the IXP technician several hundred dollars under the table to run the fiber without reporting it to the IXP operator.

In a final example, the ISPs leveraged an operations loophole at a colocation facility. In this particular colocation facility, when a cross-connect is ordered it is connected from patch panel to patch panel within the customer cages. When a disconnect order is placed, the cross-connect is simply disconnected from the two patch panels. This scenario allowed customers to order cross-connects, place the disconnect order, and then both parties would plug the two ends back to the patch panel during their next visit to their cage. In some cases, the colocation provider disconnected the fiber and pulled the cross-connect up into the overhead wiring trays. The ISPs would both then fish down the fibers and reconnect them for free cross-connect services.


The Colocation Model


Back to Taxonomy of Internet Data Centers

This material is from The Internet Peering Playbook, available from (click below) and on the iBookStore.

This material is from The Internet Peering Playbook, available from (click below) and on the iBookStore.