The access to the market(s) is interesting enough to warrant it’s own page.
For near exchange work, it’s all about proximity. If you can get a cross connect to an exchange, you want to be as physically close to them as you can. In the same datacenter is a minimum. On the same floor. In the same area. As physically close as possible. On a trip to a datacenter, one of the jobs is scoping out what’s going on. That new expansion is where the exchange is going? You should be able to get access to pick your spot. All that info is secret, of course, but that doesn’t matter. You know where the exchange is going, that giant cage with custom power and cooling next to it? That’s Citadel, I heard. But you can’t compete with Citadel that way, but you can be the next best spot.
This is the area where you’re measuring in microseconds. The differences here can also be made up in other ways. Matching your routers to the exchange you’re connecting to. Will a Cisco or a Juniper have better time? Tune the algorithm. Measure, measure, measure.
If you can’t get a cross connect directly, or you have lines coming in from elsewhere, your line makes a trip to the “meet me room” or some variation on the theme. That’s where all the customers of the datacenter have their patches (that don’t go directly to someone else), and get connected to other folks. Your outside line, your other cages maybe, exchanges, whoever it is you have to talk to. Then somewhere in your cage or cabinet you have a patch panel where you can finally connect it to your equipment.
Of course, sometimes you get buildings like where the TSX floor is located. You can’t just connect your equipment on one floor to another floor – that would be too easy. You have to connect at your floor, it goes all the way down to the meet me room in the basement, and then all the way back up to the other location you’re connecting to. All at a fee, of course.
The datacenter itself
The datacenters themselves are amazing in almost every way. The way they’re not amazing? Appearances from the outside. Unmarked. Actively nondescript. Could be a warehouse, could be an office building. Some are multiple-block-long buildings out in New Jersey. Some are closets in skyscrapers. Some are multiple story former printing press buildings. Some you can almost throw a stone to one of the busiest roads in the world.
One has a great view of the Statue of Liberty.
There tend to be similarities, though. There are rows and rows of either racks, cages, or a combination of the two. There’s alternating hot and cold aisles. Cold air is forced either through the floor or from above via vents into the cold aisle where the servers draw it in. The hot aisles collect the output from both sides and either take it in or let it go naturally up into an open space. There are overhead ladders with network wiring that then breaks off and goes into the desired cages. The wiring for the power is the same story as the cold air. Either above or below.
The biggest limiting factor? Power and a/c. You can usually get space, and with that space comes a little bit of power. How much? Between 15 and 30 amps for the whole cabinet. A 42 U space with between 8 and 18 or so used. Hope you’re using 220v power for that extra efficiency!
Working in a datacenter is, like most things in tech and trading, an exercise in contradictions. They’re super cold so you’re freezing, but if you’re moving equipment, you’re still sweating. They’re also humidity controlled, so you’re usually thirsty – but there’s absolutely no liquid allowed inside. You’re probably hooked up to one of the most advanced networks in the world, but getting online is a pain. Wifi is strictly prohibited except for very temporary setups for working. You brought your laptop, but there’s no place to plug it in!
That’s legit – no place to plug it in. The cage has a PDU designed for use in cages like this, and with servers and networking equipment. Like this: If you didn’t have the foresight to order an adapter and/or your laptop won’t work on the voltage your cage is at – then you have no power!
There’s also no food allowed, but there’s usually a break room where people stash goodies for later. Bring drinks for there too, and label your bag. The other problem is snack food does not a meal make, and they usually (on purpose) make these things as far from civilization as possible! Usually there’s a small group working at the same time, so when you’re not going flat out as fast as you can, you’re sitting and waiting to be useful again. Of course, after a few trips out you generally just start sneaking candy and drinks in anyway. “Rules” eh?
If you’re lucky enough to have a whole cage of whatever size, you don’t have enough power to fill it up anyway. So make yourself a work area. Get a folding table and chair, keep tools around, have all the lengths and colors of cabling you use on hand, have a usable power strip and a couple extra ports. It’s like a cold, loud home away from home!
When working with two or more exchanges, it’s more about latency than proximity. Verizon guarantees 8.5 milliseconds per 1,000 miles – but that’s as the wire lays. It’s 710 miles from Chicago to New York. What time will your packet make the round trip? Well, if you have a good line, 18ms is pretty good. Less than that and you can make a business out of it. These folks for example advertises 17.2ms. There’s rumors about 16.8 and less. I want to know what the bandwidth along I-80 is.
When in a group of industry folks, someone will talk about a new path to NJ they’ve found. “Does it follow I-80?” “Better!” But you won’t get anything else out of them.
Within metropolitan areas, you still have to find the best paths between places. A few hundred microseconds are too many to waste.
What is dark fiber exactly? Well, it’s a piece of fiber optic cable that you connect to your equipment at both ends. It’s literally dark until you “light” it up. There’s no telephone company routers in between to change the pathing or add latency.
Measure twice. Actually, just keep measuring.
Everything – everything everything depends on your measurements. When did we send it, when do they say they got it, when did we get it back. Does that all match what the router says?
I used to ask family members what the most important part of the measuring operations was. No-one guessed knowing the time.
You have got to have a good time infrastructure. NTP is okay, but where are you getting your TP from? The internet is way too slow to be accurate enough for what we’re measuring here. You need a network time appliance in every metropolitan area. Every single datacenter would be awesome. Meinberg is the gold standard. It’s cool technology. The other problem with NTP is it’s not accurate enough in practice (in theory it is, but that’s a nice theory). PTP. How accurate is accurate enough? When you’re measuring the time it takes for an update to get into your decision engine until the time it takes to make a decision to measure how good your software is, you need in the nanos. I’m not saying single-digit nanosecond resolution. (Yet)
And throughout the day, things can change. Your monitoring infrastructure has to keep up not only with the hardware running your network, but the Quality of the Service on it, too. Things happen. Fiber gets cut. The telco re-routes you without any notice. Suddenly you’re getting picked off because you’re a millisecond slower than everyone else.
It’s better to drop out of the market and take the opportunity cost than to be in and making other people money.