Imaginary Billboards

Gameboy on a train #0 – Intro

Posted by Chris on November 1st, 2013 filed in Games
Comment now »

A few months ago, I found myself with a few hours free every night, with everyone else in the house asleep. After upgrading computers, painting bathrooms, etc, one day I found my old Gameboy and fixed the screen and put a new speaker in. Now this is my original one from 23 years ago that I’ve lugged this whole time.

Fast forward a couple of weeks.

I get bored reading or listening to podcasts on the train every day and grabbed the gameboy. It had Tetris in it (of course), so I played it on the train and it was everything I thought it could be. Trips go by quickly (only missed my stop once), and all comments are positive.

So I had to figure out how to decide when to move on to another game. I made up an arbitrary way of “beating” it that I thought would take a while (it did) and would be satisfying (it was) and a good way to decide when to “move on” as it were. So, in the future I’ll be posting random things about games and the like because why not?

The price of oil

Posted by Chris on November 7th, 2012 filed in Trading
Comment now »

This is a true story, as far as I know. Backed up by the New York Times. Jan 2, 2008, trading floor of a prop trading shop. Oil had been steadily heading higher for weeks if not months and we were all waiting with bated breath to see when it would hit the arbitrary “wow” moment of $100 per barrel. Something happens over at the desk of the guy who is trading oil, and people are all laughing and shaking their heads as they walk away. Oil had finally hit $100 and gone back down immediately.

Apparently someone wanted to be the first person ever to trade oil at $100 so he bid it up, bought it and immediately sold it. Didn’t even cost him that much. He just wanted to have been the first and that was it.

Remembering Hurricane Katrina

Posted by Chris on October 29th, 2012 filed in Stories
Comment now »

In mid 2005, I started working at a very, very large insurance company. A few months in, in August, Katrina occurred and it was widely assumed after a while that it would be incredibly destructive – once we saw where it was heading.

Said insurance company had a big dining and presentation area next to the cafeteria and Katrina was such an event that they turned the news on on big projection screens. The middle one had the sound turned on while the other two played different channels with closed captioning.

While the hurricane was hitting New Orleans, some of the news organizations were projecting insurance losses and being an insurance company we were very interested in it. Hence the giant TVs. One, in particular, was keeping a running tally as the losses went up. The same day the projected losses passed the most expensive storm ever, a few hundred people were watching in the room at once.

We were all waiting for it to do one hundred billion Dollars worth of damage, as it was expected. The notes would come in from different wards of the city. $80 billion, $90 billion. $95 billion.

$100 billion.

There was a little bit of a cheer in the room at that point, as if we had all been part of something big and important. Something of note.

After a second, everyone realized what they had done and went back to their cubicles in silence.

Comcast – moving is not fun

Posted by Chris on July 20th, 2011 filed in Uncategorized
Comment now »

Background: Moving downstairs. Same building, same layout, same service, just downstairs. Comcast won’t actually touch my equipment because it’s a TiVO – for the cablecard install we actually had to take it and insert it because he wouldn’t touch it. Previous tenants had service – it should just work, right?

Log of chat with only address, last name, and phone numbers changed.

Problem: Move service
Chris > My Issue: Move service
Dominic > Hello Chris, Thank you for contacting Comcast Live Chat Support. My name is Dominic. Please give me one moment to review your information.
Dominic > My pleasure to have you on this chat! Remaining committed and focused on my goal which is to provide quality customer service on my fullest effort will always be my top priority. Before anything else, I want to extend apologies for any trouble, inconvenience and frustration the issue has brought along your way. I still honestly hope you’re fine.
Dominic > Hi, Chris. How may i help you today?
Chris > I’d like to move my service
Dominic > I understand that you are requesting to move your service, is that correct?
Chris > yes
Dominic > I do apologize for the inconvenience that this has caused you.
Dominic > Let me help you with your account.
Dominic > Before we proceed, May I ask if we can verify the account first for security purposes.
Dominic > Thank you for providing your account information. Would you please verify the information I received is correct?
Dominic > Account Number blahaccountnumber
Dominic > First name Chris Last name Lastname
Dominic > Street Address1 Main Street unit #102
Chris > exactly right
Dominic > May i ask for the service address where you want to move the account, please.
Chris > 1 Main Street unit #101
Chris > same building, different unit number
Dominic > Thank you for the information.
Dominic > Since this would be a transfer of your service, let me transfer you to a colleague of mine who can better assist you.
Dominic > Kindly dont close the chat window.
Chris > ok
Dominic > Please wait, while the problem is escalated to another analyst
Marry Ann > Welcome to Comcast! My name is Marry. I see you are moving; that can be a very exciting and busy time. Please give me just a couple of minutes to review the information you have submitted.
Chris > My Issue: Move service
Marry Ann > I understand that you would like to transfer your services to (new address) on ***. Is this correct?
Marry Ann > Sorry for the typo. Please disregard that last message.
Marry Ann > I understand that you would like to transfer your Digital TV and High Speed Internet services to 1 Main Street unit #101. Is this correct?
Chris > yes
Chris > digital tv is supplied by the building
Marry Ann > Wonderful! You have reached the right department and I am more than willing to assist you with your transfer request today.
Marry Ann > Just to set expectations, this chat is the preliminary step of your transfer request. I will be checking the status of your account and the serviceability of your new address. After this, I will be giving you the steps to complete the order.
Chris > okay
Marry Ann > We will now proceed in checking the status of your current account.To assure protection of your account information may I have the: 1. Account holder’s full name 2. Phone number 3. Any of the following: a. FULL account number b. Last four digits of the SSN (if account number is not handy)
Chris > Chris, 111-111-1111, blahaccountnumber
Marry Ann > Excellent! Thank you for providing these information. I know these are confidential information and I truly value the trust that you have given.
Marry Ann > Please give me 2 minutes to pull up your account.
Marry Ann > Thank you for waiting, Chris.
Marry Ann > I have pulled up your account and I can see here that your account is in good standing. Thank you for being one of our good customers.
Marry Ann > Let me go ahead and check on the status of your address.
Marry Ann > Please give me 2 – 3 minutes to check on your account and the serviceability of your new address.
Marry Ann > Thank you for waiting, Chris.
Marry Ann > Great news! Your new address is Comcast serviceable and is good to go. I have confirmed that your address is serviceable. Let’s get started on setting up your service.
Marry Ann > We will now proceed with the transfer of your services.
Chris > ok
Marry Ann > May I ask your preferred disconnection and installation date please.
Chris > July 29
Marry Ann > Thank you for the information, Chris.
Marry Ann > I will be completing your order now. I will go over your order and make sure they will be provisioned the way you want your services will be setup. After which, I will be providing you with your order confirmation number. Please bear with me while I complete everything for you.
Chris > ok
Marry Ann > While I process your order, please provide the following information: 1) Nearest cross street or street perpendicular to your new home and 2) Two telephone numbers for our technician to call on the date of installation.
Chris > it is literally the same address just a different unit number
Chris > Main Street
Chris > 111-111-1111,222-222-2222
Marry Ann > Thank you for the information, Chris.
Marry Ann > Chris, I have arranged the remote deactivation of your services on July 29. You do not need to be around during the physical disconnection of the services, our technician will only be disconnecting from the pole.
Chris > the building has free service to all tenants, I don’t think he’ll be disconnecting anythign physically
Marry Ann > Okay, Chris.
Marry Ann > Since you are moving within the same market, you can take your equipment to your new address. The technician will take care of the rest of the equipments and/or materials during installation.
Chris > um there’s no need for installation. everything is literally exactly the same, except 9 floors down
Marry Ann > Chris it is the COMCAST procedure to let our Technician visit the new address to check the house for any possible problems with wirings to ensure that you will not compromise the value of your services. No need to worry though, I will check what I can do to help you with the setup costs.
Chris > It’s a time issue. If the previous tenants had cable and internet and have been moved out for less than a week, it will work.
Marry Ann > I do apologize that the new address that you are moving to is not illegible for self installation, Chris.
Marry Ann > There will be a one time installation charges for your installation of services which normally costs $99, but instead you will be receiving our discounted installation fee/s of $34.99 as a Movers premium, and this will be for our technician’s professional service fee. This charge will be added to your new bill at your new address and is not collectable during date of installation.
Chris > how about just changing the billing address and not moving service?
Marry Ann > Let me check that for you, Chris.
Marry Ann > Thank you for waiting, Chris.
Marry Ann > Chris let me verify, you will be transferring to this new address 1 Main Street unit #101, right?
Chris > yes
Marry Ann > Thank you.
Marry Ann > Chris, it is not possible to change the billing address since you are moving to a different apartment number, we need to have a technician to hook up everything for you to make sure that you can enjoy your internet service to the new address without any problem.
Chris > considering that last time the technician wouldn’t actually touch anything because we have a tivo (customer equipment), that’s amusing
Marry Ann > Shall we proceed with the transfer now?
Marry Ann > I understand Chris, no worries though, I can give you discount for the installation fee, normally it is for $99 but I can give you a discounted price of $34.99.
Chris > I understand that, think I’ll call support. Thanks for your time
Marry Ann > You are most welcome.
Marry Ann > You can call 1-800-934-6489
Chris > thank you for that, have a good day
Marry Ann > Is there anything else I can help you with today?
Chris > nope, thanks
Marry Ann > I will do the initial transfer by notating into your account your preferred disconnection and installation date, Chris.
Chris > thanks, that will speed things up
Marry Ann > Good luck with your move! Thank you for your time and patience. if you have any further questions, please do not hesitate to chat with us again. We’ll be happy to serve you. Have a good one!
Marry Ann > Bye.
Marry Ann > Analyst has closed chat and left the room

Perl one-liner to see who is hogging all the open filehandles

Posted by Chris on March 22nd, 2011 filed in programming, sysadmin

Helpful one-liner to help fix a problem we ran into the other day.

perl -e 'map{$c{(split(/\s+/))[2]}++} `lsof`;print "$_ $c{$_}\n" for (keys %c);'

The thinking is:

Use lsof to get all the open filehandles which conveniently also shows who has it open.


Loop through them, using the ` as a cheat that it inputs an array

map {   } `lsof`;

Splitting on whitespace.  The input to each iteration of the map{ } defaults to $_, and if you don't put anything to split in a perl split, it uses $_.  Neat.


Since we just care about the count, only use the 3rd column by forcing the output of the split into an array and using a slice.


Now, we just want the count for those users so we increment a hash with the user name as they key.

$c{ }++

Of course, the split is returning the name so that gives us the user name and hash key.


And increment that.  Unlike python, for example, you can just increment it.


It will do that for every iteration of the map{ }.  i.e. every line in the output of the `lsof`.

After that, it's just a matter of printing out the key/value pairs using a easy hash printing line blatently stolen from an answer on Stack Overflow.

Homemade switched outlet strip

Posted by Chris on September 13th, 2010 filed in Hacks

Before I moved into an apartment in the city, I had a house with a giant corner desk in the corner of my bedroom.  This was back before LCD screens, so it had three *giant* CRT monitors which sucked a fair amount of power, even when off.  That, coupled with other things that had no need to stay on all the time (ahem, always powered subwoofer speaker) made me want an easier way to turn them off to save energy.  Leaving digitallyimported on all day, every day, probably didn’t help too much either.  A little romex, some plastic outlet boxes (and outlets), a plug, and a couple of light switches and I had a solution.  I left it along with the desk when I moved, and have wanted another one ever since.  It is very straightforward – just like powering a lamp, except instead of a light bulb at the end, it’s a series of outlets.

Please check the update at the end – I used the wrong clamps.



  • 1 plug (I get two because I almost always mess up the first one)
  • 1 light switch
  • 1 light switch plate
  • 1 box for switch
  • 2 wire nuts (yellow)
  • Romex/nm cable – length up to you.  I used approx 15 feet

For each outlet:

  • 1 box
  • 1 outlet
  • 1 cover
  • Romex to length


Tools I used:

  • Screwdriver (screwgun for laziness)
  • Snips
  • Wire Stripper
  • Utility Knife
  • Electric multi-tool (continuity tester)
  • Needlenose Pliers
Outlet strip tools used

Tools used

Attaching the plug

Decide how long you want the “tail” to be – that’s the part going between the switch and the wall – and add a foot for good measure.  Strip about 3-4″ of the outer shield off, and get rid of the paper around the bare copper wire.  Push the bundle through the plug end.  Strip about 1/2″ off the other two wires.  The only tricky part is getting the end right – Black is the “hot” wire, and is on the right as you face an outlet.  It also always has brass (colored) screws.  I always use needlenose to create the loop in the end of the wire before screwing it in. After that, it’s just left to re-attach the plug cover.

Wires connectedreconnected

Creating the outlet strand

I start at the end of the line.  Take one of the metal boxen, and knock one of the circles out on the end – this is the only one that will have only one open hole.  Hopefully you’re using some sort of wire protectors – I got the wrong ones apparently… oops.  I’ll be re-doing this with the correct ones, by the way.  I put about a foot in between the boxes, so cut off a 18-20″ length of romex and stripped it as before.  It’s easier to put into the outlet box if you put the hook on the end after inserting.  Outlets are the same as the plug, but there’s another hint with them – the shiny copper is the hot (black) side.  Don’t foget the ground (bare copper) goes to the green screw.  After that, simply mount the outlet in the box.  Don’t forget to tighten down the protective clamp. I wait to put the covers on until I’m sure I haven’t messed it up.  Testing as I go makes that even less of a problem.

After the first one, it is a very simple manner of doing the same thing for however many boxes you have.  Basic procedure:

  1. Knock out end(s)
  2. Add protection
  3. Cut wire to length, plus 6-8″
  4. Strip outer cover off romex
  5. Get rid of paper around ground wire
  6. Strip the black and white wires
  7. Insert wire into box
  8. Make the bends in the three wires
  9. Attach to hot side, cold side, and ground
  10. Repeat for other end going out the other side of the box, and to the other side of the outlet block (excepting the ground)
  11. Mount outlet
  12. Tighten protective clamps (Get the right ones)
  13. Test for continuity where it should be (and where it shouldn’t)
  14. Put covers on

Knockout box with steel connector

Outlet partly wired, in the series

A finished box in the series.

Connecting the switch

In this case, I knew where I wanted to mount my switch, and wanted both wires to come out of the back.  (Another advantage of the metal knock-boxes, by the way).  It’s actually hard to find single pole switches anymore, but not a big deal.  The switch will be marked “up”, and on there back there will be a pole that is differently colored.  That’s the “hot” or “common” one to connect to the outlet.  That one will always have live electricity going to it, no matter how the switch is oriented!   I’m lazy, so I used the continuity tester to see which of the other poles would be “on” when the switch was on.  The black wire from the wall cord goes to the always hot pole, and the black from the outlet strip goes to the one you just found that’s switched how you want it.  The two white wires and the two bare wires get connected via the wire nut (or grounding screw on the switch).

Switch wired up. (Only black goes to the switch)


Connect one lead of your tester to the cold pole of the plug (left as it faces the wall).  First, check there is no connection to either of the other two prongs on the plug.  Then,  going down the line, check that the meter says it’s connected to the cold half of the outlet at all of the outlets (you can just do one per pair, they’re connected within the block).  Going back down the line check that the meter says it is *not* connected to any of the hot sides or the grounding plugs.  Switch to the hot pole on the plug and leave the switch off.  It shouldn’t read as connected to anything down the line of plugs now.  Turn the switch On, and make sure it’s only connected to the hot plugs, and that the prongs still don’t read that they are connected.  If any of those steps go awry, there’s something somewhere it shouldn’t be…  I check once in a while as I go just so I don’t get confused – I just leave one lead in one side of the outlet at the end and check after I finish a plug.

Checking everything for continuity

Finishing up

If you haven’t already put the covers on, now would be a good time.  I plan to mount this facing up on the square steel going horizontally across my desk back so I attached some reasonably strong hook and loop fasteners to the back of each box.  The switch goes in the front of the desk in the upper right corner where it’s easy for me to switch on and off as I come and go, so I put it on the top and right.  To make it easy – “measure twice, cut once.  or better yet, don’t measure at all” – I put the other half of the hook and loop combo on the back of each box and just pressed it into place rather than trying to match up.  It’s held up for a while now with absolutely no worries.  Plug it into the wall (or a power strip), and plug something cheap into one of the plugs ( I used a monitor… ahem ) and flip the switch.  Done!

Updates, because I messed up:

  • Get the right clamps!  The wrong ones will either run a screw into the wire or end up fraying it causing a dangerous situation.  I have the wrong ones in these pictures.  I got done taking pictures and re-did the entire strip with the correct (safe) ones.
  • This is not for moving around at all.  Solid wire shouldn’t be moved/bent.  The only reason I’m comfortable using this is because it’s close to a permanant install.
  • Plug it into a real power strip.
  • There’s really not many reasons to do something like this instead of a good power strip.

A couple of references:

How to wire a household outlet

Wiring a 3-way switch

Using a multimeter

How to wire a plug (part of a hackaday article)

More Chicago transformers info

Posted by Chris on July 9th, 2010 filed in Uncategorized
Comment now »


Central Business District Advisory

July 9, 2010

Transformers 3 Filming – Street Closures and Central Business District Activity


Transformers 3 will be filming in the downtown area this weekend beginning tonight after the evening rush through 5am Monday morning; please review the following information:

Please remind employees and tenants of the parking restrictions as outlined below:

Streets that will be permitted for “No Parking” from 6:00PM on Friday, July 9th through 5:00AM on Monday, July 12th:

* Both sides of LaSalle from Lake to Jackson
* Both sides of Randolph from Wells to Clark
* Both sides of Washington from Wells to Clark
* Both sides of Madison from Wells to Clark
* Both sides of Monroe from Wells to Clark
* Both sides of Adams from Wells to Clark

* Both sides of Jackson from Wells to Clark
* West side of Clark from Lake to Congress
* Both sides of Franklin from Lake to Randolph

Streets that will be permitted for “No Parking” from 4:00PM on July 9th through 5:00AM on Monday July 12th:

* Both sides of Wells from Randolph to Jackson

LaSalle Street will be closed between Randolph and Jackson from 7:00PM on July 9th through 5:00AM on July 12th.

Washington will be closed between Wells and Clark from 7:00PM on
July 9th through 5:00AM on July 12th.

“Transformers 3”
Chicago Filming Dates:
July 10 – August 23


Total 7 weeks of filming

Majority to take place in the Central Business District

In most cases, filming to take place during daylight hours (approximately 7:00am-8:00pm). Set-up and strike the nights before and after filming.

8:00pm Friday July 9 – 5:00am Monday July 12
LaSalle Street closed from Jackson to Wacker

6:00am Saturday July 10 – 5:00am Monday July 12
Washington closed from Wells to Clark
Randolph closed from Wells to Wacker

3 days of filming: Friday (7/16), Saturday (7/17), Sunday (7/18)
Road Closure: Michigan Ave from Wacker to Ontario
8:00pm Thursday (7/16) through 5:00am Monday, (7/19)

3 days filming: Monday (7/19), Tuesday (7/20) & Wednesday (7/21)
Road Closure: Wacker between Michigan and Wabash
9:00pm Sunday (7/18) through 5:00am Thursday (7/22)

The Chicago Fire Department and Chicago Police Department will supervise the use of special effects and pyrotechnics, including:

simulated gunfire




For brief periods between the hours of 7:00am – 7:00pm on both Sat July 10 & Sun July 11, helicopters will be used for both aerial photography and stunt parachutists in the vicinity of:

Willis Tower

LaSalle Street Financial District

East Randolph Street & Daley Bicentennial Plaza

For safety reasons, the Chicago Police & Fire Department will need to keep spectators at a distance. Since the Transformer robots are all digital creations, you won’t be able to see them during the filming.

If you have any questions or concerns Please contact Patrick Muldoon, Key Assistant Location Manager at 312-213-3820or callRich Moskal,Director with the Chicago Film Office, 312-744-6415 (office), 312-497-1678 (cell) or email Rich at [email protected]

(Information provided by Sgt. Ron Graziano, First District Business Liaison, Chicago Police Department.)

BOMA/Chicago has uploaded several published notices provided by the production company and public agencies about the filming. They are available at Follow the link in the Transformers 3 story on the homepage.

This information has been provided to BOMA/Chicago by a state, local or federal agency and is being distributed to BOMA/Chicago members.

Why I’m disabling javascript by default

Posted by Chris on April 8th, 2010 filed in Uncategorized
Comment now »

Opera is my friend.  Preferences, Advanced, Content.  Disable javascript.  Suddenly the web is fast again.  I’m on a DS3, the internet should be super fast.  But sites have so much junk on them now that has to load before the content that it takes multiple seconds for things to come up when it should be near instant.  Articles on now load in less than a second rather than 7-10 seconds.  The consumerist loads in 1-2 rather than between 10 seconds and never.  GPF can’t break free of it’s frame anymore.  They all now load quickly, and all the content is exactly the same.  I don’t have a problem with sites running things! Be clever.  If I need to I can re-enable it very easily (unlike other browsers….).

I read all my news through RSS.  I don’t like how google’s feed reader works, it just doesn’t think the same way I do.  So I have my own.  And some sites don’t show the full content, so I pop the page itself inside an iframe in my reader.  Some (GPF, for example) want to break out for some reason.  I’m seeing all the ads on your page, seeing it in the same dimensions you want me to see it in, what more do you want?  You can have my attention, but only to the extent that my attention doesn’t stray from my reader mindset.

I’ve re-enabled it only for internal websites and Gmail so far and had no problems.  No more of those irritating balloons if you accidentally mouse anywhere within the article.  No netflix popunders (I thought we killed those) if I click anywhere on a page.  I’s wonderful.

The internet is a faster, less distracting place again.  Maybe they’ll give me reasons to enable it again, but I’m doubting it.

High frequency trading – Should I work for a trading company?

Posted by Chris on March 4th, 2010 filed in Trading

So you’ve graduated a fine  university with a degree in math, economics, CS, or some other actually useful piece of paper showing you know (theoretically anyway) how to do something of use to society.  Congratulations!  At job fairs all over the country at the best schools, or perhaps the schools that company recruiters went to anyway, trading companies want to hand you the keys to a decent salary, possible ridiculous bonus, hot secretaries, and the best office digs money can buy.  Oh, and a couple billion dollar trading limit if that’s what you’re into.

Should you take it?

Do you like money?

Do you like money more than anything else in the world?

That’s the question you have to ask yourself.  You’ll make more than enough to live on.  You’ll probably get a bonus ranging from a couple months salary to the type of money your parents wouldn’t be comfortable thinking about.  The office will be wonderfully furnished, and the work will be interesting.  They’ll only hire the hottest HR women, the prettiest secretaries.  Free drinks, snacks, bar after hours, a car service, free insurance, paid breakfast lunch and dinner.  It’s a crazy world.

It’s also all you’ll have.

Traders – you’re in it for money.  That’s what you *do* after all.  Play along for a couple of years, save your bonuses, and start a restaurant.  Live a life of boring luxury if you want.  IT folks, you’re my people.  I don’t do it for the money – it’s sure as heck nice though.  You probably do it because it’s coolest work you’ll ever find.  You can use your beloved language (ahem, perl) because it’s what you’re the fastest and most productive in. The best, fastest, newest everything you can find.  But you’re working for the traders.  The traders make the company money, you are an expense.  Sure, they give lip service saying that you’re all equal, but you’re not, are you?  When they stay late so they can have more profits on their books it’s a measurable thing that shows up at the end of the year in their bonus.  When you stay late, it’s so you don’t get yelled at.  Uptime doesn’t measure to actual dollars.

Why you should say yes

  • It pays well!
  • There’s no place in the world like the financial industry.  The companies nearly literally do nothing but create money.  Salaries are good, bonuses are better.  I don’t know of anyplace in the world you can make as much.  Period.
  • Benefits
  • Benefits match salaries. 100% paid health, vision, dental, AD&D, life insurance policies.  Free food.  Paid transportation – public, light rail, parking passes.  Drinks and frequent outings and parties.  Snacks and gym memberships.
  • It’s interesting work
  • Figuring out low latency systems, writing programs that handle billions of dollars, tying systems made of dozens or hundreds of smaller programs together.  New algorithms, different languages, the flexibility to change things.  All make for time flying doing cool things.
  • Smart people
  • The smartest people money can buy!  They’re not hidden in a think tank or back room either – they’re right next to you making your brilliant ideas sound silly and trivial in comparison.  So many great people to learn from.  Steal their book lists, read their favorite websites, pick their brains.  Making you smarter makes the company money too, after all.
  • Office environment
  • Ever seen a trader’s station?  The only limiting factor in the number of screens they get is their peripheral vision.  Six 30″ lcd’s orientated vertically with 19″ monitors on top of that.  Tech people get the hand-me-downs, of course, but you still get rather good hardware usually.  Aeron chairs, anything ergonomic the office manager can find.  Shiny toys bought by vendors for the company.  Giant-screen TV’s everywhere.  Kitchen to hang out in.  Everything is beautiful and functional.
  • Cool toys
  • Brand new routers, switches, servers.  Fiber.  Network cards with their own operating system.  Fiber fabrics that use different frequencies of light to get more bandwidth.  GPS time servers.  At least gigabit everything.  Terabyte storage arrays.  Entire floors of a building designed around cooling.  Datacenters.  It’s really freaking cool.

Why you should say no

  • You’re admitting you’re a whore
  • There’s no nicer way to put this one.  You are saying that  you value money over anything else.  Health, morality, family.  You’re only in it for the money, and the company you work for doesn’t do anything but make money.
  • You will burn out
  • Ten hour days will get you fired for not showing up.  The pace is frantic, at the end of the day your brain melts.  Adrenaline from the awesome parts of the job doesn’t do it after a while.  Sleep is for poor people.
  • The work stops being interesting
  • Yes, you can spend another year working to make your system trade faster.  Just like last year.  There isn’t even a target to shoot for.  Just “faster”.
  • Smart, but narrowly focused people
  • The mighty Wolfram is famous for this.  “I’ll see you this weekend” “No, I’m watching the super bowl.”  “What’s a super bowl?”  The people around you (and you, by extension) focus so much on the ways to make the company more money that nothing else exists.  You have to be so good at what you do to stay ahead that that’s all you can focus on.
  • Work-life balance does not exist
  • See also You will burn out, above.  They buy you for the hefty fees you are making.  Wake up early to get in before market open, work, then study your job when you get home.  We were actually planning to put convertible couches into the conference rooms for sleeping.
  • Chaotic work environment
  • Ever been to a casino?  That’s a lot like the floor.  You don’t even have the token courtesy of cubicle walls to stop the ringing, talking, yelling, sheep noises, etc.  Add to that constant interruptions and a lot of work gets done.  Badly.
  • Paranoia
  • These companies are secretive, and for good reason.  Everything from hardware, to network paths, to what programming language is fodder for the enemy.  Trading is a zero-sum game, after all.

Is it worth it?

Hell yes.  Or maybe not.  Try it.  If it doesn’t go against your ethics, you don’t mind taking a few years off from productive life, and can keep yourself broad enough while making it in the world there, it’s an amazing experience.  And the money doesn’t hurt.

At least if you can handle the negatives.  And they’re real.  You’re supporting traders.  You can be the head of IT, the lead SysAdmin, the brilliant network engineer.  Whatever.  You’re a monkey who works for the lowest trader.  If the company talks about work-life balance, it’s because they’ve had trouble with people quitting recently and that’s what they mentioned.  If they don’t, it’s because they don’t believe in it.  That said, it’s amazing work.  It’s fast work.  It’s educating work.  It’s inspiring and depressing.  If you can disassociate yourself from what you do, and be awesome at doing it, it’s really really worth it.  Personally?  I have no problems admitting I do what I do for money if the job is interesting enough that I’d do it anyway.  Give more to charity if it bothers you.  If you can’t handle time time, though, you’re not going to do well.  All I can do is try to present it so you can make an educated choice.

High Frequency Trading – Access and the Datacenters

Posted by Chris on March 4th, 2010 filed in Trading
Comment now »

The access to the market(s) is interesting enough to warrant it’s own page.


For near exchange work, it’s all about proximity.  If you can get a cross connect to an exchange,  you want to be as physically close to them as you can.  In the same datacenter is a minimum.  On the same floor.  In the same area.  As physically close as possible.  On a trip to a datacenter, one of the jobs is scoping out what’s going on.  That new expansion is where the exchange is going?  You should be able to get access to pick your spot.  All that info is secret, of course, but that doesn’t matter.  You know where the exchange is going, that giant cage with custom power and cooling next to it?  That’s Citadel, I heard.  But you can’t compete with Citadel that way, but you can be the next best spot.

This is the area where you’re measuring in microseconds.   The differences here can also be made up in other ways.  Matching your routers to the exchange you’re connecting to.  Will a Cisco or a Juniper have better time?  Tune the algorithm.  Measure, measure, measure.

If you can’t get a cross connect directly, or you have lines coming in from elsewhere, your line makes a trip to the “meet me room” or some variation on the theme.  That’s where all the customers of the datacenter have their patches (that don’t go directly to someone else), and get connected to other folks.  Your outside line, your other cages maybe, exchanges, whoever it is you have to talk to.  Then somewhere in your cage or cabinet you have a patch panel where you can finally connect it to your equipment.

Of course, sometimes you get buildings like where the TSX floor is located.  You can’t just connect your equipment on one floor to another floor – that would be too easy.  You have to connect at your floor, it goes all the way down to the meet me room in the basement, and then all the way back up to the other location you’re connecting to.  All at a fee, of course.

The datacenter itself

The datacenters themselves are amazing in almost every way.  The way they’re not amazing?  Appearances from the outside.  Unmarked.  Actively nondescript.  Could be a warehouse, could be an office building.  Some  are multiple-block-long buildings out in New Jersey.  Some are closets in skyscrapers.  Some are multiple story former printing press buildings.  Some you can almost throw a stone to one of the busiest roads in the world.

One has a great view of the Statue of Liberty.

There tend to be similarities, though.  There are rows and rows of either racks, cages, or a combination of the two.  There’s alternating hot and cold aisles.  Cold air is forced either through the floor or from above via vents into the cold aisle where the servers draw it in.  The hot aisles collect the output from both sides and either take it in or let it go naturally up into an open space.  There are overhead ladders with network wiring that then breaks off and goes into the desired cages.  The wiring for the power is the same story as the cold air.  Either above or below.

The biggest limiting factor?  Power and a/c.  You can usually get space, and with that space comes a little bit of power.  How much?  Between 15 and 30 amps for the whole cabinet.  A 42 U space with between 8  and 18 or so used.  Hope you’re using 220v power for that extra efficiency!

Working in a datacenter is, like most things in tech and trading, an exercise in contradictions.  They’re super cold so you’re freezing, but if you’re moving equipment, you’re still sweating.  They’re also humidity controlled, so you’re usually thirsty – but there’s absolutely no liquid allowed inside.  You’re probably hooked up to one of the most advanced networks in the world, but getting online is a pain.  Wifi is strictly prohibited except for very temporary setups for working.  You brought your laptop, but there’s no place to plug it in!

That’s legit – no place to plug it in.  The cage has a PDU designed for use in cages like this, and with servers and networking equipment.  Like this:  If you didn’t have the foresight to order an adapter and/or your laptop won’t work on the voltage your cage is at –  then you have no power!

There’s also no food allowed, but there’s usually a break room where people stash goodies for later.  Bring drinks for there too, and label your bag.  The other problem is snack food does not a meal make, and they usually (on purpose) make these things as far from civilization as possible!  Usually there’s a small group working at the same time, so when you’re not going flat out as fast as you can, you’re sitting and waiting to be useful again.  Of course, after a few trips out you generally just start sneaking candy and drinks in anyway.  “Rules” eh?

If you’re lucky enough to have a whole cage of whatever size, you don’t have enough power to fill it up anyway.  So make yourself a work area.  Get a folding table and chair, keep tools around, have all the lengths and colors of cabling you use on hand, have a usable power strip and a couple extra ports.  It’s like a cold, loud home away from home!


When working with two or more exchanges, it’s more about latency than proximity.   Verizon guarantees 8.5 milliseconds per 1,000 miles – but that’s as the wire lays.    It’s 710 miles from Chicago to New York.  What time will your packet make the round trip?  Well, if you have a good line, 18ms is pretty good.  Less than that and you can make a business out of it. These folks for example advertises 17.2ms.  There’s rumors about 16.8 and less.  I want to know what the bandwidth along I-80 is.

When in a group of industry folks, someone will talk about a new path to NJ they’ve found.  “Does it follow I-80?” “Better!”  But you won’t get anything else out of them.

Within metropolitan areas, you still have to find the best paths between places.  A few hundred microseconds are too many to waste.

What is  dark fiber exactly?  Well, it’s a piece of fiber optic cable that you connect to your equipment at both ends.  It’s literally dark until you “light” it up. There’s no telephone company routers in between to change the pathing or add latency.

Measure twice.  Actually, just keep measuring.

Everything – everything everything depends on your measurements.  When did we send it, when do they say they got it, when did we get it back.  Does that all match what the router says?

I used to ask family members what the most important part of the measuring operations was.  No-one guessed knowing the time.

You have got to have a good time infrastructure.  NTP is okay, but where are you getting your TP from?  The internet is way too slow to be accurate enough for what we’re measuring here.  You need a network time appliance in every metropolitan area.  Every single datacenter would be awesome.  Meinberg is the gold standard.  It’s cool technology.  The other problem with NTP is it’s not accurate enough in practice (in theory it is, but that’s a nice theory).  PTP.  How accurate is accurate enough?  When you’re measuring the time it takes for an update to get into your decision engine until the time it takes to make a decision to measure how good your software is, you need in the nanos.  I’m not saying single-digit nanosecond resolution.  (Yet)

And throughout the day, things can change.  Your monitoring infrastructure has to keep up not only with the hardware running your network, but the Quality of the Service on it, too.  Things happen.  Fiber gets cut.  The telco re-routes you without any notice. Suddenly you’re getting picked off because you’re a millisecond slower than everyone else.

It’s better to drop out of the market and take the opportunity cost than to be in and making other people money.