High Frequency Trading – Clearing

Posted by Chris on March 4th, 2010 filed in Trading

Prop trading shops don’t just get free access to all of the exchanges they’re on.  Seats are expensive, and for what they’re doing, not really necessary either.  They generally get something called Sponsored Access.  A big company will sponsor them on the exchange in return for a fee.  But it’s not just access to the exchange.  If you’re a small trading company, you may have a few dozen million dollars in the real world.  So you prove that to, say, Goldman Sachs and they say “Sure, we’ll sponsor you.  And so you can trade more, here’s some leverage to play with too!”.  Of course, they bill you for how much you trade, so it’s in their interest to let you leverage yourself.  So now your measly 8 figures has turned into 10 or more figures of leveraged money.

Look up haircut sometime if you want to know how it all relates to how much you can leverage.

So, your sponsor takes on the risk of you trading somewhat in their name.  At the end of the day, they have a list of what you traded, and you have a list of what you traded.  The folks in mid-office then have to deal with your clearing firm and your sponsor so everyone agrees just what happened when the traders were out playing all day.  The clearing firm is the go-between the two parties of the trade.  Basically, they’re a really valuable lubricant in the wheels of the market.  So GS gets the list from your clearing firm, you send what you think you traded, and hopefully you don’t get a call from them asking where the 10,000 shares of Berkshire are in your list.

Clearing – in practice

It’s fairly simple actually.  You get a method by which you send  your report.  Let’s say FTP.  You upload it with a certain filename format to a server with a given username and password.  Easy.  The hard part is generating the file.  You could have one database, you could have a hundred.  There could be one format, or dozens.  You could have to download and parse text files.  And don’t forget versioning either.  So you download all the executions you made.  Then you have to group them by exchange, symbol, and side, and add them up and average them out. So your file goes from, say, 200,000 shares of AAPL traded to three lines:

AAPL,B,100203, 190.91,blah,blah

You combine all of them together for that particular exchange, in that particular format, and upload it to  your firm.  They compare it with their (much simpler to generate, because they only have one potential input) file and if there’s a problem they email and/or call.  It’s a surprisingly simple program in perl.  Collect your data from all the different sources, put it into a huge hash, print it out to a file, upload it.  Put in error checking and notification.  Do this for every exchange you trade on, at the specified time(s) daily.

Only two important things to note:

If that file is wrong or doesn’t get uploaded – they don’t trade the next day until it’s fixed.

“The next financial crisis will be caused by a divide-by-zero error in someone’s perl script” (Citation needed)


A huge amount of time and money is spent on compliance at these firms.  And at the same time it’s a total afterthought.  The traders rule the roost, and they don’t care about it –  until they can’t trade anymore or their bonus is lower because of a failed audit.  Essentially here’s what it comes down to.  You have to keep track of every trade and every order you make, and keep it forever.  Sure, they say seven years, or five years, but it’s forever.  For one – seven years really is forever in computer terms.  Creating something that’s archivable over that long is essentially creating it forever.  And for two – there’s nothing saying that a year from now they won’t want the records for 10 years, or 15 years.

So you’re keeping track of every order and every trade.  While you’re doing it, you have to keep your positions at the time if at all possible.   Why?  Shorting stocks for example.  Traders are allowed to short, but it goes against the haircut and reduces the amount they can trade.  If it’s on the easy-to-borrow list though, it’s a different story.  Anyway, the regulators come in and you have to prove that you weren’t shorting a stock.  Which means you have to know or be able to derive your position at that time.  Then you have to prove that at the millisecond you placed the order, it wouldn’t be a short.  Which also means accurate time across the board of your systems.  It’s fun, see?

The easy to borrow list is exactly like it sounds.  A list of things that are easy to borrow.  Some stocks are so available to trade – so liquid – that they get on a special list that doesn’t really count as short selling! Link

In addition to ordering what and when, you also have to be able to prove who.  So every trader has to have a unique login so you can prove who did what.   If Bob over in the corner is doing something illegal or unethical, you have to be able to prove it was bob doing it.  That way when he’s caught he’ll just take himself down and not the company.  So you have to record every method by which someone can communicate with evil peoples.  You have to keep (like I said, pretty much forever):

  • Email
  • Chat
  • Phone
  • Tin can and string
  • Any other electronic medium

Funny story – one time the network went down for some reason.  Most places use IP phones now, which use the network.  Just to be safe, we had to call the exchanges to cancel our orders – but no phones!  So we all whip out our cell phones and call that way.  1- You can’t do that since they’re not logged and monitored. and 2- “Does anyone have something other than AT&T?  I can’t call anyone!”

Email has to go through your email servers and be backed up, instant messaging has to go through a proxy that logs all conversation, phones (at least for traders) have to keep recordings, etc etc.  One of my better scripts was actually to do a daily dump of chat logs, bundle them per user, and send it as an email to a special email address.  Killing two birds with one stone!   Regulators will seriously come in and say “we want every communication trader X made from this date to this date.”  And you say “okay” or you get a fine.  They also do spot-checks at least once per year.  “Give us everything you traded from this date to this date.”  And you say “okay” or you get a fine.  Incidentally, if you want to see roughly what the code for chat log retention looked like:

use strict;use warnings;
use include;
my $ch='chat_host';
my $cu='chat_user';
my $cp='chat_pass';
my @worries=();
my $to='[email protected]';
my $today=&include::getdate('YYYY-MM-DD');
my @senders = &include::sql($ch,$cu,$cp,"select distinct(senderid) from message_log where date(modified)='$today'")
  or &freakout($!);
foreach my $sender(@senders)
  my @lines=&include::sql($ch,$cu,$cp,"select msg_text from message_log where date(modified)='$today' and senderid='$sender'");
    push(@worries,"No lines for $sender");
  &include::send_email($to,join("_",split(" ",$sender)).'@backuphost.mycompany.com','Chat logs for $today',join("\n",@lines))
    or push(@worries,$!);
  &include::send_email('[email protected]','[email protected]','Possible problems with chat backups',join("\n",@worries));
sub freakout
  my $wtf=shift;
  &include::send_email('[email protected]','[email protected]','Error with chat logs for $today',$wtf);

It’s a  simplified but only because the hard stuff is hidden in my include module I keep around to make my life easier – it was(is) literally this short.  We traded every single day and there was chat (and chatbots) every day, so if there wasn’t anything in the logs there was definitely a problem.  In plain language:

Do some housekeeping (includes, variable setup for hostnames and the like, the date, etc)
Get a list of everyone who sent an IM that day and if there's a problem, send a freakout email with what happened.
For every person in that list, send an email with their name as the "from" address and the chats they sent as the body
  (if there's a problem, add it to the list of worrying things)
If anything is worrying, send an email to that effect to me.

The reason this is so simple is kind of a lucky coincidence.  The chat logs are in one table, the ID is the name of the person, and the text of the log includes the time and who it’s to.  If it didn’t, I’d have to have another line to get the identity of the person – no big deal.  I’d also have to have a temporary array of lines that I’d append the “to”, “time”, and “text” to as one entry, then send the email with the contents of that array instead.

One Response to “High Frequency Trading – Clearing”

  1. Shawn Says:

    You go a little far a field from clearing here. You also could go into reconciliation a little bit.

Leave a Comment