Should you take it?

Do you like money?

Do you like money more than anything else in the world?

That’s the question you have to ask yourself.  You’ll make more than enough to live on.  You’ll probably get a bonus ranging from a couple months salary to the type of money your parents wouldn’t be comfortable thinking about.  The office will be wonderfully furnished, and the work will be interesting.  They’ll only hire the hottest HR women, the prettiest secretaries.  Free drinks, snacks, bar after hours, a car service, free insurance, paid breakfast lunch and dinner.  It’s a crazy world.

It’s also all you’ll have.

Traders – you’re in it for money.  That’s what you *do* after all.  Play along for a couple of years, save your bonuses, and start a restaurant.  Live a life of boring luxury if you want.  IT folks, you’re my people.  I don’t do it for the money – it’s sure as heck nice though.  You probably do it because it’s coolest work you’ll ever find.  You can use your beloved language (ahem, perl) because it’s what you’re the fastest and most productive in. The best, fastest, newest everything you can find.  But you’re working for the traders.  The traders make the company money, you are an expense.  Sure, they give lip service saying that you’re all equal, but you’re not, are you?  When they stay late so they can have more profits on their books it’s a measurable thing that shows up at the end of the year in their bonus.  When you stay late, it’s so you don’t get yelled at.  Uptime doesn’t measure to actual dollars.

Why you should say yes

• It pays well!

There’s no place in the world like the financial industry.  The companies nearly literally do nothing but create money.  Salaries are good, bonuses are better.  I don’t know of anyplace in the world you can make as much.  Period.

• Benefits

Benefits match salaries. 100% paid health, vision, dental, AD&D, life insurance policies.  Free food.  Paid transportation – public, light rail, parking passes.  Drinks and frequent outings and parties.  Snacks and gym memberships.

• It’s interesting work

Figuring out low latency systems, writing programs that handle billions of dollars, tying systems made of dozens or hundreds of smaller programs together.  New algorithms, different languages, the flexibility to change things.  All make for time flying doing cool things.

• Smart people

The smartest people money can buy!  They’re not hidden in a think tank or back room either – they’re right next to you making your brilliant ideas sound silly and trivial in comparison.  So many great people to learn from.  Steal their book lists, read their favorite websites, pick their brains.  Making you smarter makes the company money too, after all.

• Office environment

Ever seen a trader’s station?  The only limiting factor in the number of screens they get is their peripheral vision.  Six 30″ lcd’s orientated vertically with 19″ monitors on top of that.  Tech people get the hand-me-downs, of course, but you still get rather good hardware usually.  Aeron chairs, anything ergonomic the office manager can find.  Shiny toys bought by vendors for the company.  Giant-screen TV’s everywhere.  Kitchen to hang out in.  Everything is beautiful and functional.

• Cool toys

Brand new routers, switches, servers.  Fiber.  Network cards with their own operating system.  Fiber fabrics that use different frequencies of light to get more bandwidth.  GPS time servers.  At least gigabit everything.  Terabyte storage arrays.  Entire floors of a building designed around cooling.  Datacenters.  It’s really freaking cool.

Why you should say no

• You’re admitting you’re a whore

There’s no nicer way to put this one.  You are saying that  you value money over anything else.  Health, morality, family.  You’re only in it for the money, and the company you work for doesn’t do anything but make money.

• You will burn out

Ten hour days will get you fired for not showing up.  The pace is frantic, at the end of the day your brain melts.  Adrenaline from the awesome parts of the job doesn’t do it after a while.  Sleep is for poor people.

• The work stops being interesting

Yes, you can spend another year working to make your system trade faster.  Just like last year.  There isn’t even a target to shoot for.  Just “faster”.

• Smart, but narrowly focused people

The mighty Wolfram is famous for this.  “I’ll see you this weekend” “No, I’m watching the super bowl.”  “What’s a super bowl?”  The people around you (and you, by extension) focus so much on the ways to make the company more money that nothing else exists.  You have to be so good at what you do to stay ahead that that’s all you can focus on.

• Work-life balance does not exist

See also You will burn out, above.  They buy you for the hefty fees you are making.  Wake up early to get in before market open, work, then study your job when you get home.  We were actually planning to put convertible couches into the conference rooms for sleeping.

• Chaotic work environment

Ever been to a casino?  That’s a lot like the floor.  You don’t even have the token courtesy of cubicle walls to stop the ringing, talking, yelling, sheep noises, etc.  Add to that constant interruptions and a lot of work gets done.  Badly.

• Paranoia

These companies are secretive, and for good reason.  Everything from hardware, to network paths, to what programming language is fodder for the enemy.  Trading is a zero-sum game, after all.

Is it worth it?

Hell yes.  Or maybe not.  Try it.  If it doesn’t go against your ethics, you don’t mind taking a few years off from productive life, and can keep yourself broad enough while making it in the world there, it’s an amazing experience.  And the money doesn’t hurt.

At least if you can handle the negatives.  And they’re real.  You’re supporting traders.  You can be the head of IT, the lead SysAdmin, the brilliant network engineer.  Whatever.  You’re a monkey who works for the lowest trader.  If the company talks about work-life balance, it’s because they’ve had trouble with people quitting recently and that’s what they mentioned.  If they don’t, it’s because they don’t believe in it.  That said, it’s amazing work.  It’s fast work.  It’s educating work.  It’s inspiring and depressing.  If you can disassociate yourself from what you do, and be awesome at doing it, it’s really really worth it.  Personally?  I have no problems admitting I do what I do for money if the job is interesting enough that I’d do it anyway.  Give more to charity if it bothers you.  If you can’t handle time time, though, you’re not going to do well.  All I can do is try to present it so you can make an educated choice.

Look up haircut sometime if you want to know how it all relates to how much you can leverage.

So, your sponsor takes on the risk of you trading somewhat in their name.  At the end of the day, they have a list of what you traded, and you have a list of what you traded.  The folks in mid-office then have to deal with your clearing firm and your sponsor so everyone agrees just what happened when the traders were out playing all day.  The clearing firm is the go-between the two parties of the trade.  Basically, they’re a really valuable lubricant in the wheels of the market.  So GS gets the list from your clearing firm, you send what you think you traded, and hopefully you don’t get a call from them asking where the 10,000 shares of Berkshire are in your list.

Clearing – in practice

It’s fairly simple actually.  You get a method by which you send  your report.  Let’s say FTP.  You upload it with a certain filename format to a server with a given username and password.  Easy.  The hard part is generating the file.  You could have one database, you could have a hundred.  There could be one format, or dozens.  You could have to download and parse text files.  And don’t forget versioning either.  So you download all the executions you made.  Then you have to group them by exchange, symbol, and side, and add them up and average them out. So your file goes from, say, 200,000 shares of AAPL traded to three lines:

AAPL,B,100203, 190.91,blah,blah
AAPL,S,101238,191.33,blah,blah
AAPL,SS,2231,191.02,blah,blah

You combine all of them together for that particular exchange, in that particular format, and upload it to  your firm.  They compare it with their (much simpler to generate, because they only have one potential input) file and if there’s a problem they email and/or call.  It’s a surprisingly simple program in perl.  Collect your data from all the different sources, put it into a huge hash, print it out to a file, upload it.  Put in error checking and notification.  Do this for every exchange you trade on, at the specified time(s) daily.

Only two important things to note:

If that file is wrong or doesn’t get uploaded – they don’t trade the next day until it’s fixed.

“The next financial crisis will be caused by a divide-by-zero error in someone’s perl script” (Citation needed)

Compliance

A huge amount of time and money is spent on compliance at these firms.  And at the same time it’s a total afterthought.  The traders rule the roost, and they don’t care about it –  until they can’t trade anymore or their bonus is lower because of a failed audit.  Essentially here’s what it comes down to.  You have to keep track of every trade and every order you make, and keep it forever.  Sure, they say seven years, or five years, but it’s forever.  For one – seven years really is forever in computer terms.  Creating something that’s archivable over that long is essentially creating it forever.  And for two – there’s nothing saying that a year from now they won’t want the records for 10 years, or 15 years.

So you’re keeping track of every order and every trade.  While you’re doing it, you have to keep your positions at the time if at all possible.   Why?  Shorting stocks for example.  Traders are allowed to short, but it goes against the haircut and reduces the amount they can trade.  If it’s on the easy-to-borrow list though, it’s a different story.  Anyway, the regulators come in and you have to prove that you weren’t shorting a stock.  Which means you have to know or be able to derive your position at that time.  Then you have to prove that at the millisecond you placed the order, it wouldn’t be a short.  Which also means accurate time across the board of your systems.  It’s fun, see?

The easy to borrow list is exactly like it sounds.  A list of things that are easy to borrow.  Some stocks are so available to trade – so liquid – that they get on a special list that doesn’t really count as short selling! Link

In addition to ordering what and when, you also have to be able to prove who.  So every trader has to have a unique login so you can prove who did what.   If Bob over in the corner is doing something illegal or unethical, you have to be able to prove it was bob doing it.  That way when he’s caught he’ll just take himself down and not the company.  So you have to record every method by which someone can communicate with evil peoples.  You have to keep (like I said, pretty much forever):

• Email
• Chat
• Phone
• Tin can and string
• Any other electronic medium

Funny story – one time the network went down for some reason.  Most places use IP phones now, which use the network.  Just to be safe, we had to call the exchanges to cancel our orders – but no phones!  So we all whip out our cell phones and call that way.  1- You can’t do that since they’re not logged and monitored. and 2- “Does anyone have something other than AT&T?  I can’t call anyone!”

Email has to go through your email servers and be backed up, instant messaging has to go through a proxy that logs all conversation, phones (at least for traders) have to keep recordings, etc etc.  One of my better scripts was actually to do a daily dump of chat logs, bundle them per user, and send it as an email to a special email address.  Killing two birds with one stone!   Regulators will seriously come in and say “we want every communication trader X made from this date to this date.”  And you say “okay” or you get a fine.  They also do spot-checks at least once per year.  “Give us everything you traded from this date to this date.”  And you say “okay” or you get a fine.  Incidentally, if you want to see roughly what the code for chat log retention looked like:

#!/usr/bin/perl
use strict;use warnings;
use include;
my $ch='chat_host';
my $cu='chat_user';
my $cp='chat_pass';
my @worries=();
my $to='chatlogs@mycompany.com';
my $today=&include::getdate('YYYY-MM-DD');
my @senders = &include::sql($ch,$cu,$cp,"select distinct(senderid) from message_log where date(modified)='$today'")
  or &freakout($!);
foreach my $sender(@senders)
{
  my @lines=&include::sql($ch,$cu,$cp,"select msg_text from message_log where date(modified)='$today' and senderid='$sender'");
  if(!scalar(@lines))
  {
    push(@worries,"No lines for $sender");
    next;
  }
  &include::send_email($to,join("_",split(" ",$sender)).'@backuphost.mycompany.com','Chat logs for $today',join("\n",@lines))
    or push(@worries,$!);
}
if(scalar(@worries))
{
  &include::send_email('me@mycompany.com','chat_backups@backuphost.mycompany.com','Possible problems with chat backups',join("\n",@worries));
}
sub freakout
{
  my $wtf=shift;
  &include::send_email('me@mycompany.com','chat_backups@backuphost.mycompany.com','Error with chat logs for $today',$wtf);
  exit(1);
}

It’s a  simplified but only because the hard stuff is hidden in my include module I keep around to make my life easier – it was(is) literally this short.  We traded every single day and there was chat (and chatbots) every day, so if there wasn’t anything in the logs there was definitely a problem.  In plain language:

Do some housekeeping (includes, variable setup for hostnames and the like, the date, etc)
Get a list of everyone who sent an IM that day and if there's a problem, send a freakout email with what happened.
For every person in that list, send an email with their name as the "from" address and the chats they sent as the body
  (if there's a problem, add it to the list of worrying things)
If anything is worrying, send an email to that effect to me.

The reason this is so simple is kind of a lucky coincidence.  The chat logs are in one table, the ID is the name of the person, and the text of the log includes the time and who it’s to.  If it didn’t, I’d have to have another line to get the identity of the person – no big deal.  I’d also have to have a temporary array of lines that I’d append the “to”, “time”, and “text” to as one entry, then send the email with the contents of that array instead.

#!/usr/bin/perl
use  warnings;
use strict;
#this should be run as root, otherwise nmap will probably yell at you

my $net=shift || usage();
#the lazy, lazy regex to get the subnet you're working on...
$net=~s/(\d{1,3}\.\d{1,3}\.\d{1,3}\.)\d// || usage();

foreach my $end(0..255)
{
        my $ip  ="$net$end";
        my ($fwd,$rev,$ud,$os) = ("unknown")x4;
        my $nmap  =`nmap -v -O -sT $ip`; #save for later
        my @nmap  =split("\n",$nmap);

        #get forward and reverse DNS
        chomp(my $host =`host $ip`);
        if($host!~m/NXDOMAIN/)
        {
                $fwd=(split(" ",$host))[-1];
                chomp($rev=`host $fwd`);
                $rev=(split(" ",$rev))[-1];
                $rev= "" unless $ip ne $rev; #only display if it doesn't equal the original ip
        }

        $ud = $nmap=~m/Host seems down/?'Down':'Up';
        #get the o/s
        $os=(grep(/Running/,@nmap))[0] || '';
        if($os)
        {
                $os=~s/Running: //;
                $os=substr $os,0,25;
        }
        $fwd=substr $fwd,0,40;
        printf "%-16s%-5s%-28s%-43s%-20s\n",$ip,$ud,$os,$fwd,$rev;
}
sub usage
{
        print "usage: >#!/usr/bin/perl
use warnings;
use strict;
#this should be run as root, otherwise nmap will probably yell at you

my $net=shift || usage();
#the lazy, lazy regex to get the subnet you're working on...
$net=~s/(\d{1,3}\.\d{1,3}\.\d{1,3}\.)\d/$1/ || usage();

foreach my $end(0..255)
{
        my $ip  ="$net$end";
        my ($fwd,$rev,$ud,$os) = ("unknown")x4;
        my $nmap  =`nmap -v -O -sT $ip`; #save for later
        my @nmap  =split("\n",$nmap);

        #get forward and reverse DNS
        chomp(my $host =`host $ip`);
        if($host!~m/NXDOMAIN/)
        {
                $fwd=(split(" ",$host))[-1];
                chomp($rev=`host $fwd`);
                $rev=(split(" ",$rev))[-1];
                $rev= "" unless $ip ne $rev; #only display if it doesn't equal the original ip
        }

        $ud = $nmap=~m/Host seems down/?'Down':'Up';
        #get the o/s
        $os=(grep(/Running/,@nmap))[0] || '';
        if($os)
        {
                $os=~s/Running: //;
                $os=substr $os,0,25;
        }
        $fwd=substr $fwd,0,40;
        printf "%-16s%-5s%-28s%-43s%-20s\n",$ip,$ud,$os,$fwd,$rev;
}
sub usage
{
        print "usage: $0    ex: $0 192.168.0.0\n";
        exit();
}<    ex: >#!/usr/bin/perl
use warnings;
use strict;
#this should be run as root, otherwise nmap will probably yell at you

my $net=shift || usage();
#the lazy, lazy regex to get the subnet you're working on...
$net=~s/(\d{1,3}\.\d{1,3}\.\d{1,3}\.)\d/$1/ || usage();

foreach my $end(0..255)
{
        my $ip  ="$net$end";
        my ($fwd,$rev,$ud,$os) = ("unknown")x4;
        my $nmap  =`nmap -v -O -sT $ip`; #save for later
        my @nmap  =split("\n",$nmap);

        #get forward and reverse DNS
        chomp(my $host =`host $ip`);
        if($host!~m/NXDOMAIN/)
        {
                $fwd=(split(" ",$host))[-1];
                chomp($rev=`host $fwd`);
                $rev=(split(" ",$rev))[-1];
                $rev= "" unless $ip ne $rev; #only display if it doesn't equal the original ip
        }

        $ud = $nmap=~m/Host seems down/?'Down':'Up';
        #get the o/s
        $os=(grep(/Running/,@nmap))[0] || '';
        if($os)
        {
                $os=~s/Running: //;
                $os=substr $os,0,25;
        }
        $fwd=substr $fwd,0,40;
        printf "%-16s%-5s%-28s%-43s%-20s\n",$ip,$ud,$os,$fwd,$rev;
}
sub usage
{
        print "usage: $0    ex: $0 192.168.0.0\n";
        exit();
}< 192.168.0.0\n";
        exit();
}

Example output:

monitor:~ imaginarybillboards$ sudo perl Documents/check_network.pl 192.168.2.0
192.168.2.0   Down                             unknown                                  unknown
192.168.2.1   Up   SonicWALL SonicOS 3.X       firewall.private.blah.com.
192.168.2.2   Down                             switch.private.blah.com.
192.168.2.3   Up   Cisco IOS 12.X              ck-sw0.private.blah.com.
192.168.2.4   Down                             unknown                                  unknown
192.168.2.5   Down                             unknown                                  unknown

And without down hosts (a little more directly useful, perhaps):

monitor:~ imaginarybillboards$ sudo perl Documents/check_network.pl 192.168.2.0 | grep -v Down
192.168.2.102 Up   Apple Mac OS X 10.5.X       monitor.private.blah.com.             192.168.2.105
192.168.2.103 Up   Linux 2.6.X                 cartman.private.blah.com.
192.168.2.104 Up   Linux 2.6.X                 kenny.private.blah.com.
192.168.2.105 Up   Apple Mac OS X 10.5.X       monitor.private.blah.com.
192.168.2.107 Up   Microsoft Windows XP        unknown                                  unknown
192.168.2.108 Up   Apple iPhone OS 1.X|2.X|3   unknown                                  unknown
192.168.2.110 Up   Apple Mac OS X 10.5.X       unknown                                  unknown
192.168.2.112 Up   Apple Mac OS X 10.5.X       unknown                                  unknown

Obviously, I have a bit of work to do with that monitor DNS.  This gives me a decent idea of what's around.  Servers and desktops (and iphones apparently) are all mixed on the same network.

Also, once I've (re-)written this, I put into a cron job so I can keep a running track of what's going on.  Disk space is cheap, and it can't hurt anything.

crontab -l
0 2 * * * /bin/bash -login -c 'perl /Users/chriskaufmann/Documents/check_network.pl 192.168.200.0 > \
    /Users/chriskaufmann/Documents/NetworkReports/`date +\%y-\%m-\%d`'

And then you can just diff them to see when something came onto the network.

Enter ‘watch’.  Here’s how it goes – watch -n<number of seconds to wait before refreshing> <command to do this to>

And here’s my big watch command (which I’ve aliased to watcher).  In this I’m looking at perl and python processes, the amount of free memory and the cpu usage, and any mysql processes going on.

watch -n1 ' ps aux | egrep "(perl|python)" | grep -v grep;uptime;free -m; echo "show processlist" | mysql -uroot | grep -v Sleep'

This shows details on any perl or python processes (without the grep command itself), the load, the memory state, and any non-sleeping mysql processes.  Every second.   I just keep it up in a corner of the monitor.

2. Semi-related – killing a bunch of mysql processes.

Sometimes I’ll fire up a bunch of processes and decide they should go away.  Easy – killall perl works wonders.  But the mysql processes remain – enter another one:

echo "show processlist" | mysql |egrep -v "(Id|processlist)" | awk {'print "kill "$1";"'} | mysql

What it does:  'echo "show proesslist" ' just echoes “show processlist” to standard output.  That is then piped to the input of mysql.

#echo "show processlist" | mysql
Id User Host db Command Time State Info
455093 root localhost NULL Query 0 NULL show processlist

Next, it reverse-greps (as I like to call it) for ‘Id’ because I don’t want to see that.

# echo "show processlist" | mysql |grep -v Id
455095 root localhost NULL Query 0 NULL show processlist

Next, it inputs it into a very short awk program – what this does is split it up by spaces, and set each of those to a $<number> variable.  So we print "kill $1;" to standard out – that’ll be the command we want to send to mysql to kill them all. So we end up with:

# echo "show processlist" | mysql |grep -v Id| awk {'print "kill "$1";"'}
kill 455098;

Finally, pipe that into mysql, like so:

# echo "show processlist" | mysql |grep -v Id| awk {'print "kill "$1";"'} | mysql
ERROR 1094 (HY000) at line 1: Unknown thread id: 455101

What happened there?  Well, since I’ve been killing threads for a good two minutes now while working on this shortcut the only thread left is the “show processlist” thread – which ends as soon as the processlist is shown.  Which makes sense.  So cheat and either add another grep -v to get rid of it or  egrep with a simple regex: egrep -v "(Id|processlist)"

#echo "show processlist" | mysql |egrep -v "(Id|processlist)" | awk {'print "kill "$1";"'} | mysql
#

Yes, both of these could be done in better ways!   But in my defense, part of doing these things is to share them with others and help them be better.  Just like using simple perl is better a lot of the time than really complex code, simple things that are easy to handle are easier to show others and let them modify.  Plus, this is actually how I go about helping someone solve one of these problems so they can do it themselves in the future.

Oh, and now I can just send this link…

Both have a shebang line, and after that import (use) lines.  Perl’s are mostly optional – for sysadmin stuff you’re usually just doing your boilerplate strict and warnings.  Of course, even that is optional.  Technically anyway.  For python, you need to import something to do absolutely anything.   Which is okay – it shows you what is being used.

Then on to the real work.  In perl, you start out with the program.  It’s right there.  If you want to see the logic, just open the file – it’s usually at the top.  Python is the opposite – you have to declare your objects and functions higher up in the file before you can use them.   I couldn’t say you have to declare them before you use them because in practice you’re coding along and think “hey, this should be a function” and zoom down a bit and add it, then go back to the logic.  You’re still doing them before.

So you have your listing of objects and functions somewhere, and the actual program logic somewhere.  But this shows one difference between the two.

Perl cares about doing things.  Python cares about defining things.

Their production may be only average or above average, but everyone in the zone around them is way above average.  That’s because when they find some way to automate the widget approval application process, they get excited and share it.  Suddenly widget applications go from hours a day to seconds (this has actually happened to me) and everyone gets way more widgets out.   Meanwhile he’s been using his newfound time to try to find a way to get a chat room for everyone in the team so they don’t have to yell or call all day.

He’ll automate things to make not just his life easier, but everyone around’s life easier also.

How do you find your enabler?  I think it starts with being nice.  Having too much empathy for everyone else helps too.  Does someone not only order their drink  at the Tastes Burnt with a smile and say please, but pay attention to how they yell them to each other so they don’t have to translate it?  That fellow down the row who strangely requests a garbage can for a strategic spot in the middle?  Depends: did you find yourself (along with him) saving time and reducing hassle by having one in just the right place?

Why would you want to become one?  Well, there’s some drawbacks and some benefits.

Drawbacks:

• Less time to get your stuff done
• You look like a kissup
• You look like you’re trying to involve yourself everwhere
• You’ll get lots and lots of questions

On the plus side:

• It’s genuinely helping people in that tiny way that makes a huge difference
• The office is a more pleasant place for all involved
• Your tools get used
• Your brain gets used
• That stuff you do for everyone else?   You get to do it for you, too.
• You don’t have to learn to say “no”
• You get to do more, and more interesting, things day to day

In short (too late!) – I think there’s someone who acts as the grease that makes any office run smoother – and that it’s important they’re there.

I have a giant (and I mean giant) dataset that I have to load and do some things with the data, then put it somewhere else.  Once I managed to corrupt almost every single table.  So, have to repair them all.

perl  -e 'foreach $a qw{things whosit mrmr}{$cmd=qq{echo "repair table $a" | mysql -uroot mydb};print "$cmd\n";`$cmd`;}'

Silly use for the perls, but – it really works!

my  $count=0;
my $max_inserts = 15000;
my @inserts=();
my $printed_example=0;  #will be 1 when we print an example of the insert;
my $insert_sql = "insert into some_table (col1,col2,col,etc) values ";
while(my @things=$sth->fetchrow_array())
{
    $count++;
    my $bigline="('".join("','",@things)."')";
    push(@inserts,$bigline);
    #flush the @inserts after 1k (or whatever) rows get pushed onto it
    if(scalar(@inserts)>$max_inserts)
    {
         $dbh->do($insert_sql.join(",",@inserts))
            or die "Couldn't insert: $!\n";
        @inserts=();       #empty back out the array.
    }
}
#don't forget to insert any leftovers :)
$dbh->do($insert_sql.join(",",@inserts)) if scalar(@inserts);
@inserts=();

What it’s doing: Getting a bunch of rows from a database (or whatever). While it’s looping through the giant list it gets, it’s pushing part of an insert statement onto an array. Basically, the back half of an insert statement. Once that array gets big enough, it “flushes” it into the database and empties it out again. Finally, after it’s done looping through the giant list, it “flushes” any left over. I find this is faster than preparing and executing one at a time by a factor of 10 to 50. Must do some number crunching soon…

Just a couple of notes here – I don’t like the usual unix-style for some reason.  I’m not going to remember that -d is debug, and that -a <email address> is email address.  Plus, one of the problems with programmers is that the first way we learn to do something is the way we’ll do it forever.  When I started with cgi, we did the full “my $cgi = new CGI; my $thing=$cgi->param(‘thing’)”.  You *can* do  just the standara param(‘thing’) but then you have to use “use CGI qw/:standard/;” which I can never remember.  No big deal either way.

So here goes.  For this one, I have a program that gets a bunch of stuff from a bunch of machines and does some manipulation of the data.  I want to be able to limit the machines it goes to, whether or not debug is on, how many threads, how many concurrent inserts, where it logs, and the date it runs for.  (assuming it gets the list of machines from a DB if not specified).

perl loader.pl host=all_hosts threads=10 inserts=15000 logfile=~/logs/`date +\%y-\%m-\%d` debug=0 date=`date +\%y-\%m-\%d`

And the code that runs:

#!/usr/bin/env  perl
use CGI;
my $cgi = new CGI;
my $date = $cgi->param('date') || `date +\%y-\%m-\%d`; chomp($date); #gives a default, with no newline
my $debug = $cgi->param('debug') || 0;
my $host = $cgi->param('host') || usage();  # you have to pass at least one!
my @hosts = $host eq 'all_hosts' ? all_hosts() : split(',',$host);   #ternary test, gets a list, depending on what's passed
my $threads = $cgi->param('threads') || 10;
my $inserts = $cgi->param('inserts') || 15000;
my $logfile = $cgi->param('logfile') || "~/logs/$date";

#do something
foreach my $hostname(@hosts)
{
  #get some data and insert it, log, etc.
}

sub usage()
{
  print "Usage: perl loader.pl host= [threads=10 inserts=15000 logfile=<logfile> debug=1 date=<YYYY-MM-DD>]\n";
  exit();
}

While somewhat on the subject, that date hack is a great one to remember. A lot of the time you want to be explicit about the date to run for. Alternatively, I have a lot of programs I want to run for yesterday. It’s almost as simple.

`date +\%Y-\%m-\%d --date='1 days ago'`

The above line returns the date in YYYY-MM-DD format for yesterday. That’s it! Put it into a cron job the next morning for yesterday and look cool to those windows guys.