Other people I eventually figured out what motivated them. One guy was really good at tinkering and getting a prototype working. He wrote sysadmin Perl. He could wire stuff together, write shell scripts, and write really bad Perl and really bad C, but kind of get it working. Then we would be like, “Holy crap, you researched all this stuff, you got all these components talking to each other?”

We were setting up a voice bridge to LiveJournal so you record something and post it to LiveJournal. There were just so many moving parts involved. I thought it was painful as hell. He loved it. He figured it all out and got it working. Then we just rewrote it all. And we figured out that was the way it worked with him. He figured out the interface and we would fix it all up. Once I figured out that was his role, we got along great.

Seibel: So you’ve hired for your own company, and I assume you’ve been involved in hiring at Google. How do you recognize a great programmer?

Fitzpatrick: I often look for people that have done, like, a lot of stuff on their own that wasn’t asked of them. Not just their school project or just what their previous employer had them do. Somebody who was passionate about something and had some side project. How did they maintain it and how serious did they get with it? Or do they do a lot of quick hacks and abandon them?

Seibel: Do you have favorite interview questions?

Fitzpatrick: One of the ones I’ve given a few times because it was on my AP programming test is given two decimal numbers as strings of arbitrary length, multiply them. There are a lot of different ways that they could do it. If they’re really good at math—like I’m not—they can find some clever ways to do it really efficiently. Worst case, they can make a class that does just addition repeatedly.

I tell them from the beginning, “Don’t stress out. You don’t have to do it efficiently. Just get it done somehow.” Some people stress out and have no clue where to begin. That’s kind of a bad sign. The worst case, you just implement the algorithm you do in grade school.

I actually wrote a program in grade school to do my long division and multiplication and show the work. Including all the steps and where to cross out. So then we would get these problems, like ten per page or something, and I would type it into the computer and then just reproduce the problems in scribbles. I did the same thing in chemistry to find the orbitals of electrons. But the thing I find is by writing a program to cheat, you learn because you have to learn it really in depth to write that program.

Seibel: Do you think that would work for anyone? Instead of teaching kids long division, should we teach them how to program and then say, “OK, now your task is to write a program to do this long-division procedure”? By the time they’ve actually written that program, they’ll understand division. Or does that only work if you have some natural inclination that way?

Fitzpatrick: It worked for me. A lot of times, someone could teach you something and you’re like, “Yeah, yeah, sure. I understand.” You delude yourself but once you actually have to get down and do it at a real level and understand all the corner cases, it forces you to actually learn the thing. But I don’t know if that would work for everyone.

Seibel: Google has a bit of a reputation, as Microsoft also does, of interviewers using puzzle questions.

Fitzpatrick: I think those are kind of banned. Or strongly discouraged. Maybe some people still do them, but I think, in general, they’re discouraged.

Seibel: What did they ask you in your interview?

Fitzpatrick: One question was, imagine you have a bunch of computers on a switch and they turn on the whole rack; come up with an algorithm so every machine on the rack knows the status of all the other ones about whether they’re on or off. So basically a presence thing. That was pretty much the constraint. Basically, they described Ethernet: you could send a broadcast to everyone, or you could send it to a specific MAC address. So I just kind of walked through all the different strategies to minimize bandwidth and to minimize latency discovering when something’s dead. That was a fun one.

Seibel: What’s the worst bug you ever had to track down?

Fitzpatrick: I try not to remember them. I hate it when it’s something where your assumptions are so far off. The other day—this is definitely not an example of the worst one ever—I spent 90 minutes debugging something because I was writing to one output file and reading another file named the same thing but with one path component missing. I kept rerunning this huge MapReduce and seeing the output and putting it in GDB and stepping through it. “What the fuck? It’s not changing!” Finally I looked at the paths and I was like, “Holy crap.” I don’t know why I spent 90 minutes on it; I was so obsessed that I didn’t step back and check, is my command line correct?

There’s a lot of that. We always had some good stuff with Perl like the $_ isn’t lexically scoped. So if you fuck with $_ in a sort, you can mess with somebody else’s far away. So we had this bug that took us forever and we had a bunch of corruption going on. We finally figured that out. Then I audited all our code we had a new policy of “never do this.”

Seibel: What are your debugging tools? Debuggers? Printlns? Something else?

Fitzpatrick: Println if I’m in an environment where I can do that. Debugger, if I’m in an environment that has good debuggers. GDB is really well maintained at Google and is kind of irreplaceable when you need it. I try not to need it too often. I’m not that great at it, but I can look around and kind of figure things out generally. If I have to go in there, I generally can find my way out. I love strace. Strace, I don’t think I could live without. If I don’t know what some program is doing, or what my program is doing, I run it under strace and see exactly what’s happening. If I could only have one tool, it would probably be that. All the Valgrind tools, Callgrind and all that, those are good.

But a lot of times lately, if there’s something weird going on, I’m like, “OK, that function is too big; let’s break that up into smaller parts and unit-test each one of them separately to figure out where my assumptions are wrong, rather than just sticking in random printlns.”

Then maybe in the process of refactoring, I have to think about the code more, and then it becomes obvious. I could, at that point, go back to the big, ugly state where it was one big function and fix it but I’m already halfway there; I might as well continue making it simpler for the next maintainer.

Seibel: How do you use invariants in your code? Some people throw in ad hoc asserts and some people put in invariants at every step so they can prove formal properties of their programs, and there’s a big range in the middle.

Fitzpatrick: I don’t go all the way to formal. My basic rule is, if it could possibly come from the end user, it’s not a run-time crash. But if it is my code to my code, I crash it as hard as possible—fail as early as possible.

I try to think mostly in terms of preconditions, and checking things in the constructor and the beginning of a function. Debug checks, if possible, so it compiles away. There are probably a lot of schools of thought and I’m probably not educated about what the proper way to do it is. There are languages where all this stuff is actually a formal part of the language. Pretty much all the languages I write in, it’s up to you.

Seibel: You wrote once that optimization is your favorite part of programming. Is that still true?

Fitzpatrick: Optimization is fun because it’s not necessary. If you’re doing that, you’ve got your thing working and nothing else is more important and you’re either saving money or doing it because it’s like a Perl golf contest—how short can I make this or how much faster. We would identify hotspots in LiveJournal, and I would send out some contests. “Here’s some code. Here’s the benchmark. Make it fast.” I sent our load balancer’s header parsing. We were all writing crazy regexps that didn’t backtrack and tried to capture things with the most efficient capture groups. And we were all competing, getting faster and faster and faster. Then one guy comes over the next day. He had written it all in C++ with XS, and so he was like, “I win.”