Seibel: As a programmer, do you consider yourself a scientist, an engineer, an artist, or a craftsman?

Norvig: Well, I know when you compare the various titles of books and so on, I always thought the “craft” was the right answer. So I thought art was a little pretentious because the purpose of art is to be beautiful or to have an emotional contact or emotional impact, and I don’t feel like that’s anything that I try to do. Certainly I want programs to be pretty in some ways, and sometimes I feel like I spend too much time doing that. I’ve been in a position where I’ve had the luxury to say, “Gee, I have time to go back and pretty this up a little bit.” And places where I’ve been able to write for a publication, you spend more time doing that than you would if it was just for your own professional growth.

But I don’t think of that as art. I think craft is really the right word for it. You can make a chair, and it’s good looking, but it’s mostly functional—it’s a chair.

Seibel: How do you recognize a great programmer, particularly when you’re hiring? You guys have hired a lot of programmers and you obviously try to hire really good programmers. How do you do it?

Norvig: We still don’t know.

Seibel: Google is somewhat famous for asking puzzle questions in interviews. Do you think that’s a good approach?

Norvig: I don’t think it’s important whether people can solve the puzzles or not. I don’t like the trick puzzle questions. I think it’s important to put them in a technical situation and not just chitchat and get a feeling if they’re a nice guy. Though it is important to have someone that you can get along with. But you really have to see, can they technically do what they said they can do. And there are a lot of different ways to demonstrate that. And many times you can see it from the résumé. I think our best signal is if somebody has worked with one of our employees before and the employee can vouch for them. But we still try to draw it out on-site during the interview. It’s more you want to get a feeling for how this person thinks and how they work together, so do they know the basic ideas? Can they say, “Well, in order to solve this, I need to know A, B, and C,” and they start putting it together. And I think you can demonstrate that while still failing on a puzzle. You can say, “Well, here’s how I attack this puzzle. Well, I first think about this. Then I do that. Then I do that, but geez, here’s this part I don’t quite understand.” For some people that little part clicks and for some it doesn’t. And you can do fine if it doesn’t click as long as you’ve demonstrated the basic competency and fluency in how you think about it. And then you really want to have people write code on the board if you’re interviewing them for a coding job. Because some people have forgotten or didn’t quite know and you could see that pretty quickly.

Seibel: So is that just a negative indicator? If they can’t write reasonable code, that’s a bad sign. But if they don’t stumble over that hurdle, it’s hard to tell whether they’re actually going to write really good code in a larger context.

Norvig: Right. You could tell to a certain degree but at other levels you can’t. And we’ve studied this pretty carefully because we’ve gotten a lot of applications and we look at it at two levels. One, we say, from all the résumés we get, are we interviewing the right set of people? And then, from the interviews we get, are we hiring the right set of people?

Seibel: So how do you measure that? You don’t know about the one you didn’t talk to or didn’t hire.

Norvig: Yeah, so that’s hard. At both levels, you’ve only got half your sample, so it’s this biased problem, but I guess basically what we’re doing is saying, “Of the people that we interviewed and did well, what did their résumé look like,” and try to find more of those. Is having so many years of experience important? Is working on an open-source programming project important? How does that compare to winning a programming contest?

Seibel: Do you really take all these things and shove them into a database?

Norvig: Yes, we do, and when we’re doing the hiring, we have these scores that come up that say, “The résumé predictor says such and such, and the interview predictor says such and such.” We don’t take them as gospel, but they’re just another piece of input along with all the other feedback we have.

Seibel: Do the people who are doing the interviews have those numbers beforehand?

Norvig: No, we only see that when they’re in the hiring committee, once we’ve gathered all the feedback. One of the interesting things we found, when trying to predict how well somebody we’ve hired is going to perform when we evaluate them a year or two later, is one of the best indicators of success within the company was getting the worst possible score on one of your interviews. We rank people from one to four, and if you got a one on one of your interviews, that was a really good indicator of success.

Seibel: But you had to do well enough on something else that you actually got hired?

Norvig: Right, so that’s the thing. Ninety-nine percent of the people who got a one in one of their interviews we didn’t hire. But the rest of them, in order for us to hire them somebody else had to be so passionate that they pounded on the table and said, “I have to hire this person because I see something in him that’s so great, and this guy who thought he was no good is wrong, and I’ve got to stand up for him and put my reputation on the line.”

Seibel: So you’re surrounded by top-notch programmers here at Google. Given how pervasive computers and software are in our society, do you think everybody needs to understand a bit about programming just to get along in or understand the world they live in?

Norvig: You probably want an educated person to understand how software is made to the same degree they understand how a car is made. The other interesting thing is, how much does an informed citizen have to be a programmer? Certainly the average person now can do word processing and many of them can do spreadsheets, and so if you’re a little bit experienced with spreadsheets, you’re starting to be a programmer.

Lots of attempts at end-user programming, and programming for everyone, haven’t been very successful. I don’t know how easy it is. Is there a way of thinking that people have that we’ve gotten all the people that it’s easy to teach, and the other ones are going to be really hard, or is it just that we’ve missed the model and there’s some simple model in place of many people that could influence programming if we created it?

Seibel: A lot of people that I have talked to for this book, and elsewhere, got into computers both because they just enjoyed it and because they felt like it would change the world. Some of the folks I talked to for this book did that in the past and now are depressed by how little they feel the world has changed as a result. How do you feel about that?

Norvig: Well, I’m in the right place. We have hundreds of millions of users and we can make a difference for them, and we can launch new services for them quickly. I think that’s great. I can’t imagine anything else I could be doing to have that level of impact.