Seibel: We talked a bit about some of the really hideous bugs you had to track down like that thing with GDB. But let’s talk a little bit more about debugging. For starters, what’s your preferred debugging tool? Print statements? Symbolic debuggers? Formal proofs of correctness?

Zawinski: That’s changed over the years a lot. When I was using the Lisp machines it was all about running the program and stopping it and exploring the data—there was an inspector tool that let you browse through memory and I changed it so basically the Lisp listener became an inspector. So anytime it printed out an object there was a context menu on it so you could click on this thing here and have that value returned. Just to make it easier to follow around chains of objects and that sort of thing. So early on that was how I thought about things. Getting down in the middle of the code and chasing it around and experimenting.

Then when I started writing C and using GDB inside of Emacs I kind of tried to keep up that same way of doing things. That was the model we built Energize around. And that just never seemed like it worked very well. And as time went by I gradually stopped even trying to use tools like that and just stick in print statements and run the thing again. Repeat until done. Especially when you get to more and more primitive environments like JavaScript and Perl and stuff like that, it’s the only choice you have—there aren’t any debuggers.

People these days seem confused about the notion of what a debugger is. “Oh, why would you need that? What does it do—put print statements in for you? I don’t understand. What are these strange words you use?” Mostly these days it’s print statements.

Seibel: How much of that was due to the differences between Lisp and C, as opposed to the tools—one difference is that in Lisp you can test small parts—you can call a small function you’re not sure is working right and then put a break in the middle of it and then inspect what’s going on. Whereas C it’s like, run the whole program in all of its complex glory and put a break point somewhere.

Zawinski: Lisp-like languages lend themselves more to that than C. Perl and Python and languages like that have a little more of the Lisp nature in that way but I still don’t see people doing it that way very much.

Seibel: But GDB does give you the ability to inspect stuff. What about it makes it not usable for you?

Zawinski: I always found it unpleasant. Part of it is just intrinsic to being C. Poking around in an array and now I’m looking at a bunch of numbers and now I have to go in there and cast the thing to whatever it really is. It just never managed that right, the way a better language would.

Seibel: Whereas in Lisp, if you’re looking at a Lisp array, they’ll just be printed as those things because it knows what they are.

Zawinski: Exactly. It always just seemed in GDB like bouncing up and down, the stack things would just get messed up. You’d go up the stack and things have changed out from under you and often that’s because GDB is malfunctioning in some way. Or, oh well, it was expecting this register to be here and you’re in a different stack frame, so that doesn’t count anymore.

It just always felt like I couldn’t really trust what the debugger was actually telling me. It would print something and, look, there’s a number. Is that true or not? I don’t know. And a lot of times you’d end up with no debug info. So you’re in a stack frame and it looks like it has no arguments and then I’d spend ten minutes trying to remember the register that argument zero goes in is. Then give up, relink and put in a print statement.

It seemed like as time went by the debugging facilities just kept getting worse and worse. But on the other hand now people are finally realizing that manual memory allocation is not the way to go; it kind of doesn’t matter as much any more because the sorts of really complicated bugs where you’d have to dig deep into data structures don’t really happen as often because those, often, in C anyway, were memory-corruption issues.

Seibel: Do you use assertions or other more or less formal ways of documenting or actually checking invariants?

Zawinski: We went back and forth about what to do about assertions in the Netscape code base. Obviously putting in assert statements is always a good idea for debugging and like you said, for documentation purposes. It expressed the intent. We did that a lot.

But then the question is, well, what happens when the assertion fails and you’re in production code; what do you do? And we sort of decided that what you do is return zero and hope it keeps going. Because having the browser go down is really bad—it’s worse than returning to the idle loop and maybe having leaked a bunch of memory or something. That’s going to upset people less than the alternative.

A lot of programmers have the instinct of, “You’ve got to present the error message!” No you don’t. No one cares about that. That sort of stuff is a lot easier to manage in languages like Java that actually have an exception system. Where, at the top loop of your idle state, you just catch everything and you’re done. No need to bother the user with telling them that some value was zero.

Seibel: Did you ever just step through a program—either to debug it or, as some people recommend, to just step through it once you’ve written it as a way of checking it?

Zawinski: No, not really. I only really do stepping when I’m debugging something. I guess sometimes to make sure I wrote it right. But not that often.

Seibel: So how do you go about debugging?

Zawinski: I just eyeball the code first. Read through it until I think, well, this can’t happen because that’s going on right there. And then I put in something to try and resolve that contradiction. Or if I read it and it looks fine then I’ll stop in the middle or something and see where it is. It depends. It’s hard to generalize about that.

Seibel: As far as the assertions—how formally do you think? Some people just use ad hoc assertions—here’s something that I think should be true here. And some people think very formally—functions have preconditions and postconditions and there are global invariants. Where are you on that scale?

Zawinski: I definitely don’t think about things in a mathematically provable way. I’m definitely more ad hoc than that. You know, it’s always helpful when you’ve got inputs to a function to at least have an idea in your head what their bounds are. Can this be an empty string? That sort of thing.

Seibel: Related to debugging, there’s testing. At Netscape, did you guys have a separate QA group or did you guys test everything yourselves?

Zawinski: We did both. We were all running it all the time, which is always your best front-line QA. Then we had a QA group and they had formal tests they went through. And every time there was a new release they’d go down the list and try this thing. Go to this page, click on this. You should see this. Or you shouldn’t see this.

Seibel: What about developer-level tests like unit tests?

Zawinski: Nah. We never did any of that. I did occasionally for some things. The date parser for mail headers had a gigantic set of test cases. Back then, at least, no one really paid a whole lot of attention to the standards. So you got all kinds of crap in the headers. And whatever you’re throwing at us, people are going to be annoyed if their mail sorts wrong. So I collected a whole bunch of examples online and just made stuff up and had this giant list of crappily formatted dates and the number I thought that should turn into. And every time I’d change the code I’d run through the tests and some of them would flip. Well, do I agree with that or not?