Butler Lampson’s hope that millions of ordinary people would write “non-trivial programs” and thus become poets of logic has proved elusive. From the sixties onwards, numerous technologists have promised that their new programming languages would make programmers redundant, that “managers [could] now do their own programming; engineers [could] now do their own programming.”²⁵ Advertisements touted the magical abilities of “automatic programming systems.” But Knuth’s “Software is hard” dictum still remains true, and business users have found that getting custom software out of IT departments requires large budgets and lots of patience. This is because programmers — at their best — try to build software out of elegant code that is modular, secure, and legible, which takes time and money. Instead of waiting, mere mortals often hack something together in the programs they already have available on their machines. This is why, according to the economics blogger James Kwak, “Microsoft Excel is one of the greatest, most powerful, most important software applications of all time.”²⁶ Much of the planet’s business data is stored in Excel, and its intuitive interface allows non-programmers access to some very powerful capabilities. Executives and marketers and secretaries write formulae and macros to extract information when they need it, and are therefore able to take action in a timely fashion. The trouble is that in Excel there

is no way to trace where your data come from, there’s no audit trail (so you can overtype numbers and not know it), and there’s no easy way to test spreadsheets … The biggest problem is that anyone can create Excel spreadsheets — badly. Because it’s so easy to use, the creation of even important spreadsheets is not restricted to people who understand programming and do it in a methodical, well-documented way.²⁷

Sloppy Excel-wrangling can lead to some very bad decisions, as in the “London Whale” trading disaster of 2012, which caused the financial services firm JPMorgan Chase a loss of approximately six billion dollars; the company’s internal investigation listed as one of the contributing factors a financial modeling process which required cutting and pasting data through a series of spreadsheets. One of these spreadsheets contained a formula dividing by the sum of some numbers instead of their average.²⁸

The day that millions will dash off beautiful programs — as easily as with a pencil — still remains distant. The “lovely gems and brilliant coups” of coding remain hidden and largely incomprehensible to outsiders. But the beauty that programmers pursue leads to their own happiness, and — not incidentally — to the robustness of the systems they create, so the aesthetics of code impact your life more than you know.

For example, one of the problems that have always plagued programmers is the “maintenance of state.” Suppose you have a hospital that sends out invoices for services provided, accepts payments, and also sends out reminders for overdue payments. On Tuesday evening, Ted creates an invoice for a patient, but then leaves the office for an early dinner; there is now an “Invoice” object in the system. This object has its “InvoiceNumber” field set to 56847, and its “Status” field set to “Created.” All of these current settings together constitute this invoice’s “state.” The next morning, Ted comes in and adds a couple of line items to this invoice. Those inserted line items and a new “Status” setting of “Edited” along with all the other data fields are now the invoice’s state. After a coffee break, Ted deletes the second line item and adds two more. He has changed the invoice’s state again. Notice that we’ve already lost some information — from now on, we can’t ever work out that Ted once inserted and deleted a line item. If you wanted to track historical changes to the invoice, you would have to build a whole complex system to store various versions.

Things get even more complicated in our brave new world of networked systems. Ted and his colleagues can’t keep up with the work, so an offshored staff is hired to help, and the invoice records are now stored on a central server in Idaho. On Thursday afternoon, Ted begins to add more line items to invoice 56847, but then is called away by a supervisor. Now Ramesh in Hyderabad signs on and begins to work on the same invoice. How should the program deal with this? Should it allow Ramesh to make changes to invoice 56847? But maybe he’ll put in duplicate line items that Ted has already begun working on. He may overwrite information — change the “Status” field to “Sent”—and thereby introduce inconsistencies into the system. You could lock the entire invoice record for 56847 on a first come, first served basis, and tell Ramesh he can’t access this invoice because someone else is editing it. But what if Ted decides to go to lunch, leaving 56847 open on his terminal? Do you maintain the lock for two hours?

Guarding against inconsistencies, deadlocks of resources by multiple users, and information loss has traditionally required reams of extremely complex code. If you’ve ever had a program or a website lose or mangle your data, there’s a good likelihood that object state was mismanaged somewhere in the code. A blogger named Jonathan Oliver describes working on a large system:

It was crazy — crazy big, crazy hard to debug, and crazy hard to figure out what was happening through the rat’s nest of dependencies. And this wasn’t even legacy code — we were in the middle of the project. Crazy. We were fighting an uphill battle and in a very real danger of losing despite us being a bunch of really smart guys.²⁹

The solution that Oliver finally came to was event sourcing. With this technique, you never store the state of an object, only events that have happened to the object. So when Ted first creates invoice 56847 and leaves the office, what the program sends to CentralServer in Idaho are the events “InvoiceCreated” (which contains the new invoice number) and “InvoiceStatusChanged” (which contains the new status). When Ted comes back the next morning and wants to continue working on the invoice, the system will retrieve the events related to this invoice from CentralServer and do something like:

Invoice newInvoice = new Invoice();

foreach(singleEvent in listOfEventsFromCentralServer)

{

newInvoice.Replay(singleEvent);

}

That is, you reconstitute the state of an object by creating a new object and then “replaying” events over it. Ted now has the most current version of invoice 56847, conjured up through a kind of temporally shifted rerun of events that have already happened. In this new system, history is never lost; when Ted adds a line item, a “LineItemAdded” event will be generated, and when he deletes one, a “LineItemDeleted” event will be stored. If, at some point in the future, you wanted to know what the invoice looked like on Wednesday morning, you would just fire off your “Replay” routine and tell it to cease replaying events once it got past 9 a.m. on Wednesday morning. You can stop locking resources: because events can be generated at a very fine granular level, it becomes much easier to write code that will cause CentralServer to reject events that would introduce inconsistencies, to resolve conflicts, and — if necessary — pop up messages on Ted’s and Ramesh’s screens. Events are typically small objects, inexpensive to transfer over the wire and store, and server space grows cheaper every day, so you don’t incur any substantial added costs by creating all these events.

Oliver writes that when he discovered event sourcing

it was as if a light went on. I could literally see how adoption of event sourcing could shed a massive amount of incidental and technical complexity from my project … Fast forward to today. [I] now have a number of systems in production with several more that are only weeks away and I literally could not be happier. I have significantly more confidence in my software than I had in the past. The code is dramatically cleaner and infinitely more explicit than it would have been otherwise. But that’s only the starting point. Our ability to expand, adapt, and scale — to be agile from a business perspective — is infinitely greater than it ever has been, even with each application being significantly larger and each associated domain exponentially more complex than before — all with a smaller team.³⁰