How To Make a Mess of Things

The other day, I was trying to buy a new phone and sign up for a mobile service plan online. I added a phone to my cart, and then when I selected a payment method, the page disappeared, and I found myself looking at a different page full of phones I didn’t want.

“OK,” I said. “Let’s try that again.”

Another try, and I got the same result. So I tried again without specifying the payment option. After I added the first phone to my cart, the site asked if I would like to purchase a second phone. Yes. I put the second phone in the cart, and the first one disappeared. Again, and again, and again. There was just no way to get two phones in the cart at the same time. And the service plan I had selected kept changing, even though I wasn’t changing it.

How can a major corporation run a website that’s THIS bad?

Well, I happen to be a programmer who has built a number of sites like this, so I have some idea of what’s going on. I know from someone who worked at this company that they outsourced development of the site to a major contractor, who brought in a huge team of programmers to build it.

This is the root of the problem. Try getting hundreds of people to collaborate seamlessly on ANY project. Try getting hundreds of people to build a coherent product of ANY kind.

Imagine you want to write a 500-page novel, and you need to do it quick, because every day it’s not out there selling on Amazon and Barnes and Noble, you’re losing money. (This is how corporate managers think.) You need to hire someone right away. But it would take a writer months to crank out 500 pages. Maybe weeks, if you keep the pressure on. But still, that’s just too long.

So how about we hire 500 writers and have them each write a page? Brilliant! We’ll have our novel by noon!

We just have to give the writers a few parameters. We’ll say it’s a Romance novel, because Romance sells well, and the formula is pretty straightforward. You have a woman who is lonely or unfulfilled. You have a new guy appear on the scene who is single. They meet under less-than-ideal circumstances, there’s lots of tension between them, and the reader knows they’re meant for each other, but they can’t be together because…

I don’t know, just throw some obstacles in their way. Maybe 400 pages worth of obstacles, but make sure we get to know and like the characters while their being obstacled to death. Then — THEN! — though fate and luck and force of will, they get together, and it’s oh so satisfying! Your readers are delighted, and they can’t wait for your next book.

We’ll put our 500 writers in the lunchroom, and we’ll assign a manager to walk around and make sure they’re actually writing and not wasting company time on Facebook. We’ll give them these parameters:

Heroine: Rebecca, a passionate but repressed red-headed librarian, age 30.

Hero: Maxibillion, a wealthy maverick technologist from the Scottish Highlands with a devil-may-care attitude and nothing on under his kilt, age 34.

Location: Portland, OR

We’ll call the book Maximum Satisfaction.

At 10:00 AM the writers are in the lunchroom and they have their instructions. OK, everyone, pens to paper, and… GO!

At noon, the proctor collects the papers, and voilà, we’re done!

Except we’re not, because the story doesn’t make any sense. Rebecca is sensitive, and then she’s not. She met the blonde-haired Max in a coffee shop on page 10, and when she meets him at the pool on page 30, she’s smitten by “the tall dark-haired man she’d never seen before.” The whole book is riddled with these inconsistencies. There are no transitions, and the tone changes from page to page. Chronologically, it’s a mess. You might as well throw all the pages up in the air and read them in whatever order they land.

Those damn writers! Why do they screw everything up?

We have to call them all back in, and we’re going to make them read the entire novel and identify all of the problems and inconsistencies. Then we’re going to outline the entire plot, scene by scene. We’ll workshop it. Then we’ll assign each writer a page and have them get to work. If the guy writing page 400 needs a bearskin rug in the living room, he’s going to have to coordinate with the guy writing page 20, where the living room is first described.

Each writer will have to consult with each of the other 499 writers to ensure that his page is consistent in fact, spirit, character, and tone with what the others have written. This will require a great number of meetings. We’ll have to cater lunches and pay overtime.

18 months into our project, we see Lonely Red sitting at the top of the bestseller lists. The passionate librarian has snagged her kilted billionaire to the delight of readers everywhere.

But this is not our book! According to the three dozen managers who are now overseeing the project, our book won’t be ready for at least another 18 months. And this Lonely Red, this bestseller that was OUR original utterly unique once-in-a-lifetime idea, was cranked out by some lone self-published author in just nine months! What the hell?

As absurd as this story sounds, this is how many large organizations develop software. They look at developers as interchangeable commodities, ten of whom should reasonably be able to produce ten times as much software as a single programmer working alone.

When they have a big project, they put together a big team, and then find that communication and management are more difficult and time-consuming than the original problem their programmers set out to solve. Though Fred Brooks exposed these issues over 40 years ago in The Mythical Man Month, some organizations still don’t get it.

In the 2000’s, many companies doubled down on their big-team philosophy by choosing the Java and C# programming languages. The hallmark of both those languages is their restrictiveness. They were designed primarily to prevent mediocre programmers from doing bad things on big projects, and they came with Integrated Development Environments (IDEs) to help enforce consistency.

If our 500 writers had IDEs like the ones the Java/C# developers use, the guy writing page 400 would not be able to say Rebecca’s raincoat is green, because the guy who wrote page 20 said it was black. The IDE knows this, and will put a squiggly red line under the word “green” when the page-400 writer types it.

So now, at least, our book is consistent, and the manager who made everyone use the IDE will get a raise for having solved an entire class of problems in one fell swoop. A $200 IDE license for each of our 500 writers cost the company $100,000 up front, but will save $200,000 in editing and proofreading over the life of the project. Which is now at least 36 months from completion. And even though the book is still going to suck compared to Lonely Red, it WILL be error free!

In his 2010 book, The Design of Design, Brooks notes that virtually every major creative and innovative work of the last century has been the product of a single mind, with a few rare exceptions coming from two well-matched minds. (Brooks lists Apple, Inc. as an example of the latter, with Steve Jobs and Steve Wozniak being an extraordinarily fortuitous match.)

In the software world of the past decade, most people working outside of Large Dysfunctional Organizations (LDOs) have recognized that smaller teams of talented people will vastly outperform large teams that have huge communications and organizational overhead. The non-LDOs have also recognized that the creative and innovative aspects of programming flourish under less restrictive programming languages, like Ruby, Python, Go, and Clojure. Instead of the Java/C# design principle of “don’t trust the programmer,” these languages were designed to let programmers express themselves freely, clearly, and simply.

Is it any wonder then, that for the past ten years or so the startups have been running circles around the LDOs?

I came away from that website the other day thinking about all this. And about how, in the past, I’ve worked with teams of 3–5 developers to produce fully functional, high-traffic retail sites that work quite well.

In the end, I wound up going with a different cell carrier, because if a company can’t even take my money when I’m trying to hand it to them, what kind of service can I expect them to provide?

By the way, LDO Publishing’s Maximum Satisfaction will be published under the pen name Duke Newcomb sometime in 2016 2017 2018 the future. While you’re waiting for that, enjoy the next seven books in the Lonely Red series.

A Cool App with Real World Value

I don’t often write about my day job here, but I have to say, it’s always nice to see a project you worked on having an impact in the real world. Several years ago, an entrepreneur named Phil Reitenour had a scary run-in with an enraged driver. That gave him an idea. What if someone built a mobile app that could stream live audio and video of an unfolding emergency directly to a security monitoring station? Better yet, what if it could stream that data to the nearest police or public safety office? What if it could even show security personnel your movements along a map as the incident unfolded?

Phil founded a company called EmergenSee and built a prototype of an app that does just that. The prototype worked, but that initial version wouldn’t handle a high number of concurrent users, which would be a problem in an event like a bombing or a riot. So EmergenSee hired the company I used to work for, and we rewrote the app so it could scale up to heavy user loads.

Now it’s used by organizations all over the US. Universities, for example, can define an area they want to monitor, simply by bringing up a map in the app and drawing an outline around the campus. When someone on campus opens the app and starts recording, the video, audio, and location data appears immediately on the computer monitors of campus security. This can be really useful for a woman walking alone late at night. The campus police instantly see and hear what’s going on, and they know exactly where she is. This is what they see:

emergensee

The app has some other cool features. Organizations can push out alerts to subscribers to warn of dangerous situations. Users can set a timer that will automatically alert security, and tell them where they are, if they don’t arrive home within an expected time frame.

This is one of the cooler and more useful projects I’ve worked on in my 18 years as a software developer, and it looks like they’re finally starting to get some attention. Verizon recently awarded them $500,000 to help develop and market the product in their 2015 Powerful Answers Award contest.

If you’re interested, you can get a free version in the App Store or on Google Play. The free version lets you specify personal emergency contacts, and when you begin an “incident,” your contacts will get an alert and be able to watch the incident in real time, with a view that looks just like the one above.

 

Writing and Programming

I’ve been a software developer since 1998. I didn’t actually like computers until I started working at Amazon.com back in the days when they sold only books. I got a job in their customer service department after a brief stint of teaching in the Seattle public schools. At the time, Amazon’s customer support was delivered primarily through email. If you had a question, you’d write to orders@amazon.com, and one of the customer reps in Seattle would write back.

Everyone in customer service worked at a Unix terminal, using Emacs as a mail client, and a set of command line tools with names like orderstat and customerstat to track down problems.

Emacs is a text editor used by software developers to write code. It does not have layout or formatting features like a word processor, because you don’t need those features when you’re writing code. It does have lots of advanced features, like a built-in email client, web browser, web server, code evaluator, debugger, psychiatrist, and more. It’s difficult to learn, because all those thousands of features are hidden behind an interface that looks like this:

Emacs

Put that in front of someone who’s never seen it before and tell them to get to work. They’ll have no idea where to begin.

The first week of training for customer service reps focused on how to use Emacs and the Unix command line. Learning Unix was an epiphany. The entire system was built around a few simple principles that applied everywhere. If you ever got lost, you could go back to those principles and reason your way toward where you wanted to be.

Around this same time, I was having trouble getting my home computer to work with a new printer. I realized how much I depended on the computer for writing, email and other communication, and I wanted to know how it worked. So I bought a book on C programming and started making little programs.

Attempting to understand your computer by buying a learn-to-program book is incredibly naïve. It’s like attempting to understand the a foreign culture by reading one of its primary school grammar books. But I started to see in the C language the same principles that pervaded Unix. This was no coincidence. Unix was written in C, by the creators of C at Bell Labs in the early 1970s.

I became fascinated by programming, and for the next several years, I’d stay up until midnight and get up at 5:30 AM just to tinker and learn. By 1999, I was doing it professionally. It was the dot-com boom, and programmers were in such demand that even a self-taught developer with no real-world experience could get work.

While I spent most of my time coding, I never lost the love of writing. But programming was my living, and writing went to the back burner.

I recently read Vikram Chandra’s Geek Sublime. Chandra is another software developer and writer, and much of his book is about where the two practices converge and overlap. Some reviewers criticized the book for not wrapping everything up neatly, like a freshman compare-and-contrast essay. But the book is quite rich, and it has plenty to offer even without a neat conclusion. I’ll review it another day.

Like Chandra, I’ve found a number of similarities between writing code and writing prose, and the practice of one can help develop skill in the other. Both require immense concentration and extended periods of solitary, focused work. Both require you to keep track of a huge number of concepts, interactions, side-effects and timelines. You have to manage these in your head as you write, and you have to understand how a little change in one part of the work will require changes to some other distant part that you haven’t even looked at in three weeks.

In programming, the concepts are data structures, algorithms, requirements and business rules. The interactions are functions, methods and calls to external systems. Side effects occur when concepts or interactions change. You have to know how all the pieces fit together to anticipate the side effects of any change. Timelines in software generally relate to whether certain actions have occurred or conditions have been met. If the program has all the input it needs and can communicate with external systems, it can proceed. If not, it may need to ask the user for more information, or it may need to put some work aside and proceed with other tasks.

In writing, the concepts are character, setting, theme and tone (to name a few). The interactions are plot, confrontation and dialog (to name a few). All events and all interactions have side-effects, and you have to manage them in ways that make sense. For example, if character A gets into a fight with character B, these characters’ attitudes toward each other will be different after the fight. There may be more tension between them, or there may be less.

Timelines are also important. Imagine a story about adultery where the wife knows from the beginning that her husband is having an affair. Now imagine a story where she learns of the affair after it’s been going on for three years. They involve the same characters in the same setting, but they are two completely different stories.

When you’re writing a book or a large software program, you’re always making little changes to some part of the work and keeping mental notes about how these will require you to change other parts. You don’t write everything in order. You write the difficult and essential pieces when the inspiration hits, and you slog through the more tedious parts when you have the patience and the time.

Good software developers write their code specifically to be read by other developers, because any program that is actually useful will have a lifetime of several years. During that time, other developers will have to adapt it to new systems, fix bugs and add features. While code in general tells the computer what to do, well-written code describes to developers the purpose of the program and of each of its pieces.

Well-written code is not simply a set of instructions. It is a written communication from a developer in the past to a developer in the future that says in the clearest possible terms, “This is what’s going on here.”

A well-written story is essentially the same thing. You may be communicating things that are much more subtle and complex, such as emotional states, but the point of your words is to communicate to some reader you have never met “This is what’s going on here.”

Much of the mental work for both writing and programming happens away from the keyboard. One well-known developer calls it hammock-driven development, which is nicely summed up in this little graphic. You do your thinking away from the computer, where your mind is free to wander and is not pressured to produce. You’re not staring at a little box that you need to fill with code or sentences. You’re outside the box, where you can try on ideas and cast them off at will.

There are two huge qualitative differences between writing software and writing prose (fiction or non): who you are working with, and who you are working for.

Professional developers typically write for companies. They work with managers to create a product whose value is measured in dollars. Sometimes the dollars come from selling the product, but more often they come from making essential processes more efficient and less error-prone.

Writers work for readers. They work with editors to create a product that shares meaning and experience through mental and emotional engagement.

Readers and editors understand that writing is a creative process; companies and managers do not. Creativity happens beneath the surface of activity, or often in the absence of activity. The reader and the editor imagine that the writer is composing or reminiscing as he stares off into space, while the manger thinks the programmer is slacking when he’s not at the keyboard typing.

It’s been this way for a long time. The Parable of the Two Programmers, written back in 1985, sums it up pretty well. It’s a short read, but if you decide to skip it, here’s the summary:

Two companies need to write the same program. At one company, a team goes right to work. At the other company, a developer spends many days away from his computer thinking about what the program needs to do. After much quiet, internal consideration and analysis, the lone programmer distills the problem to its essence, and writes a short, clear program that solves it.

At the other company, the team never takes the time to distill the problem. They write a large, complex, buggy program that mostly does what the company needs.

Then the managers review the work at both companies. The team that implemented the complex solution with unreadable code is told they did a good job in solving a problem that, from the looks of the code, was very complex. The lone developer at the other company is told that, judging from the looks of his short, clear, concise code, the problem he solved must have been quite simple. His manager says it should not have taken so long to solve such a simple problem, and lets him go.

What the managers don’t get is that the real work of writing code (and writing prose) is distillation. It is the act of taking something too big, too complex, and too messy to express in writing and boiling it down to something that is clear and accessible. The programmer reduces it to a set of instructions that he knows will flow through the machine’s central processor in such a way as to produce the results he wants. The writer reduces it to a set of words that he hopes will reproduce in the reader’s mind something close to what he has experienced or imagined in his own mind.

I have always thought that this was the whole point of Hemingway’s The Old Man and the Sea. The old man goes out in a little boat by himself and hooks a giant fish. After a mighty struggle, he hauls it in and ties it to the side of the boat. While he’s adrift at sea (where writers spend much of their time), he expends all his strength trying to fight off the sharks who pick all the meat off the fish’s carcass. He returns to the marina with only the great skeleton. But the image of that skeleton and the exhausted old man are enough to tell the other fishermen what has happened. They can infer the story and the struggle. Like good readers, they reconstruct the fuller reality from the limited representation. And the writer goes exhausted to his bed to recuperate, hoping it was all worth the struggle.

It’s not just writers who distill and compress the meaning of experience. All people do it. You do it whenever you tell a friend in five minutes about some meaningful life event that unfolded over a period of weeks or months. You give them the skeleton when you describe the key players and the sequence of events in a tone of sorrow or wonder or warm humor. And then you leave it to them to put the meat on the bones. Good friends and good readers can do that. Friends and readers who are dull, inexperienced, or unwilling to extend sympathy and understanding cannot.

In the corporate world, the parable of the two programmers has a perverse side effect: the less capable team that implemented the inferior solution gets rewarded, while the more capable developer is discarded. In the software world, this leads to huge, buggy programs that developers are afraid to touch. In the book world, it leads to big weighty tomes that critics praise and readers feel guilty about not being able to finish.

This happens less often with writing than with software, because unlike computer users who can’t see the messy code behind a program’s shiny facade, readers are looking right at the words. Most readers prefer understanding to confusion, so they’ll avoid the bad stuff. But many mistake obscurity and complexity for depth and intelligence, especially in academia. Many writers (and programmers) are too lazy or too rushed to do the really hard work of sorting out complex ideas, clarifying vague concepts and distilling pages of meandering prose into a few striking sentences.

None of this is new. In 1710, in his conclusion to A Tale of a Tub, Jonathan Swift wrote:

I conceive therefore, as to the business of being profound, that it is with writers as it is with wells—a person with good eyes may see to the bottom of the deepest, provided any water be there; and that often, when there is nothing in the world at the bottom, besides dryness and dirt, though it be but a yard and a half underground, it shall pass, however, for wondrous deep, upon no wiser a reason than because it is wondrous dark.